CUSOLVER_STATUS_INTERNAL_ERROR with RTX 4080 Super in MATLAB 2022a C MEX Function

Question

Stefanos 2024-6-20

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2130416-cusolver_status_internal_error-with-rtx-4080-super-in-matlab-2022a-c-mex-function

评论： Stefanos 2024-6-26，8:49

Hello,

I'm encountering a problem when running a CUDA-based C MEX function in MATLAB 2022a on a machine equipped with an RTX 4080 Super GPU. The specific error occurs when executing the line:

cusolverStatus_t status = cusolverDnCreate(&handle);

Here are the details:

The MEX function compiles successfully using the mexcuda command.
The same code operates correctly on this machine when using an RTX 2080Ti GPU with exactly the same software setup.
When the RTX 4080 Super is used, this error appears, even though the function runs without issues in Visual Studio, indicating that CUDA (Version 12.5) is properly configured on my system.
Upgrading to MATLAB 2024a resolves the issue, suggesting a compatibility problem with MATLAB 2022a. However, I am restricted to using MATLAB 2022a due to dependencies on other external software.

Additional context:

NVIDIA Driver Version: 555.99
CUDA Toolkit Version: 12.5

Has anyone else experienced similar issues with newer NVIDIA GPUs in MATLAB 2022a? Any insights or suggestions on how to resolve or circumvent this error with the RTX 4080 Super would be greatly appreciated.

Thank you!

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Umar 2024-6-21

Hi Stefanos, It seems that the compatibility issue lies between MATLAB 2022a and the RTX 4080 Super GPU. Upgrading to MATLAB 2024a resolves the problem, indicating a version-specific conflict. Since you are constrained to MATLAB 2022a, consider checking for any MATLAB updates or patches that address GPU compatibility issues. Additionally, ensure that all CUDA libraries and drivers are up to date. You may also try adjusting compiler options or CUDA configurations specific to the RTX 4080 Super GPU. Seeking support from MathWorks or NVIDIA forums could provide further insights or workarounds tailored to your setup.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Joss Knight 2024-6-24，9:34

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2130416-cusolver_status_internal_error-with-rtx-4080-super-in-matlab-2022a-c-mex-function#answer_1476231

编辑：Joss Knight 2024-6-25，11:51

在 MATLAB Online 中打开

This is a known issue with CUDA 11.2, Ada cards, and cusolver. The solution is to upgrade MATLAB to R2023a or above, because it requires a new version of CUDA which cannot be provided in an update release.

It is possible that you can be walked through dropping a fixed version of cusparse and cusolver from CUDA 11.3, that would be compatible with MATLAB R2022a and not broken. However, it would require you to edit your MATLAB installation. (SPOILER: This turned out to be the only solution that worked.) Upgrading is definitely preferable.

Since this is a mex function, you have the option to compile using your own CUDA toolkit rather than the one shipped with MATLAB. To do this you will need to make some edits to mex options in order to allow mexcuda to use a version of the toolkit that doesn't match the version of MATLAB. I can walk you through this if this is your preferred route. At the moment mexcuda is not using the 12.5 tookit you installed because the version is wrong.

EDIT: Actually, I think I know how to force mexcuda to compile with your toolkit without any options file changes:

mexcuda -v CUDA_LIB_PATH='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64' CUDA_ROOT='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5' LIBDEVICE='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\nvvm\libdevice' cusolver_handle_mex.cu

The -v will help you work out whether the correct version of nvcc was used. What I can't be sure of is that using CUDA 12.5 alongside CUDA 11.2 won't have unforeseen problems. If it does you might be better off installing CUDA 11.3 instead, which will be compatible with R2022a but won't have the cusolver bug.

18 个评论
显示 16更早的评论隐藏 16更早的评论

Joss Knight 2024-6-25，8:40

在 MATLAB Online 中打开

Well, it's still definitely loading the older version of cusolver and not the newer one. You can see for yourself in the output of version -modules:

C:\Program Files\MATLAB\R2022a\bin\win64\cusolver64_11.dll Version <unknown>

I can't tell whether it's because the OS doesn't know not to load that version, or whether it's something to do with the fact that the link line in your output doesn't look quite right, it says

link C:\Users\SFLORE~1\AppData\Local\Temp\mex_349698245540363_18908\cusolver_handle_mex.obj C:\Users\SFLORE~1\AppData\Local\Temp\mex_349698245540363_18908\c_mexapi_version.obj  cusolver.lib   "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64\cusolver.lib"

This looks okay except the /LIBPATH line looks suspect, because it specifies the file directly instead of just the path to the lib directory (I was expecting /LIBPATH:"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64"). I think perhaps this is just because of you trying what you said above as -L. But on a positive note, this is the first time your output link line has actually contained the path to the lib folder with all the path separators in the right direction.

So let's try one more time, but this time put the -L back, and then let's look at the mexcuda verbose output. You want

mexcuda -v ...
    CUDA_LIB_PATH='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64' ...
    CUDA_ROOT='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5' ...
    LIBDEVICE='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\nvvm\libdevice' ...
    'NVCCFLAGS=-gencode=arch=compute_89,code=sm_89' ...
    cusolver_handle_mex.cu ...
    -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64" ...
    -lcusolver

Failing this I feel our only choices are to go with one of the other two options:

Upgrade to at least R2023a
Edit your MATLAB installation to add unbroken versions of cusolver and cusparse DLLs from CUDA 11.3

Joss Knight 2024-6-25，11:50

Good to hear. Pity we had to resort to that though.

Stefanos 2024-6-26，8:49

Yes, it would have been nice to not have to hack it this way but since it works it is a good solution for the time being until we can use a newer Matlab release. Thank you again!

请先登录，再进行评论。

Answer 2

Joss Knight 2024-6-24，7:29

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2130416-cusolver_status_internal_error-with-rtx-4080-super-in-matlab-2022a-c-mex-function#answer_1476136

It's probably necessary for you to provide your code, since any number of things could have gone wrong prior to intializing cusolver which could prevent it from intializing successfully. For instance, did you start your mex function by calling mxGPUInit?

2 个评论
显示无隐藏无

Stefanos 2024-6-24，9:05

在 MATLAB Online 中打开

I am not sure what mxGPUInit as I cannot seem to find any documentation on it. Could you point me to a page or reference? Also, it seems weird that the same code works on the 2022a version with another GPU or with the actual GPU I have the issue with the 2024a version. In addition, I have never had to use this previously mxGPUInit and have been working with MEX GPU code for quite a while.

Here is a simplified version of my MEX code that produces the error in question:

#include "mex.h"
#include <cusolverDn.h>
#include <cuda_runtime.h>
#include <cstdio>
// Function to convert cuSOLVER status to a string
const char *cusolverGetErrorString(cusolverStatus_t status)
{
    switch (status)
    {
    case CUSOLVER_STATUS_SUCCESS:
        return "CUSOLVER_STATUS_SUCCESS";
    case CUSOLVER_STATUS_NOT_INITIALIZED:
        return "CUSOLVER_STATUS_NOT_INITIALIZED";
    case CUSOLVER_STATUS_ALLOC_FAILED:
        return "CUSOLVER_STATUS_ALLOC_FAILED";
    case CUSOLVER_STATUS_INVALID_VALUE:
        return "CUSOLVER_STATUS_INVALID_VALUE";
    case CUSOLVER_STATUS_ARCH_MISMATCH:
        return "CUSOLVER_STATUS_ARCH_MISMATCH";
    case CUSOLVER_STATUS_MAPPING_ERROR:
        return "CUSOLVER_STATUS_MAPPING_ERROR";
    case CUSOLVER_STATUS_EXECUTION_FAILED:
        return "CUSOLVER_STATUS_EXECUTION_FAILED";
    case CUSOLVER_STATUS_INTERNAL_ERROR:
        return "CUSOLVER_STATUS_INTERNAL_ERROR";
    case CUSOLVER_STATUS_MATRIX_TYPE_NOT_SUPPORTED:
        return "CUSOLVER_STATUS_MATRIX_TYPE_NOT_SUPPORTED";
    default:
        return "Unknown cuSOLVER error";
    }
}
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
    // Print initial diagnostic information
    mexPrintf("Starting cuSOLVER handle initialization.\n");
    // Declare a cuSOLVER handle
    cusolverDnHandle_t handle;
    // Initialize the cuSOLVER handle
    cusolverStatus_t status = cusolverDnCreate(&handle);
    if (status != CUSOLVER_STATUS_SUCCESS)
    {
        mexErrMsgIdAndTxt("cusolver:initializationFailed",
                          "CUSOLVER initialization failed: %s", cusolverGetErrorString(status));
    }
    mexPrintf("cuSOLVER handle initialized successfully.\n");
    // Destroy the cuSOLVER handle
    status = cusolverDnDestroy(handle);
    if (status != CUSOLVER_STATUS_SUCCESS)
    {
        mexErrMsgIdAndTxt("cusolver:destructionFailed",
                          "CUSOLVER destruction failed: %s", cusolverGetErrorString(status));
    }
    mexPrintf("cuSOLVER handle destroyed successfully.\n");
}

In addition here is the compilation command used:

% Set CUDA_PATH to the appropriate path
% setenv('MW_CUDA_FORWARD_COMPATIBILITY', '1');
setenv('CUDA_PATH', 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5');
% Add CUDA binaries to the system PATH
setenv('PATH', [getenv('CUDA_PATH') '\bin;' getenv('PATH')]);
% Compile the MEX function
mexcuda -v cusolver_handle_mex.cu ...
   -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64" ...
   -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\include" ...
   -lcusolver -lcudart -lcublas