GPU Coder used, but got error: Error generated while running CUDA-enabled program: [700,cudaE​rrorIllega​lAddress]

8 次查看（过去 30 天）
Tong Zhao
Tong Zhao 2021-6-14
编辑： Gautam 2023-1-19
I'm using GPU coder to accelerate N-D interpolation function interpn().
Everything worked fine when my problem scale was small, i.e. interpolating in 1-D or 2-D dimensions. But when I throw a larger problem, of 5-D interpolation to it, MATLAB threw the following error message:
Error generated while running CUDA-enabled program: [700,cudaErrorIllegalAddress] an
illegal memory access was encountered. To generate additional debugging information,
enable the 'SafeBuild' GPU code generation option.
I followed the error message suggestion above and added the 'SafeBuild' GPU code generation option, and got the following error instead:
Error generated while running CUDA-enabled
program: [46,cudaErrorDevicesUnavailable] all
CUDA-capable devices are busy or unavailable
I know that when I wasn't using GPU Coder, the code still worked but kind of slow. But now when GPU coder is enabled, it is not working. I have NVIDIA RTX 2080ti with 11G memory. When I was using CPU to do the calculation, the problem took 1 hour and peak RAM use was around 40GB out of 64GB RAM. Could this error be because I ran out of GPU memory by establishing my problem too large? How can I tell if this is the case?
OK update: I rebooted my Windows PC, and ran the same program again, the following error showed instead:
Error generated while running CUDA-enabled
program: [2,cudaErrorMemoryAllocation] out of
memory
I think this confirms that indeed my problem is causing GPU to run out of memory just to start the problem. Now I know the problem, is there any suggestions how to split the problem for interpn? Thanks!
  3 个评论
显示 1更早的评论
Tong Zhao
Tong Zhao 2021-8-2
Hi Hari,
Thank you for your attention. I got answer to this problem with some help. Indeed my problem was so big that it created arrays with size larger than 4GB (double arrays with size 2^32) to be interpolated. There's a limit to how large the indices of an array could be (int32), I think this might be why. I'm still curious though as to how one could expand the size limit to beyond (int32).
First
First 2021-12-7
hi may i know the solution ？ i met same error of cuda out of memory but its quite weird. Same code can run cpu1+gtx980 but error above when i run on cpu2+gtx980 and cpu2 has more much more cores and memory than cpu1.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Gautam
Gautam 2023-1-19
编辑：Gautam 2023-1-19
Hi,
The error is most likely that a matrix is growing beyond it's intended size. Take a careful look at your code to analyze which matrix is growing beyond the intended value.
Growing size matrices are silently handled by MATLAB, but the coder generates CUDA C which does not handle growing sizes so it is mostly seg-faulting when you run the generated code.
Try running your code in MATLAB and check if every matrix at the end of its computation is having the intended size.
Also, when you run into a "cudaErrorIllegalAddress" you have to restart MATLAB and run your code. This is because, when the coder runs the generated code for the first time, it creates a CUDA context and it reuses the same context in the subsequent runs till the MATLAB application is closed. Now, when you encounter a seg-fault when running the generated code, the context gets corrupted, MATLAB doesn't know about it and so it tries to run the same context in the next run, which will fail. So the solution is to restart MATLAB.
Gautam

类别

在 Help Center 和 File Exchange 中查找有关 Kernel Creation from MATLAB Code 的更多信息

标签

产品


版本

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by