CUDA_ERROR_ILLEGAL_ADDRESS with "trainNetwork" on NVidia RTX8000
4 次查看(过去 30 天)
显示 更早的评论
I am using Matlab R2020b to train a deep neural network on an image classification task with trainNetwork(). My hardware is an NVidia Quadro RTX8000 and I have CUDA 11.2 installed on Windows Server 2019. My input images are 512x512 px, in total I have ~1M training images. If I train with a mini batch size of 512, the training works well and 22 GB of GPU RAM are used. However, my GPU has 48 GB of RAM and I want to fully use it, so I increase my mini batch size to 1024.
When I do this, I get the error
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In nnet.internal.cnn/DAGNetwork/forwardPropagationWithMemory (line 347)
In nnet.internal.cnn/DAGNetwork/computeGradientsForTraining (line 716)
In nnet.internal.cnn/Trainer/computeGradients (line 200)
In nnet.internal.cnn/Trainer/train (line 119)
In nnet.internal.cnn.trainNetwork.doTrainNetwork (line 91)
In trainNetwork (line 181)
In trainMyNetwork (line 36)
this is the output of gpuDevice
>> gpuDevice
ans =
CUDADevice with properties:
Name: 'Quadro RTX 8000'
Index: 1
ComputeCapability: '7.5'
SupportsDouble: 1
DriverVersion: 11.2000
ToolkitVersion: 10.2000
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [2.1475e+09 65535 65535]
SIMDWidth: 32
TotalMemory: 5.1540e+10
AvailableMemory: 4.3099e+10
MultiprocessorCount: 72
ClockRateKHz: 1770000
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1
It seems to me that Matlab has an issue to access more than 24 GB of GPU RAM, but how do I fix this?
Thank you!
0 个评论
回答(1 个)
Aditya Patil
2021-2-17
As this error occurs only for large batch size, one possibility is that the driver is outdated and not detecting out of memory errors properly. You can update the driver to latest version and try again. If that doesn't work, you can comment here or report a bug to MathWorks via the contact us page.
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 GPU Computing 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!