Error using parallel-cpu with trainnet function
显示 更早的评论
Hi, I got this error when trying to train a neural network using the trainnet function. The first time I trained the network everything works fine, but when I tried to use validation data I got an error. Then I tried to retrain the network without validation data but I got this error:
Error detected on worker 3.
net = train(trainer, net, mbq);
[net,info] = deep.internal.train.trainnet(mbq, net, loss, options, ...
Caused by:
Out of Memory during deserialization
I have deleted the Jobs of the cluster but nothing worked.
Can anybody help me to solve the issue?
3 个评论
Saurabh
2024-11-22
Could you please share the details regarding the neural network and the memory allocation per worker? The minimum required memory is 4GB, with 8GB recommended. To optimize performance, consider increasing the memory available to each worker, which can typically be achieved by running fewer workers per compute node.
Ramiro
2024-11-22
Subhajyoti
2024-12-2
回答（1 个）
Sivsankar
2025-3-7
0 个投票
Without the ability to execute the scripts, it is challenging to pinpoint the exact solution to this error. However, I can offer some troubleshooting steps that may assist you.
As indicated by the error message, this could be a memory-related issue concerning the workers. Ensure that sufficient memory is allocated per core on the cluster. Since the error occurred during network retraining, it is important to verify that the allocated memory is properly cleared.
Additionally, the error may suggest that the GPU lacks adequate memory to complete the network training. To mitigate memory requirements, consider decreasing the "MiniBatchSize" training option or reducing the size of your training dataset. Also, ensure that the GPU memory is cleared before initiating retraining.
I hope these suggestions prove helpful.
类别
在 帮助中心 和 File Exchange 中查找有关 Pattern Recognition 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
