Error using parallel-cpu with trainnet function

Question

Ramiro 2024-11-21

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2167878-error-using-parallel-cpu-with-trainnet-function

回答： Sivsankar 2025-3-7

Hi, I got this error when trying to train a neural network using the trainnet function. The first time I trained the network everything works fine, but when I tried to use validation data I got an error. Then I tried to retrain the network without validation data but I got this error:

Error using deep.internal.train.Trainer/train (line 74)

Error detected on worker 3.

Error in deep.internal.train.trainnet (line 54)

net = train(trainer, net, mbq);

Error in trainnet (line 42)

[net,info] = deep.internal.train.trainnet(mbq, net, loss, options, ...

Caused by:

Error using deep.internal.train.ParallelTrainer/computeTraining (line 101)

Out of Memory during deserialization

I have deleted the Jobs of the cluster but nothing worked.

Can anybody help me to solve the issue?

3 个评论
显示 1更早的评论隐藏 1更早的评论

Ramiro 2024-11-22

SPCDataPre.mlx

Hi @Saurabh, I have attached the files of the source code.

Regards

Ramiro

Subhajyoti 2024-12-2

Hi @Ramiro,

Also share the train and test MAT files.

They are required to run the script.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Sivsankar 2025-3-7

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2167878-error-using-parallel-cpu-with-trainnet-function#answer_1561358

Hi @Ramiro,

Without the ability to execute the scripts, it is challenging to pinpoint the exact solution to this error. However, I can offer some troubleshooting steps that may assist you.

As indicated by the error message, this could be a memory-related issue concerning the workers. Ensure that sufficient memory is allocated per core on the cluster. Since the error occurred during network retraining, it is important to verify that the allocated memory is properly cleared.

Additionally, the error may suggest that the GPU lacks adequate memory to complete the network training. To mitigate memory requirements, consider decreasing the "MiniBatchSize" training option or reducing the size of your training dataset. Also, ensure that the GPU memory is cleared before initiating retraining.

I hope these suggestions prove helpful.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Error using parallel-cpu with trainnet function

3 个评论
显示 1更早的评论隐藏 1更早的评论

回答（1 个）

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Error using parallel-cpu with trainnet function

3 个评论 显示 1更早的评论隐藏 1更早的评论

回答（1 个）

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

3 个评论
显示 1更早的评论隐藏 1更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论