trainNetwork stops after one iteration

3 次查看（过去 30 天）
Niklas
Niklas 2018-5-25
评论： Niklas 2018-8-16
Hi all, i run into trouble using trainNetwork. Sometimes training stops after one iteration without any error message and the GPU utilization goes down to 0%. The process itself runs with 100% CPU utilization.
We use 4D-Imagearrays for training. The trainingOptions are:
training_options = trainingOptions('sgdm','MaxEpochs',200,...
'InitialLearnRate', 0.01,...
'ExecutionEnvironment', 'gpu',...
'Shuffle', 'every-epoch',...
'ValidationData', {validation_images, validation_labels},...
'ValidationPatience', 8,...
'MiniBatchSize', 256);
Our systems have 1080Ti's and 12G memory. For some reason this error does not occur on K80 GPUs. For me it seems that this is related to the architecture of the GPUs.
Does someone run into the same issue? Thanks for suggestions to solve this.
Edit: I tested all nodes. The training runs normal on the ones with K80 GPUs. On all machines with 1080TIs the training aborts after one iteration with high CPU utilization.
  2 个评论
Joss Knight
Joss Knight 2018-6-2
The training stops after a single iteration with no message at all? Or do you mean it stops after displaying only a single line of verbose output?
Set 'VerboseFrequency' to 1 and 'ValidationPatience' to Inf and run again, just to check.
Niklas
Niklas 2018-8-16
Hi Joss,
yes the training stops without any further messages. I submitted a service request and we weren't able to track this error.
We now switched to 2018a and everything seems fine so far.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Parallel and Cloud 的更多信息

标签

产品


版本

R2017b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by