trainNetwork stops after one iteration

1 次查看（过去 30 天）

Niklas 2018-5-25

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/402519-trainnetwork-stops-after-one-iteration

评论： Niklas 2018-8-16

Hi all, i run into trouble using trainNetwork. Sometimes training stops after one iteration without any error message and the GPU utilization goes down to 0%. The process itself runs with 100% CPU utilization.

We use 4D-Imagearrays for training. The trainingOptions are:

training_options = trainingOptions('sgdm','MaxEpochs',200,...
    'InitialLearnRate', 0.01,...
    'ExecutionEnvironment', 'gpu',...
    'Shuffle', 'every-epoch',...
    'ValidationData', {validation_images, validation_labels},...
    'ValidationPatience', 8,...
    'MiniBatchSize', 256);

Our systems have 1080Ti's and 12G memory. For some reason this error does not occur on K80 GPUs. For me it seems that this is related to the architecture of the GPUs.

Does someone run into the same issue? Thanks for suggestions to solve this.

Edit: I tested all nodes. The training runs normal on the ones with K80 GPUs. On all machines with 1080TIs the training aborts after one iteration with high CPU utilization.