CNN Performance: CPU Consistency vs. GPU Variance - Why?

Question

Hamza 2023-11-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2050842-cnn-performance-cpu-consistency-vs-gpu-variance-why

回答： Ruth 2023-11-23

Hello everyone,

I have executed CNN code multiple times using rng(0) with CPU and consistently obtained the same result. However, when I attempted to accelerate the training process using the GPU, the results differed. Has anyone else faced this issue?

Thank you in advance!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Ruth 2023-11-23

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2050842-cnn-performance-cpu-consistency-vs-gpu-variance-why#answer_1358587

Hi Hamza,

Even when using "gpurng" some small non-deterministic behavior is expected to happen in the GPU during training, particularly during the backward pass. This is out of our control.

However the behavior should be deterministic in the forward pass and subsequently at prediction time.

If one sets the learning rate to be almost zero (e.g. 1e-16, meaning nothing is updated in the backward pass), the output of training (using "rng" and "gpurng") should look deterministic.

Best wishes,

Ruth