How do I clear the Cache Storage when training a Neural Network with a Custom Training Loop?

Question

Julian Rehfeldt 2023-6-15

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1983439-how-do-i-clear-the-cache-storage-when-training-a-neural-network-with-a-custom-training-loop

回答： Himanshu 2023-8-28

Hello everybody,

I was playing around with the code found on here:

https://de.mathworks.com/help/deeplearning/ug/solve-partial-differential-equations-using-deep-learning.html

First I trained the net without changing anything in the code but the number of epochs to 300 instead of 3000. I got this result:

Everything like you would expect.

After that I modified the loss-function in the function "modelLoss":

I changed "loss = lossF + lossU;" to "loss = lossU;".

So I basically trained the net only with the Initial (U = sin(-pi*x)) and Boundary (U = 0) Conditions and got the following result:

That made me realize MATLAB has to store the results from the training before and used it here, otherwise the net could not produce a output that close to the true solution wihtout even knowing the PDE.

I tried to delete as much as possible between the two different trainings, so I tried "clear all" instead of "clear" and added "clearCache(accfun)" after the line "accfun = dlaccelerate(@modelLoss);". It had an inpact, but the output was still depending on the trainings I did beforehand.

This is the output after training only with Initial and Boundary Conditions when I freshly started MATLAB:

As you would expect just nonsense.

And this is the output after training only with Initial and Boundary Conditions when I trained with the "default"-loss-function before. The code included "clear all" and "clearCache(accfun)".

Still not a good solution but much better than the one after just starting MATLAB.

This made me conclude "clear all" and "clearCache(accfun)" is not enough to reset everything.

I researched a lot for a solution, but I wasn't successful so I would be very thankful if someone here has an idea how to approach this!

Thanks!

Julian

2 个评论
显示无隐藏无

Ayush 2023-8-11

编辑：Ayush 2023-8-11

在 MATLAB Online 中打开

Did you try reseting & clearing GPU using following command?

reset(gpudev);

You may refer to the documentation here: Reset GPU device and clear its memory - MATLAB reset - MathWorks India

Julian Rehfeldt 2023-8-13

MATLAB can't find a supported GPU device, but thanks!

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Himanshu 2023-8-28

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1983439-how-do-i-clear-the-cache-storage-when-training-a-neural-network-with-a-custom-training-loop#answer_1295037

Hello Julian,

I understand that you are facing an issue where the output of your neural network is influenced by previous training sessions, even after attempting to clear variables and cache.

The error likely occurred due to the internal states and cached information from the underlying framework affecting the network's behaviour.

You can resolve the issue by using the following steps:

Set the random seed before training each time to ensure consistent random initialization of network parameters.
After each training session, explicitly reinitialize the model's parameters before starting the next training session.
If you are using an optimizer like Adam, consider reinitializing the optimizer's internal states before each training session.
Instead of "clear all," consider using "clearvars" to clear variables while retaining necessary configurations and paths.
In cases where the internal state of the framework might be influencing training, consider restarting MATLAB itself.

You can refer to the below documentation to learn more about Random Number Generation and the "clearvars" function in MATLAB.

Control random number generator - MATLAB rng - MathWorks India

Clear variables from memory - MATLAB clearvars - MathWorks India