Reinforcement learning - DDPG - minibatch - Continuos action saturation

Question

Oscar Emilio Aponte Rengifo 2021-11-10

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1583354-reinforcement-learning-ddpg-minibatch-continuos-action-saturation

回答： Yash 2024-2-21

what is the influence of the minibatch? since in this case its value determines the moment in which the continuous action signal begins to saturate (it only adopts the values of the limits-blue line). In the case in which the value of the minibatch is very large, the weights of the nets do not change over time. Action[-0.15 0.15],

I don't know why before the minibatch which is when it starts from random weights it does not saturate but it is the value of the minibatch that determines where the saturation begins

data:

GradientThreesHold: 1

Learnrate: 0.03

agentOpts.NoiseOptions.StandardDeviation = 0.001;

agentOpts.NoiseOptions.StandardDeviationDecayRate =0.00001;

minibatch: 100 --- x axis=500

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Yash 2024-2-21

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1583354-reinforcement-learning-ddpg-minibatch-continuos-action-saturation#answer_1413503

The minibatch size in a neural network training process refers to the number of samples used in each iteration to update the weights of the network. A larger minibatch size can lead to slower weight updates and potentially slower convergence, while a smaller minibatch size can result in faster weight updates but with more noisy updates.

In your case, it seems that a larger minibatch size is causing the action signal to saturate earlier. This could be because with a larger minibatch size, the weight updates are slower, and therefore the network takes longer to adapt to the changing environment. As a result, the action signal reaches its limits ([-0.15, 0.15]) earlier. It may be worth experimenting with different minibatch sizes to find the optimal value for your specific case.