The reward gets stuck on a single value during training or randomly fluctuates (Reinforcement Learning)

Question

Vasiliy Polushkin 2020-4-28

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/521467-the-reward-gets-stuck-on-a-single-value-during-training-or-randomly-fluctuates-reinforcement-learni

编辑： Ari Biswas 2021-6-13

I train the reinforcement learning system, and on the reward plot I have some failures during which the reward does not change. This doesn’t look normal, especially when compared with examples (Biped Robot, etc.) I believe that some rlDDPGAgentOptions settings are responsible for this, but it seems that I changed all the possible settings, but even after several thousand episodes, the system does not learn. What can be the reason for this behavior of this graph during training?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Ari Biswas 2020-5-5

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/521467-the-reward-gets-stuck-on-a-single-value-during-training-or-randomly-fluctuates-reinforcement-learni#answer_430438

It could mean that the training is experiencing a local minima. You can try out a few things:

1. Change the OU noise options to favor more exploration so that the robot can explore more states and get new rewards.

2. Design a different reward function that is not too dependent on sparse rewards. From the graph (flatlines) it looks like you have a sparse reward for a state that the agent is continuously visiting.

In most cases, designing better reward functions will improve training. That being said, 350 episodes might be too early to expect good results. I would let it run for a few 1000 episodes at least before coming to a conclusion that something needs to change.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Abd Al-Rahman Al-Remal 2021-6-12

Hi,

When you say to change the noise options to favour more exploration: how would this be implemented? i.e what parameters should be changed and in what manner?

My case is slightly different than OP's however as my agent just stays at the same reward value consistently (I've never tested it for more than 100 episodes or so however).

Many thanks!

Ari Biswas 2021-6-13