Reinforcement learning DDPG action fluctuations
显示 更早的评论
Upon attempting to train the path following control example in MATLAB, the training process generated the behviour shown in the picture.
- The steering angle is constantly fluctuating.
- The acceleration is also constantly flucutating.
- The reward convergence is very noisy and seems to jump between a high reward and low reward.
What could be causing this issue? This also happened for other projects I used. One method I used was to penalise the fluctuation in the reward function using this term inspired by a paper published by Wang et. al:
10*[ (d/dt(current_action) * d/dt(previous_action) < 0]
Please let me know how to avoid this problem. Thank you very much!
2 个评论
Emmanouil Tzorakoleftherakis
2020-11-17
Hello,
One clarification - the scope signals you are showing on the right, are you getting these during training or after training?
Tech Logg Ding
2020-11-17
采纳的回答
Emmanouil Tzorakoleftherakis
2020-11-22
0 个投票
Hello,
During training, DDPG explores the action space by adding noise to the output of the actor (see step 1 here). That explains the variance during training.
Even after training you may see small variations in the actor output for observations that are different but close enough. After all you are effectively using a function approximator to approximate a nonlinear relationship between inputs (observations) and outputs (actions). If you want to get the policy to be more accurate near the setpoint, you could consider training further near the values of interest.
Also, the result you get on your machine may differ from the one posted in the documentation. Please see this post for an explanation.
Hope that helps
2 个评论
sungho park
2022-2-23
for me after training, the actor output is always constant. can you explain why?
Karim Darwich
2024-7-1
@sungho park I have the same problem. Did you fix it ?
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Policies and Value Functions 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
