How to reduce noise from SAC RL-agent?

2 次查看（过去 30 天）
Oleg Makarov
Oleg Makarov 2023-10-11
编辑： Emmanouil Tzorakoleftherakis 2023-10-13
Good afternon!
I trained SAC agent to balance pendulum (normal one, not inverted) which is the part of crane system. The results are apropriate but actions from agent include not only mean actions but also standart deviations, which agent used during training. In that way the response of my pendulum is not stable enough. Agent do actions with noise but I need only mean part of these actions without noise. How can I do that? Is there any possible way to exclude standart deviations from actions after training?
I checked this example - https://www.mathworks.com/help/reinforcement-learning/ug/train-sac-agent-for-ball-balance-control.html
In the conclusion the response of the ball is very smooth without any shaking. How did they do that?

请先登录，再回答此问题。

回答（2 个）

Emmanouil Tzorakoleftherakis
编辑：Emmanouil Tzorakoleftherakis 2023-10-13
Hi,
There is an agent option that achieves exactly what you want and outputs the mean output values after training:
sacagent.UseExplorationPolicy = 0;
You also need to keep in mind that there is no guarantee that your action signal will be continuous, even if you only use the mean value. That's how the SAC algorithm works. You may either need to train further to reduce this variance, or you could potentially add a penalty term in the reward to make the action signal smoother. Another option is to try using DDPG which are deterministic in nature and generate smoother actions.
The example that you shared in your question only shows the plot of the ball position - the SAC actions could still be noisy/discontinuous though.
Hope this helps
Oleg Makarov
Oleg Makarov 2023-10-13
编辑：Oleg Makarov 2023-10-13
Thank you for answer. I tried to use the option UseExplorationPolicy = 0; and nothing has changed. The Agent still uses the action with noise inside the action limits prescribed during agent creation (-20;20). Please look the attached figure.
  1 个评论
Emmanouil Tzorakoleftherakis
Emmanouil Tzorakoleftherakis 2023-10-13
A couple of suggestions:
Once you set UseExplorationPolicy = 0, try to run 'getAction(agent, obs)' a couple of times for the same observation input to make sure you are getting the same action output. If you do get the same output, then this means that you are indeed not sampling your actions anymore.
You also need to keep in mind that there is no guarantee that your action signal will be continuous, even if you only use the mean value. That's how the SAC algorithm works. You may either need to train further to reduce this variance, or you could potentially add a penalty term in the reward to make the action signal smoother. Another option is to try using DDPG which are deterministic in nature and generate smoother actions.
The example that you shared in your question only shows the plot of the ball position - the SAC actions could still be noisy/discontinuous though.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Reinforcement Learning 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by