Incorrect tanhLayer output in RL agent

Question

Mohammad Ashraful Islam 2020-4-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/515602-incorrect-tanhlayer-output-in-rl-agent

评论： H. M. 2022-10-20

Last layer in my actor network is set to tanhLayer. However, I am seeing output that goes above 1 or below -1 from the RL agent block. Is this normal behavior of RL agent?

4 个评论
显示 2更早的评论隐藏 2更早的评论

Asvin Kumar 2020-4-11

I am unable to reproduce the error. Here's what I got:

Each view corresponds to a leg of the bipedal robot. The three signals are the normalized torques applied to the ankle, knee and hip.

Mind sharing your model to have a look?

Mohammad Ashraful Islam 2020-4-13

在 MATLAB Online 中打开

I am just making sure, did you make the following change?

replace:

actInfo = rlNumericSpec([numAct 1],'LowerLimit',-1,'UpperLimit',1);

with

actInfo = rlNumericSpec([numAct 1]);

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Asvin Kumar 2020-4-13

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/515602-incorrect-tanhlayer-output-in-rl-agent#answer_425717

I’ve tried this. I still don’t see the values going beyond [–1, 1]. However, I might be able to answer your question. If you have a look at the helper functions createTD3Agent.m and createDDPGAgent.m, you will notice the ‘agentoptions’ object. The parameters called ‘ExplorationModel’ or ‘NoiseModel’ specify details about the kind of noise added to the predicted action. This can either be an ‘OrnsteinUhlenbeckActionNoise’ object or a ‘GaussianActionNoise’ object each with their own set of parameters. Have a more detailed look at the Noise Options here: rlDDPGAgentoptions and rlTD3AgentOptions. This noise is added to encourage the agent to explore the environment.

The output action from the tanhLayer in the ‘actorNetwork’ will still be in the range of [–1, 1]. Once the noise is added, the new action values will be saturated to the limits specified in the ‘ActorInfo’. These limits will be [-Inf, Inf] by default and won’t saturate your action values when not mentioned.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Abdul Basith Ashraf 2021-4-5

编辑：Abdul Basith Ashraf 2021-4-5

If only I knew when that noise was added, I could have saved a ton of my time. Finally I know it. Noise is added to the output of the actor. I had used a default value for variance of 0.6, but my input was in the range -1e-4 to +1e-4 And because of that my output was always saturated

H. M. 2022-10-20