why I get a different action result every new time with same sample observations after deploying trained RL policies?

Question

de y 2021-2-23

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/753454-why-i-get-a-different-action-result-every-new-time-with-same-sample-observations-after-deploying-tra

编辑： liang zhang 2022-3-2

采纳的回答： Emmanouil Tzorakoleftherakis

load("agent0218_300016_40000.mat","agent");

obsInfo = getObservationInfo(agent);

actInfo = getActionInfo(agent);

ResetHandle = @() myResetFunction(test_sss);

StepHandle = @(Action,LoggedSignals) myStepFunction(Action,LoggedSignals,test_sss);

envT = rlFunctionEnv(obsInfo,actInfo,StepHandle,ResetHandle);

simOpts = rlSimulationOptions('MaxSteps',size(test_sss,1));

experience = sim(envT,agent,simOpts);

ac3=squeeze(experience.Action.bs.Data);

%******************************************************************************

generatePolicyFunction(agent);

%******************************************************************************

for iii=1:size(ac3,1)

observation1=test_sss{iii,:};

action1(iii,1) = evaluatePolicy(observation1);

end

sum(abs(ac3-action1))

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Emmanouil Tzorakoleftherakis 2021-2-23

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/753454-why-i-get-a-different-action-result-every-new-time-with-same-sample-observations-after-deploying-tra#answer_631234

Which agent are you using? Some agents are stochastic, meaning that the output is sampled based on probability distributions so by construction they won't give you the same result.

Another possible reason is the reset function. It seems you are saving simulation data and running inference again, but every time you call 'sim', the reset function is called first. So if there are any components that randomize initial conditions/parameters, then you are not comparing with the same data.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

liang zhang 2022-3-2

编辑：liang zhang 2022-3-2

I also encountered the same problem when I used the DDPG agent for verification, my reset function doesn't randomize initial any conditions/parameters，I guess if the trained DDPG agent also has its own noise? Shouldn't a trained agent be a fixed set of neural network parameters?

请先登录，再进行评论。

Answer 2

de y 2021-2-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/753454-why-i-get-a-different-action-result-every-new-time-with-same-sample-observations-after-deploying-tra#answer_631674

Thanks @Emmanouil Tzorakoleftherakis a lot.

I am using PPO and SAC agent, the same question came out. My codes indicated the agent had trainned to a satisfied and balanced result, I want to use it to decide action. But my wonder is that SIM is one of simulation way,whereas generatePolicyFunction() and evaluatePolicy is another way, my observations of every step is the same,why every running evaluatePolicy with the same observations happened , the different action result with SIM() came out. It confused me because that there didn't had any components that randomize initial conditions/parameters