PPO Agent training - Is it possible to control the number of epochs dynamically?

Question

Federico Toso 2024-3-17

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2095436-ppo-agent-training-is-it-possible-to-control-the-number-of-epochs-dynamically

评论： Federico Toso 2024-3-25

In the deault implementation of PPO agent in Matlab, the number of epochs is a static property that must be selected before the training starts.

However I've seen that state-of-the-art implentations of PPO sometimes select dynamically the number of epochs: basically, for each learning phase, the algorithm decides whether to execute a new epoch or not, basing on the value of the KL divergence just calculated. This seems to help the robustness of the algorithm significanlty.

Is it possible for a user to implement such a routine in Matlab in the context of PPO training, possibly applying some slight modifications to the default process?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Kartik Saxena 2024-3-22

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2095436-ppo-agent-training-is-it-possible-to-control-the-number-of-epochs-dynamically#answer_1429461

在 MATLAB Online 中打开

Hi,

Given below is the code snippet depicting the logic/pseudo algorithm you can refer to for this purpose:

% Assume env is your environment and agent is your PPO agent
for episode = 1:maxEpisodes
    experiences = collectExperiences(env, agent);
    klDivergence = inf;
    epochCount = 0;
    
    while klDivergence > klThreshold && epochCount < maxEpochs
        oldPolicy = getPolicy(agent);
        agent = updateAgent(agent, experiences);
        newPolicy = getPolicy(agent);
        
        klDivergence = calculateKLDivergence(oldPolicy, newPolicy);
        epochCount = epochCount + 1;
    end
end