why agent failed to get accelerated after training?

Question

Kun Cheng 2023-4-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1949243-why-agent-failed-to-get-accelerated-after-training

回答： Piyush Dubey 2023-6-2

Hi,

I trained an pre-trained agent in the same environment. I expect that, model should converge faster but it did not happen.

first pic: first training

second pic: with trained agent

it seems agent do the same training once again. My question is why the second one not faster?

agent setting:

agentOpts=rlDQNAgentOptions(...
    'UseDoubleDQN',true,...
    'MiniBatchSize', 64, ...
    'SaveExperienceBufferWithAgent',true);
'rlDQNAgentOptions' requires Reinforcement Learning Toolbox.
agentOpts.EpsilonGreedyExploration.EpsilonDecay=1e-3;
agentOpts.EpsilonGreedyExploration.Epsilon=0.9;
agentOpts.CriticOptimizerOptions.LearnRate=0.01;
agentOpts.CriticOptimizerOptions.GradientThreshold=1;
Train_Old_Model = true; % Set to true, to use pre-trained
agentOpts.ResetExperienceBufferBeforeTraining = not(Train_Old_Model);
if Train_Old_Model
    % Load experiences from pre-trained agent    
    load("XYAgent.mat",'agent');
    
else
    %new DQN Agent
    agent = rlDQNAgent(critic,agentOpts);
end

traning setting

maxEpisodes = 1300;
maxStepsPerEpisode = 20;
trainOpts = rlTrainingOptions(...
    MaxEpisodes=maxEpisodes, ...
    MaxStepsPerEpisode=maxStepsPerEpisode, ...
    Verbose=false, ...
    ScoreAveragingWindowLength=100,...
    Plots="training-progress",...
    StopTrainingCriteria="EpisodeCount",...
    StopTrainingValue=maxEpisodes);
plot(env)
%train
doTraining = true;
if doTraining
    % Train the agent.
    trainingStats = train(agent,env,trainOpts);
    save("XYAgent.mat","agent")
else
    % Load the pretrained agent for the example.
    load("XYAgent.mat","agent")
end

Thank you!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Piyush Dubey 2023-6-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1949243-why-agent-failed-to-get-accelerated-after-training#answer_1249159

Hi Kun,

There are various reasons because of which an agent may take longer to converge. Various ways by which a model can be saved, and the training can be resumed can be found in the documentation below:

https://www.mathworks.com/help/reinforcement-learning/ug/train-reinforcement-learning-agents.html

The reasons why a pre-trained agent can take longer in the same environment are: