Dont need to save 'savedAgentResultStruct' with RL agent

Question

Sayak Mukherjee 2021-2-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/753089-dont-need-to-save-savedagentresultstruct-with-rl-agent

评论： Sayak Mukherjee 2021-12-2

When I am saving agents during RL iterations using 'EpisodeReward' criteria, matlab is also saving 'savedAgentResultStruct' along with the agent which is increasing the file size. Is there any option to turn off saving the 'savedAgentResultStruct' file.

Thanks

4 个评论
显示 2更早的评论隐藏 2更早的评论

Sayak Mukherjee 2021-2-23

在 MATLAB Online 中打开

Hi Emmanouil,

Thanks for your reply. I am first defining the agent ptions

agentOptions = rlTD3AgentOptions;
agentOptions.SampleTime = Ts;
agentOptions.DiscountFactor = 0.99;
agentOptions.MiniBatchSize = 100;
agentOptions.ExperienceBufferLength = 1e4;
agentOptions.SaveExperienceBufferWithAgent=false;
agentOptions.TargetSmoothFactor = 5e-3;
agentOptions.TargetPolicySmoothModel.Variance = 0.2; % target policy noise
agentOptions.TargetPolicySmoothModel.LowerLimit = -0.5;
agentOptions.TargetPolicySmoothModel.UpperLimit = 0.5;
agentOptions.ExplorationModel = rl.option.OrnsteinUhlenbeckActionNoise; % set up OU noise as exploration noise (default is Gaussian for rlTD3AgentOptions)
agentOptions.ExplorationModel.MeanAttractionConstant = 1;
agentOptions.ExplorationModel.Variance = 0.09;

Then define the agent

agent = rlTD3Agent(actor, [critic1,critic2], agentOptions);

and then define the training options

trainOpts = rlTrainingOptions(...
    'MaxEpisodes',maxEpisodes,...
    'MaxStepsPerEpisode',maxSteps,...
    'ScoreAveragingWindowLength',250,...
    'Verbose',true,...
    'StopTrainingCriteria','AverageReward',...
    'StopTrainingValue',250,...                   
    'SaveAgentCriteria','EpisodeReward',... 
    'SaveAgentValue',50); 

then training

trainingStats = train(agent,env,trainOpts);

All of these steps I have used directly from the bipedal robot example. Let me know if changing any of the parameters will stop saving the 'savedAgentResultStruct' file.

I donot need to save the file along with every agent, I only need to save it when the training stops, wich iam saving

save(['filename' datestr(now,'mm_DD_YYYY_HHMM')],'trainingStats','-v7.3')

M.Y. C. 2021-3-5

编辑：M.Y. C. 2021-3-5

在 MATLAB Online 中打开

Unfortunately, there is no direct solution. Luckily, I found an indirect solution by manipulating reset function. My code includes getlatestfile.m function but not in function format. You may find the related code below, copy & paste the code as localResetFcn(in). The code takes apart your 'Agent__.mat' file into 2 .mat files: 'Agent__.mat' and 'savedAgentResultStruct.mat'. 'Agent__.mat' file contains only trained agent at given iteration. On the other hand, 'savedAgentResultStruct.mat' file is overwritten at every iteration. My code is not optimized, so you should set 'SaveAgentValue' at rlTrainingOptions to '-inf'. If you optimized the code, please share under this question.

function in = localResetFcn(in)
persistent a
if ~isempty(a)
    
    %This function returns the latest file from the directory passsed as input
    %argument
    
    %Get the directory contents
    dirc = dir('savedAgents');
    
    %Filter out all the folders.
    dirc = dirc(find(~cellfun(@isdir,{dirc(:).name})));
    
    %I contains the index to the biggest number which is the latest file
    [A,I] = max([dirc(:).datenum]);
    
    if ~isempty(I)
        latestfile = dirc(I).name;
        % Simplify agent
        FileName = latestfile
        FolderName = 'savedAgents';
        File = fullfile(FolderName, FileName)
        
        load(File)
        
        ResultFile = fullfile(FolderName, 'savedAgentResultStruct')
        save(ResultFile,'savedAgentResultStruct')
        
        save(File, 'saved_agent','-mat')
    end
    
    
end
    a = 1;
    
% Define random inital values as your simulation requires. 
end

Sayak Mukherjee 2021-12-2

Hi M.Y.C.

Were you able to optimize the code?

请先登录，再进行评论。

请先登录，再回答此问题。

Dont need to save 'savedAgentResultStruct' with RL agent

4 个评论
显示 2更早的评论隐藏 2更早的评论

回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Dont need to save 'savedAgentResultStruct' with RL agent

4 个评论 显示 2更早的评论隐藏 2更早的评论

回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

4 个评论
显示 2更早的评论隐藏 2更早的评论