Use current simulation data to initialize new simulation - RL training
2 次查看(过去 30 天)
显示 更早的评论
In the context of PPO Agent training, I would like to use Welford algorithm to calculate the runninig average & and standard deviation of my observations, in order to standardize them and improve the convergence of actor & critic neural networks.
I implemented the algorithm, but I don't know how to keep track of the current running statistics (average and standard deviation) every time a new simulation starts, during the training. This is what I would like to do:
- Whenever a simulation terminates (i.e. "isDone" flag is set to 1) , save the current value of runnig statistics in Matlab workspace
- While initializing the new simulation, set the starting value of the running statistics to match the values just saved in Matlab workspace
Note that I'm using the standard "train" function to run the training, so the transition between one simulation and the next one is handled automatically and I don't have much flexibility in this sense.
I thought about using the "ResetFcn" function handle within my "SimulinkEnvWithAgent" object to accomplish the task, but I am still not able to programmatically save the last value of my signal to the Workspace at the end of a simulation, and then pass it to the ResetFcn as additional argument in order to initialize the next one
0 个评论
采纳的回答
Poorna
2024-3-31
Hi Federico Toso,
I see you want to save simulation data to workspace to later use it in your "ResetFcn". A suitable tool for this is the "rlDataLogger" object, which enables you to log simulation data at various points, such as after each step, episode and after each learn subroutine. You can craft a custom function for logging the specific statistics you're interested in and then assign this function to the appropriate callback property of the rlDataLogger. Although logging typically saves data to a folder after training concludes, your custom callback function can be used to immediately write the necessary statistics to the MATLAB workspace.
You can create a "rlDataLogger" object as below:
logger = rlDataLogger();
For instance, to log the ActorLoss value after every episode, your episode finish callback function could be structured like this:
function dataToLog = episodeFinish(data)
assignin('base', 'actorLoss', data.ActorLoss);
dataToLog = data.ActorLoss;
end
And then assign the function handle to the corresponding callback property of the data logger object as below:
logger.EpisodeFinishedFcn = @episodeFinish;
To learn more about the "rlDataLogger" function refer to the below documentation:
Hope this Helps!
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Training and Simulation 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!