PPO agent low reward episodes

10 次查看（过去 30 天）
Sourabh
Sourabh 2023-12-11
评论： Sourabh 2024-1-3
I am trying to implement PPO agent and i m getting rewards as shown and i have tried tuning hyperparameter settings but still training looks like this and I dont know what is the isssue I also tried to use Isdone signal to terminatee the episode when reward reaches below a certain value but still no use . can someone help.?
  1 个评论
Sourabh
Sourabh 2023-12-14
it feels like my algorithm never starts exploiting and is just exploring throughout the training how can i reduce the exploration?

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Shivansh
Shivansh 2023-12-27
Hi Sourabh,
I understand that you are training a PPO agent which is stuck in exploration and not able to learn a stable policy. This is resulting in poor training as shown in the provided graph.
Here are a few steps to improve the performance of your model:
  1. Try normalizing or scaling down the rewards to a smaller range.
  2. A PPO agent uses entropy regularization to encourage exploration. Try reducing the entropy regularization coefficient gradually and observe the results.
  3. You can try verifying your “IsDone” flag’s implementation. It can terminate the episode at the right time to prevent the agent from learning undesired states.
Apart from the above points, you can also try experimenting with learning rates, hyperparameter tuning, and network architecture, etc to find the best configurations for your problem.
You can use a known benchmark environment to verify that your PPO implementation works as expected. Once you have confirmed that the agent can learn in a simpler setting, gradually reintroduce complexity.
You can refer to the common lander vehicle example to learn more about the working of a PPO agent in MATLAB: https://in.mathworks.com/help/reinforcement-learning/ug/train-ppo-agent-to-land-vehicle.html.
For more information on PPO agents, refer to the following documentation: https://in.mathworks.com/help/reinforcement-learning/ug/ppo-agents.html.
If the issue persists, provide more information related to problem statement and the environment. It will help to get a better understanding of the issue you are facing.
Hope it helps!
  3 个评论
显示 1更早的评论
Shivansh
Shivansh 2024-1-3
Hi Sourabh!
You don't have to change the structure of reward function to normalize rewards. You can use reward normalization techniques like dividing by a maximum possible reward or using statistical normalization (subtracting the mean and dividing by the standard deviation of the rewards).
Adding a large positive constant might not be a good way as this might lead to suboptimal policies. If adding a constant improves learning, there might be a case that the original rewards might be too negative or sparse. You can try revising your reward function to better balance positive and negative rewards, guiding the agent more effectively towards the optimal policy.
Hope it helps!
Sourabh
Sourabh 2024-1-3
so do i need to modify my script or change something in my env model (sorry but I am very new to all these and dont have much idea) i am attaching the script and env file for reference.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Reinforcement Learning 的更多信息

标签

产品


版本

R2023a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by