Is my DDPG agent learning?

4 次查看（过去 30 天）
Bryan
Bryan 2024-5-2
回答： Yatharth 2024-5-20
Hello everyone,
Can I conclude that my agent is learning? (maximum reward per episode is 20).
In the first image, the reward was low (-5) for the first episode, and it can be observed that the average reward starts to increase from episode 80. However, it fluctuates between 5 and 20 after episode 100. Reading other questions, it was mentioned that Q0 could help determine if the agent is learning, and as it approaches the maximum reward, I think it could be determined that it is learning. However, what makes me doubt the learning are the fluctuations in rewards after episode 100.
image1
Another thing that makes me doubt if the agent is learning is that, while conducting another training session (image 2), the fluctuations in the average rewards are more noticeable. Even though Q0 still tends towards the maximum reward (20), in both training sessions, they continue to receive negative rewards (more than expected).
So it's difficult for me to determine if the agent is learning. If that's not the case, what should I modify? The reward? The agent's hyperparameters?
I would greatly appreciate your guidance.

请先登录，再回答此问题。

采纳的回答

Yatharth
Yatharth 2024-5-20
Hi Bryan,
It is very hard to pinpoint the exact reason for the sudden drops in episodic reward (sum of rewards at each step) without knowing anything about the environment or reward function. RL training is stochastic, so it is likely that the agent may be entering states in certain episodes that cause early termination or large penalties. This can obviously have a large impact on the cumulative reward. A suggestion would be to run a short training, save the agent information, and investigate whether the reward function is being evaluated correctly by the agents.
Fluctuations can be a sign that the agent is still exploring the environment. It's essential to balance exploration with exploitation. You might need to adjust parameters related to exploration.
Refer to the following documentation which provides further details about the training algorithm. https://www.mathworks.com/help/reinforcement-learning/ug/ddpg-agents.html

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Reinforcement Learning 的更多信息

标签

产品


版本

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by