Hi Luca,
I understand that you want to know the possible reasons for the flat DDPG training curve despite varying the parameters and hyperparameters.
Following are the possible reasons for the a flat training curve:
- Reward Function: The reward function may not be providing meaningful gradients or sufficient learning signals to improve its policy.
- Normalization issues: Lack of normalization for states, actions, or rewards could lead to training instability.
- Gradient Problems:There could be vanishing or exploding gradients within the actor or critic networks.
- Reward Scaling: The rewards may not be scaled properly, leading to insignificant updates to the policy.
- Learning Rates: The learning rates for the actor and critic might be inappropriate, possibly too low or too high.
- Target Network Update Rate: The target networks for the actor and critic may not be updating at a suitable rate.
Following are the possible ways to diagnose and resolve the issue:
- Analyze Reward Function: Ensure the reward function provides a clear gradient for the agent to learn effectively.
- Hyperparameter Optimization: Experiment with different hyperparameters, including learning rates and the discount factor.
- Network Architecture Review: Check the actor and critic network architectures to ensure they are suitable for the complexity of the task.
- Agent Options: Try varying the parameters in "rlDDPGAgent" and "rlDDPGAgentOptions" .
- Target Update Method: Change the method of updating the target networks.
Please refer to the following documentation links to learn more about "DDPG Agents", "rlDDPGAgent" and "rlDDPGAgentOptions":
- https://in.mathworks.com/help/reinforcement-learning/ug/ddpg-agents.html
- https://in.mathworks.com/help/reinforcement-learning/ref/rl.agent.rlddpgagent.html
- https://www.mathworks.com/help/reinforcement-learning/ref/rl.option.rlddpgagentoptions.html
Additionally, you can find a related example in the documentation in the following link:
Hope this helps.
