Hello Gülin Sayal,
I understand that the model does not converge, which might be due to “Q₀”.
“Q₀” is the initial state-action value estimate computed from the target critic network at the beginning of each episode (typically at time step t=0).
Mathematically, for a state “s₀”, the agent computes:
Q₀ = max_a Q_target(s₀, a)
Where:
- Q_target is the target critic network, updated periodically.
- max_a implies it selects the maximum Q-value over all possible actions at that state.
If Q₀ diverges, it indicates instability in the value estimates that might happen due to:
- High learning rate
- Poor network structure
- Unnormalized input/output
- Unstable reward scale
- Improper target network update frequency
To resolve diverging Q₀ values and non-converging rewards in DQN training, you may refer to the steps mentioned below:
1. Scale rewards in the environment (e.g., divide by a constant) to keep them within a range like [-1, 1].
reward = rawReward / 100;
2. Use Double DQN to reduce Q-value overestimation.
agentOpts = rlDQNAgentOptions('UseDoubleDQN', true, ...);
You can refer to the following documentation to gain further understanding “rlDQNAgentOptions” function: https://www.mathworks.com/help/releases/R2021a/reinforcement-learning/ref/rldqnagentoptions.html
3. Add a tanhLayer after the output to bound Q-values between -1 and 1.
dnn = [
featureInputLayer(nI,'Name','state','Normalization','none')
fullyConnectedLayer(64,'Name','fc1')
reluLayer
fullyConnectedLayer(64,'Name','fc2')
reluLayer
fullyConnectedLayer(nO,'Name','output')
tanhLayer('Name','tanhOut')];
You can refer to the following documentation for more information on “tanhLayer” function: https://www.mathworks.com/help/releases/R2021a/deeplearning/ref/nnet.cnn.layer.tanhlayer.html
4. Reduce the critic's learning rate for stable updates.
criticOpts = rlRepresentationOptions('LearnRate',1e-4,'GradientThreshold',0.5);
5. Make target updates less frequent to reduce instability.
agentOpts.TargetUpdateFrequency = 20;
6. Use deeper networks to improve approximation capability.
% Replace 'nL = 24' with:
fullyConnectedLayer(64)
reluLayer
fullyConnectedLayer(64)
7. Ensure exploration reduces over time:
agentOpts.EpsilonGreedyExploration.Epsilon = 1;
agentOpts.EpsilonGreedyExploration.EpsilonMin = 0.1;
agentOpts.EpsilonGreedyExploration.EpsilonDecay = 0.01;
I hope this resolves the issue.
