some of the saved agents in DQN reinforcement learning algorithm do not reproduce the training rewards

7 次查看（过去 30 天）
masoud k
masoud k 2020-9-21
评论： Abdul Basith Ashraf 2021-4-3
Hello everyone,
I am training a DQN agent in my reinforcement learning problem in Simulink Matlab 2019a. I have set a save criteria of agents with episode rewards equal to 6. My problem is that after a number of episodes, some agents will save based on my save criteria, but when I use them, some of the agents do not reproduce rewards equal to 6 or more but the amount of rewards is less than 6. If anyone could help me I will appriciate.
  2 个评论
Madhav Thakker
Madhav Thakker 2020-9-24
编辑：Madhav Thakker 2020-9-24
Hi masoud,
Do you save the agent as soon as you get an episode with reward 6? Are you using an epsilon-greedy policy to train the agents? How less are the rewards when you use the saved agent?
masoud k
masoud k 2020-9-24
Hi Madhav,
The criteria for saving the candidate agents is getting reward of 6 or more of an episode.
I use epsilon-greedy policy to train agents with epsilon equal to 0.6 and the decay rate of 0.001.
Some of saved agents come up with an episode reward around 1 and some other with 6 or more.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Emmanouil Tzorakoleftherakis
This is a common misconception. When you train an agent and you get a certain reward out of a specific episode, if you were to stop training and run the same episode again with the same agent you would most likely get a different response from the agent and thus a different episode reward.
There are a few reasons for that:
1) During training, the agent explores different options based on some probabilistic exploration strategy. For DQN this strategy is related to the epsilon parameter in DQN options. After training, the agent does not explore anymore and only relies on the underlying neural network for inference, so it makes sense that results are different.
2) It is best practice to randomize various elements of the environment during training to get a more robust policy. In that case there will also be differences in how the environment behaves during and after training, so agent will respond differently as well.
3) Some agents are stochastic. This by itself implies different decisions/behavior even if everything else in the environment remains deterministic.
Hope that helps
  3 个评论
显示 1更早的评论
Emmanouil Tzorakoleftherakis
Emmanouil Tzorakoleftherakis 2020-9-24
Typically, exploration slows down over time and the agent starts exploiting more. For example, you can see how epsilon decays for DQN here. If you start getting consistently good rewards at the later stage of training where exploration is low, then that's a good sign that this behavior will carry over after training. You may still not get exactly the same rewards (especially if the minimum epsilon value is nonzero) but you should overall get desirable performance.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Reinforcement Learning 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by