How does the type of solver (fixed vs variable step) affect reinforcement learning in Simulink environment?

Question

Vincent 2025-8-13，17:10

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2179342-how-does-the-type-of-solver-fixed-vs-variable-step-affect-reinforcement-learning-in-simulink-envir

回答： Chuguang Pan 2025-8-14，2:14

So I successfully trained a robot manipulator to grab a object with a TD3 agent with the variable solver ode45. But when I changed the solver to a fixed solver (backward euler) with step size 0.01 (I also used rate transition block between the agent and the state,reward,done signal but the block read no operation since the agent and fixed solver was both at sample time = 0.01 so I'm assuming it had no affect) my agent performed worse and did not learn.

The reason why I used the rate transition block and changed to a fixed solver is because I'm trying to use a custom RL agent that I programmed on my own but I can't seem to use a variable solver otherwise the model doesn't compile when I try to train it. But it seems that fixed solver has negative effects on training.

Has anyone successfully trained a custom agent with a fixed solver or had similar issues? Any adivce or insight is greatly appreicated

3 个评论
显示 1更早的评论隐藏 1更早的评论

Vincent 2025-8-13，18:48

编辑：Vincent 2025-8-13，18:49

I'll to reduce it and see how that affects it. I also had a additional question on how the rate translation block works. Say for example my agent has a sample time of 0.1 and my fixed solver has a step size of 0.01. Say my agent takes in the state from the environment then outputs the action. Since my solver step size is 10 times smaller than the agent sample time., is it the case my solver will update 10 times and then on the 10th update pass the resulting state as the next state into my agent?

Torsten 2025-8-13，20:05

Since my solver step size is 10 times smaller than the agent sample time., is it the case my solver will update 10 times and then on the 10th update pass the resulting state as the next state into my agent?

I have no experience with Simulink. Maybe you should ask this as a new question in the forum and/or ask MATLAB Support.

请先登录，再进行评论。

请先登录，再回答此问题。