使用PPO和TRPO​算法在reinfor​cement learning design app输出连续动作时，动作值不在设定好的区间内

9 次查看（过去 30 天）
宝
宝 2023-2-17
回答： Aiswarya 2024-2-27
%Open model
mdl='FCEV';
blk='FCEV/RL Agent';
%open_system(mdl);
%(s,a)
obsInfo = rlNumericSpec([3 1]);
obsInfo.Name = 'observations';
obsInfo.Description = 'A, B, and C';
numObservations = obsInfo.Dimension(1);
actInfo = rlNumericSpec([1 1],'LowerLimit',0,'UpperLimit',35);
actInfo.Name = 'Pbat_factor';
numActions = actInfo.Dimension(1);
%env
env = rlSimulinkEnv(mdl,blk,...
obsInfo,actInfo);
需要的动作区间是在0到35连续输出，但是在reinforcement learning design app中使用PPO或者TRPO算法输出的动作不在区间内，甚至一直输出负值。请问是网络结构设置有问题还是什么原因？

请先登录，再回答此问题。

回答（1 个）

Aiswarya
Aiswarya 2024-2-27
请注意，我将用英语回答这个问题。
(Please note that I will be answering the question in English.)
I understand that you are using the PPO/TRPO agent and trying to set the "LowerLimit" and "UpperLimit" of data space as "actInfo" using "rlNumericSpec" function. But the action output values are not in the range specified by you.
The action bounds depend on the type of agent. Both PPO and TRPO are on-policy agents and they don’t enforce constraints set in the action specification (specified using "rlNumericSpec") for continuous action spaces. If you want to enforce these limits you have to do it explicitly on the environment side.
You may refer to the "rlTRPOAgent" documentation which also mentions the fact that action bounds need to be set by the user within the environment:
https://www.mathworks.com/help/reinforcement-learning/ref/rl.agent.rltrpoagent.html
Also note that this is not the case with agents like SAC, for which the action bounds can be enforced with "rlNumericSpec".

类别

在 Help Center 和 File Exchange 中查找有关 Reinforcement Learning 的更多信息

标签

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!