Error in rl.agent.AbstractAgent/train while executing example "Train Reinforcement Learning Agent in MDP Environment"

Question

David Carrasco Martínez 2022-6-16

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1741780-error-in-rl-agent-abstractagent-train-while-executing-example-train-reinforcement-learning-agent-in

编辑： MULI 2024-11-18，4:39

Hi,

I'm trying to run an example from the Reinforcement learning Toolbox examples, but the function train() does not work correctly. The example is "Train Reinforcement Learning Agent in MDP Environment". I have only changed the value of doTraining to true to be able to execute that part of the code.

This is the code of the example:

MDP = createMDP(8,["up";"down"]);

% State 1 transition and reward

MDP.T(1,2,1) = 1;

MDP.R(1,2,1) = 3;

MDP.T(1,3,2) = 1;

MDP.R(1,3,2) = 1;

% State 2 transition and reward

MDP.T(2,4,1) = 1;

MDP.R(2,4,1) = 2;

MDP.T(2,5,2) = 1;

MDP.R(2,5,2) = 1;

% State 3 transition and reward

MDP.T(3,5,1) = 1;

MDP.R(3,5,1) = 2;

MDP.T(3,6,2) = 1;

MDP.R(3,6,2) = 4;

% State 4 transition and reward

MDP.T(4,7,1) = 1;

MDP.R(4,7,1) = 3;

MDP.T(4,8,2) = 1;

MDP.R(4,8,2) = 2;

% State 5 transition and reward

MDP.T(5,7,1) = 1;

MDP.R(5,7,1) = 1;

MDP.T(5,8,2) = 1;

MDP.R(5,8,2) = 9;

% State 6 transition and reward

MDP.T(6,7,1) = 1;

MDP.R(6,7,1) = 5;

MDP.T(6,8,2) = 1;

MDP.R(6,8,2) = 1;

% State 7 transition and reward

MDP.T(7,7,1) = 1;

MDP.R(7,7,1) = 0;

MDP.T(7,7,2) = 1;

MDP.R(7,7,2) = 0;

% State 8 transition and reward

MDP.T(8,8,1) = 1;

MDP.R(8,8,1) = 0;

MDP.T(8,8,2) = 1;

MDP.R(8,8,2) = 0;

MDP.TerminalStates = ["s7";"s8"];

env = rlMDPEnv(MDP);

env.ResetFcn = @() 1;

rng(0);

obsInfo = getObservationInfo(env);

actInfo = getActionInfo(env);

qTable = rlTable(obsInfo, actInfo);

qFunction = rlQValueFunction(qTable, obsInfo, actInfo);

qOptions = rlOptimizerOptions("LearnRate",1);

agentOpts = rlQAgentOptions;

agentOpts.DiscountFactor = 1;

agentOpts.EpsilonGreedyExploration.Epsilon = 0.9;

agentOpts.EpsilonGreedyExploration.EpsilonDecay = 0.01;

agentOpts.CriticOptimizerOptions = qOptions;

qAgent = rlQAgent(qFunction,agentOpts); %#ok<NASGU>

trainOpts = rlTrainingOptions;

trainOpts.MaxStepsPerEpisode = 50;

trainOpts.MaxEpisodes = 500;

trainOpts.StopTrainingCriteria = "AverageReward";

trainOpts.StopTrainingValue = 13;

trainOpts.ScoreAveragingWindowLength = 30;

doTraining = true;

if doTraining

% Train the agent.

trainingStats = train(qAgent,env,trainOpts); %#ok<UNRCH>

else

% Load pretrained agent for the example.

load('genericMDPQAgent.mat','qAgent');

end

Data = sim(qAgent,env);

And this is the error:

Error using rl.train.SeriesTrainer/run

There was an error executing the ProcessExperienceFcn.

Caused by:

Error using rl.function.AbstractFunction/gradient

Unable to compute gradient from function model.

Error in rl.agent.rlQAgent/learn_ (line 228)

CriticGradient = gradient(this.Critic_,lossFcn,...

Error in rl.agent.AbstractAgent/learn (line 29)

this = learn_(this,experience);

Error in rl.util.agentProcessStepExperience (line 6)

learn(Agent,Exp);

Error in rl.env.internal.FunctionHandlePolicyExperienceProcessor/processExperience_ (line 31)

[this.Policy_,this.Data_] = feval(this.Fcn_,...

Error in rl.env.internal.ExperienceProcessorInterface/processExperienceInternal_ (line 137)

processExperience_(this,experience,getEpisodeInfoData(this));

Error in rl.env.internal.ExperienceProcessorInterface/processExperience (line 78)

stopsim = processExperienceInternal_(this,experience,simTime);

Error in rl.env.internal.MATLABSimulator/simInternal_ (line 128)

stopsim = processExperience(expProcessor,exp,i*ts);

Error in rl.env.internal.MATLABSimulator/sim_ (line 67)

out = simInternal_(this,simPkg);

Error in rl.env.internal.AbstractSimulator/sim (line 30)

out = sim_(this,simData,policy,processExpFcn,processExpData);

Error in rl.env.AbstractEnv/runEpisode (line 144)

out = sim(simulator,simData,policy,processExpFcn,processExpData);

Error in rl.train.SeriesTrainer/run (line 32)

out = runEpisode(...

Error in rl.train.TrainingManager/train (line 429)

run(trainer);

Error in rl.train.TrainingManager/run (line 218)

train(this);

Error in rl.agent.AbstractAgent/train (line 83)

trainingResult = run(trainMgr,checkpoint);

Error in test_IA (line 76)

trainingStats = train(qAgent,env,trainOpts); %#ok<UNRCH>

Caused by:

Undefined function 'struct2array' for input arguments of type 'struct'.

Error in rl.train.TrainingManager/train (line 429)

run(trainer);

Error in rl.train.TrainingManager/run (line 218)

train(this);

Error in rl.agent.AbstractAgent/train (line 83)

trainingResult = run(trainMgr,checkpoint);

Error in test_IA (line 76)

trainingStats = train(qAgent,env,trainOpts); %#ok<UNRCH>

2 个评论
显示无隐藏无

Gunawan 2023-5-4

Any solution for this problem? I also have the same error.

David Carrasco Martínez 2023-5-4

I think the train function only works in Matlab Online. I dont know why, but the example finally worked in the online version.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Yichen 2022-10-14

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1741780-error-in-rl-agent-abstractagent-train-while-executing-example-train-reinforcement-learning-agent-in#answer_1074633

I meet the same problem !

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Gunawan 2023-5-4

Hello, have you resolved this error? how to overcome this error? thanks.

请先登录，再进行评论。

Answer 2

MULI 2024-11-18，4:39

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1741780-error-in-rl-agent-abstractagent-train-while-executing-example-train-reinforcement-learning-agent-in#answer_1546173

编辑：MULI 2024-11-18，4:39

在 MATLAB Online 中打开

Hi @David Carrasco Martínez,

I understand that you are facing an issue when “doTraining” flag is set to “true” in "Train Reinforcement Learning Agent in MDP Environment" example.

You can try the below suggestions to resolve this issue:

Check whether you have “signal processing toolbox” and “DSP system toolbox” installed.
Also Check if the“struct2array”function is available in your MATLAB installation. You can do this by typing:

>>which struct2array

You can refer to the below MATLAB answer to get some additional insights on this issue:

https://www.mathworks.com/matlabcentral/answers/493612-any-fixes-for-undefined-function-or-variable-struct2array

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Error in rl.agent.AbstractAgent/train while executing example "Train Reinforcement Learning Agent in MDP Environment"

2 个评论
显示无隐藏无

回答（2 个）

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Error in rl.agent.A​bstractAge​nt/train while executing example "Train Reinforcement Learning Agent in MDP Environment"

2 个评论 显示 无隐藏 无

回答（2 个）

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Error in rl.agent.AbstractAgent/train while executing example "Train Reinforcement Learning Agent in MDP Environment"

2 个评论
显示无隐藏无

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论