Reinforcement Learning Toolbox- Multiple Discrete Actions for actor critic agent (imageInputLayer issues)

Question

Anthony 2019-9-26

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/482331-reinforcement-learning-toolbox-multiple-discrete-actions-for-actor-critic-agent-imageinputlayer-is

编辑： Huzaifah Shamim 2020-7-10

I am workig on setting up a rlACAgent using the reinforcement learning toolbox. I have succesffully created this agent with a system that has only one set of finite actions but I am looking to expand the set of finite actions to any arbitary number but in this case 4 and I think i am messing up something with the layer creation.

My code is below everything runs successfully until I try to create the Agent and I get the following error: "The dimensions of observations are not compatible with those of Observation Info."

I feel like I'm missing something fundamental about the layer construction here but I've been scratching my head for a while. Any help would be appreciated!

obsInfo = rlNumericSpec([2 1]);
obsInfo.Name = 'Car Position';
obsInfo.Description = {'x, y'};
% Actions
actInfo = rlFiniteSetSpec({[-1 -.8 -.6 -.4 -.2 0 .2 .4 .6 .8 1],...
 [-1 -.8 -.6 -.4 -.2 0 .2 .4 .6 .8 1],...
 [-1 -.8 -.6 -.4 -.2 0 .2 .4 .6 .8 1],...
 [-1 -.8 -.6 -.4 -.2 0 .2 .4 .6 .8 1]});
actInfo.Name='Wheel Speeds';
actInfo.Description = {'Front Right Speed','Front Left Speed','Rear Right Speed',...
 'Rear Left Speed'};
%% Build Custom Environment
env=rlFunctionEnv(obsInfo,actInfo,'DriveStepFunction','DriveResetFunction')
%% Extract Data from Environment
obsInfo = getObservationInfo(env)
numObservation = obsInfo.Dimension(1);
actInfo = getActionInfo(env)
numActions = actInfo.Dimension(2);
%% Develop Critic
criticNetwork = [
 imageInputLayer([numObservation numActions 1],'Normalization','none','Name','state')
 fullyConnectedLayer(numObservation,'Name','CriticFC')];
criticOpts = rlRepresentationOptions('LearnRate',.01,'GradientThreshold',1);
critic = rlRepresentation(criticNetwork,obsInfo,'Observation',{'state'},criticOpts);
%% Develop Actor
actorNetwork = [
 imageInputLayer([numObservation numActions 1],'Normalization','none','Name','state')
 fullyConnectedLayer(numActions,'Name','action')];
actorOpts = rlRepresentationOptions('LearnRate',.01,'GradientThreshold',1);
actor = rlRepresentation(actorNetwork,obsInfo,actInfo,...
 'Observation',{'state'},'Action',{'action'},actorOpts);
%% Develop Agent
agentOpts = rlACAgentOptions(...
 'NumStepsToLookAhead',5,...
 'DiscountFactor',1,...
 'EntropyLossWeight',.4);
agent = rlACAgent(actor,critic,agentOpts);

3 个评论
显示 1更早的评论隐藏 1更早的评论

Anthony 2020-7-9

I never managed to get this to work with an Actor Critic Agent but I switched to a DQN and got it to work

Huzaifah Shamim 2020-7-10

编辑：Huzaifah Shamim 2020-7-10

Ah nice. For the DQN network, did you set up with observation and action input or just observation?

could i take a look at ur environment and how you set up certain things?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Emmanouil Tzorakoleftherakis 2019-10-4

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/482331-reinforcement-learning-toolbox-multiple-discrete-actions-for-actor-critic-agent-imageinputlayer-is#answer_394833

Hi Anthony,

I believe this link should help. Looks like the action space is not set up correctly. For multiple discrete actions, you need to calculate all possible combinations of discrete actions, and use these with rlFiniteSetSpec.

2 个评论
显示无隐藏无

Anthony 2019-10-4

在 MATLAB Online 中打开

Hi Emmanouil,

Thanks for the help! I fixed the way I am representing my action space with the following code:

vectors = { [-1 -.8 -.6 -.4 0 .4 .6 .8 1]', ...
    [-1 -.8 -.6 -.4 0 .4 .6 .8 1]', ...
    [-1 -.8 -.6 -.4 0 .4 .6 .8 1]', ...
    [-1 -.8 -.6 -.4 0 .4 .6 .8 1]'};
%input data: cell array of vectors
n = numel(vectors); % number of vectors
combs = cell(1,n); % pre-define to generate comma-separated list
[combs{end:-1:1}] = ndgrid(vectors{end:-1:1}); % the reverse order in these two
% comma-separated lists is needed to produce the rows of the result matrix
combs = cat(n+1, combs{:}); %concat the n n-dim arrays along dimension n+1
combs = reshape(combs,[],n);%reshape to obtain desired matrix
combs=combs';
% Actions
actInfo = rlFiniteSetSpec(num2cell(combs,1));

However, I am still having the same issue and getting the error:

"The dimensions of observations are not compatible with those of Observation Info."

Emmanouil Tzorakoleftherakis 2019-10-5

编辑：Emmanouil Tzorakoleftherakis 2019-10-5

在 MATLAB Online 中打开

I don't know the specifics of your environment, but the input and output dimensions of the actor and the critic are not set up properly. For instance, for the critic you need to have 1 output (since this is a number) and the input would be determined by the number of observations.

criticNetwork = [
 imageInputLayer([numObservation 1 1],'Normalization','none','Name','state')
 fullyConnectedLayer(1,'Name','CriticFC')];

Along the same lines, total number of actions is 6561 (number of possible combinations of your discrete inputs) so your actor input would be the same as the critic network and the output would be 6561

actorNetwork = [
 imageInputLayer([numObservation 1 1],'Normalization','none','Name','state')
 fullyConnectedLayer(6561,'Name','action')];

This example should be helpful to get an idea how to set up these dimensions.

请先登录，再进行评论。

Reinforcement Learning Toolbox- Multiple Discrete Actions for actor critic agent (imageInputLayer issues)

3 个评论
显示 1更早的评论隐藏 1更早的评论

采纳的回答

2 个评论
显示无隐藏无

更多回答（0 个）

另请参阅

类别

标签

Community Treasure Hunt

Reinforcement Learning Toolbox- Multiple Discrete Actions for actor critic agent (imageInputLayer issues)

3 个评论 显示 1更早的评论隐藏 1更早的评论

采纳的回答

2 个评论 显示 无隐藏 无

更多回答（0 个）

另请参阅

类别

标签

Community Treasure Hunt

3 个评论
显示 1更早的评论隐藏 1更早的评论

2 个评论
显示无隐藏无