I want to print out multiple actions in reinforcement learning

Question

SEONGJAE SHIN 2023-7-6

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1992378-i-want-to-print-out-multiple-actions-in-reinforcement-learning

回答： Emmanouil Tzorakoleftherakis 2023-7-11

this is agent code

function agent = createAgent(observationInfo,actionInfo,Ts)
L = 16; % number of neurons
statePath = [
    featureInputLayer(observationInfo.Dimension(1),'Normalization','none','Name','observation')
    %fullyConnectedLayer(L,'Name','fc1')
    %reluLayer('Name','relu1')
    fullyConnectedLayer(L,'Name','fc2')
    additionLayer(2,'Name','add')
    reluLayer('Name','relu2')
    %fullyConnectedLayer(L,'Name','fc3')
    %reluLayer('Name','relu3')
    fullyConnectedLayer(1,'Name','fc4')];
actionPath = [
    featureInputLayer(actionInfo.Dimension(1),'Normalization','none','Name','action')
    fullyConnectedLayer(L, 'Name', 'fc5')];
criticNetwork = layerGraph(statePath);
criticNetwork = addLayers(criticNetwork, actionPath);
criticNetwork = connectLayers(criticNetwork,'fc5','add/in2');
criticNetwork = dlnetwork(criticNetwork);
 
criticOptions = rlOptimizerOptions('LearnRate',1e-3,'GradientThreshold',1);
critic = rlQValueFunction(criticNetwork,observationInfo,actionInfo,...
    'ObservationInputNames','observation','ActionInputNames','action');
actorNetwork = [
    featureInputLayer(observationInfo.Dimension(1),'Normalization','none','Name','observation')
    fullyConnectedLayer(L,'Name','fc1')
    reluLayer('Name','relu1')
    %fullyConnectedLayer(L,'Name','fc2')
    %reluLayer('Name','relu2')
    %fullyConnectedLayer(L,'Name','fc3')
    %reluLayer('Name','relu3')
    fullyConnectedLayer(1,'Name','fc4')
    tanhLayer('Name','tanh1')
    scalingLayer('Name','ActorScaling1','Scale',2.5,'Bias',-0.5)];
actorNetwork = dlnetwork(actorNetwork);
actorOptions = rlOptimizerOptions('LearnRate',1e-3,'GradientThreshold', 1);
actor = rlContinuousDeterministicActor(actorNetwork,observationInfo,actionInfo);
agentOptions = rlDDPGAgentOptions(...
    'SampleTime',Ts,...
    'CriticOptimizerOptions',criticOptions,...
    'ActorOptimizerOptions',actorOptions,...
    'ExperienceBufferLength',1e6);
agentOptions.NoiseOptions.Variance = 0.6;
agentOptions.NoiseOptions.VarianceDecayRate = 1e-5;
agent = rlDDPGAgent(actor,critic,agentOptions);

This is run code

%%
clear;
clc;
%% Create Environment
% load environment parameters
parameters
mdl = 'my_CAV_slx';
% open_system(mdl);
obsInfo = rlNumericSpec([10  1]);
actInfo = rlNumericSpec([1 1], 'LowerLimit', accMin, 'UpperLimit', accMax);
blks = mdl + "/RL Agent1";
env = rlSimulinkEnv(mdl, blks, obsInfo, actInfo);
% set reset function
env.ResetFcn = @(in)ResetParamFcn(in);
%% Create Agents
% fix seed
rng(0)
% longitudinal control
agent1 = createAgent(obsInfo, actInfo, Ts);
%% Train Agents
trainingOpts = rlTrainingOptions('MaxEpisodes',maxEpisodes,'MaxStepsPerEpisode',maxSteps,'Verbose',false,'Plots','training-progress','StopTrainingCriteria','AverageReward');
doTraining = true;
if doTraining
    % Train the agent.
    trainingStats = train(agent1, env, trainingOpts);
    save('my_CAV.mat', 'agent1')
else
    % Load pretrained agents for the example.
    load('my_CAV.mat')
end
%% Simulate Agents
simOptions = rlSimulationOptions('MaxSteps',maxSteps);
experience = sim(env, agent1, simOptions);

2 个评论
显示无隐藏无

Emmanouil Tzorakoleftherakis 2023-7-10

I am not sure what you mean by print out. Do you want your agent to output multiple actions? Or do you want to know how to do inference with the agent you created above and see what the output action is?

SEONGJAE SHIN 2023-7-11

Thank you for comment. I want to output multiple actions.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Emmanouil Tzorakoleftherakis 2023-7-11

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1992378-i-want-to-print-out-multiple-actions-in-reinforcement-learning#answer_1271198

Hi,

If you want to create an agent that outputs multiple actions, you need to make sure the actor network is set up accordingly. I would recommend using the 'default agent' feature as shown here. In summary, this feature creates a default network architecture for your actor and critic based on the observation and action spaces you provide. If your action space indicates that you have multiple actions, the actor and critic that will be created will reflect that. You can use this shortcut as a way to get an initial design which you can then tweak as needed.

The link I shared above also shows you how to perform inference on the actor/agent you already created using 'getAction'.

Hope that helps

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

I want to print out multiple actions in reinforcement learning

2 个评论
显示无隐藏无

回答（1 个）

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

I want to print out multiple actions in reinforcement learning

2 个评论 显示 无隐藏 无

回答（1 个）

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

2 个评论
显示无隐藏无

0 个评论
显示 -2更早的评论隐藏 -2更早的评论