Unable to generate RLAgent

2 次查看（过去 30 天）
Srivatsank
Srivatsank 2024-5-13
回答： Karan Singh 2024-6-18
Hi,
I am new to Reinforcement Learning Toolbox and its utilities. I have created a continuous Observation and Action Environment basis my requirements.
%% Code for Setting up Environment
% Author : Srivatsank P
% Created : 05/12/2024
% Edited : 05/13/2024
clear;
clc;
%% Setup Dynamics
% learned dynamics
model_path = 'doubleInt2D_net_longtime.mat';
% Load neural dynamics
load(model_path);
nnet = FCNReLUDyns([6, 5, 5, 4], 'state_idx', 1:4, 'control_idx', 5:6);
nnet = nnet.load_model_from_SeriesNetwork(net);
% Load black box agent
A = doubleInt2D_NN(nnet);
%% Create RL Environment
% States
ObservationInfo = rlNumericSpec([8 1]);
ObservationInfo.Name = 'DoubleInt2D_States';
ObservationInfo.Description = 'x,y,xdot,ydot,x_goal,y_goal,xdot_goal,ydot_goal';
ObservationInfo.LowerLimit = [A.min_x;A.min_x];
ObservationInfo.UpperLimit = [A.max_x;A.max_x];
% Control Variables
ActionInfo = rlNumericSpec([2 1]);
ActionInfo.Name = 'DoubleInt2D_Control';
ActionInfo.Description = 'u1,u2';
ActionInfo.LowerLimit = A.min_u;
ActionInfo.UpperLimit = A.max_u;
% Functions that define Initialization and Reward Calculation
reset_handle = @() reset_dynamics(A);
reward_handle = @(Action,LoggedSignals) dynamics_and_reward(Action,LoggedSignals,A);
%Environment
doubleInt2D_env = rlFunctionEnv(ObservationInfo,ActionInfo,reward_handle,reset_handle);
doubleInt2D_NN is an equivalent NN model I am using for the dynamics, the details of which I can't share unfortunately.
After this, I tried using "Reinforcement Learning Designer" to generate an agent. Agent details are shown below:
I run into the following error on my console:
Warning: Error occurred while executing the listener callback for event ButtonPushed defined for class matlab.ui.control.Button:
Error using dlnetwork/connectLayers (line 250)
Dot indexing is not supported for variables of this type.
Error in rl.util.default.createSingleChannelOutNet (line 12)
Net = connectLayers(Net,BridgeOutputName,OutputLayerName);
Error in rl.function.rlContinuousDeterministicActor.createDefault (line 200)
[actorNet,actionLayerName] = rl.util.default.createSingleChannelOutNet(inputGraph,bridgeOutputName,numOutput);
Error in rlTD3Agent (line 99)
Actor = rl.function.rlContinuousDeterministicActor.createDefault(ObservationInfo, ActionInfo, InitOptions);
Error in rl.util.createAgentFactory (line 16)
Agent = rlTD3Agent(Oinfo,Ainfo,AgentInitOpts);
Error in rl.util.createAgentFromEnvFactory (line 11)
Agent = rl.util.createAgentFactory(Type,Oinfo,Ainfo,AgentInitOpts);
Error in rl.internal.app.dialog.NewAgentFromEnvironmentDialog/createAgent (line 297)
Agent = rl.util.createAgentFromEnvFactory(AgentType,Env,AgentInitOpts);
Error in rl.internal.app.dialog.NewAgentFromEnvironmentDialog/callbackOK (line 274)
Agent = createAgent(obj);
Error in rl.internal.app.dialog.NewAgentFromEnvironmentDialog>@(es,ed)callbackOK(obj) (line 183)
addlistener(obj.OKButton,'ButtonPushed',@(es,ed) callbackOK(obj));
Error in appdesservices.internal.interfaces.model.AbstractModel/executeUserCallback (line 282)
notify(obj, matlabEventName, matlabEventData);
Error in matlab.ui.control.internal.controller.ComponentController/handleUserInteraction (line 442)
obj.Model.executeUserCallback(callbackInfo{:});
Error in matlab.ui.control.internal.controller.PushButtonController/handleEvent (line 95)
obj.handleUserInteraction('ButtonPushed', event.Data, {'ButtonPushed', eventData});
Error in appdesservices.internal.interfaces.controller.AbstractController>@(varargin)obj.handleEvent(varargin{:}) (line 214)
obj.Listeners = addlistener(obj.ViewModel, 'peerEvent', @obj.handleEvent);
Error in
viewmodel.internal.factory.ManagerFactoryProducer>@(src,event)callback(src,viewmodel.internal.factory.ManagerFactoryProducer.convertStructToEventData(event))
(line 79)
proxyCallback = @(src, event)callback(src, ...
> In appdesservices.internal.interfaces.model/AbstractModel/executeUserCallback (line 282)
In matlab.ui.control.internal.controller/ComponentController/handleUserInteraction (line 442)
In matlab.ui.control.internal.controller/PushButtonController/handleEvent (line 95)
In appdesservices.internal.interfaces.controller.AbstractController>@(varargin)obj.handleEvent(varargin{:}) (line 214)
In viewmodel.internal.factory.ManagerFactoryProducer>@(src,event)callback(src,viewmodel.internal.factory.ManagerFactoryProducer.convertStructToEventData(event)) (line 79)
>>
I run into the same error even with the default provided environments. I am currently on R2024a and have tried the same on Linux(ubuntu) and Windows 11. I do not know how to proceed and am unable to create agents. TIA for help!
  1 个评论
Srivatsank
Srivatsank 2024-5-13
Adding Further detail:
I also tried generating an agent in the m file using the folliwng commands:
% Number of Neurons per layer
initOpts = rlAgentInitializationOptions(NumHiddenUnit=256);
%Actor-Critic Agent as this can generate continuous policy
agent = rlACAgent(ObservationInfo,ActionInfo,initOpts)
I run into the following error:
Error using dlnetwork/connectLayers (line 250)
Dot indexing is not supported for variables of this type.
Error in rl.function.rlContinuousGaussianActor>localCreateMeanStdOutput (line 335)
net = connectLayers(net,bridgeOutputName,'fc_mean');
Error in rl.function.rlContinuousGaussianActor.createDefault (line 263)
[actorNet,meanName,stdName] = localCreateMeanStdOutput(inputGraph,bridgeOutputName,actionInfo,initOptions);
Error in rl.internal.util.parseOnPolicyAgentInitializationInputs (line 12)
actor = rl.function.rlContinuousGaussianActor.createDefault(observationInfo, actionInfo, useSquashInNetwork, initOptions);
Error in rlACAgent (line 87)
[Actor, Critic, AgentOptions] = rl.internal.util.parseOnPolicyAgentInitializationInputs(AgentType,varargin{:});
Error in trainRLAgent (line 59)
agent = rlACAgent(ObservationInfo,ActionInfo,initOpts)
This is why I tried using the "Reinforcement Learning Designer" app.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Karan Singh
Karan Singh 2024-6-18
Hi Srivatsank,
The code seems about right, but I believe the issue originates from the toolbox itself. The best advice would be to update the toolbox or rehash the toolbox. You can find information about this here: https://www.mathworks.com/help/matlab/matlab_env/toolbox-path-caching-in-the-matlab-program.html .
Also, I can suggest you one other way to create a reinforcement learning agent. Try creating the agent directly via the command line rather than the Reinforcement Learning Designer. Here's an example of creating a “TD3” agent:
% Define the observation and action spaces
ObservationInfo = rlNumericSpec([8 1], 'LowerLimit', [-inf; -inf; -inf; -inf; -inf; -inf; -inf; -inf], 'UpperLimit', [inf; inf; inf; inf; inf; inf; inf; inf]);
ObservationInfo.Name = 'observations';
ActionInfo = rlNumericSpec([2 1], 'LowerLimit', -1, 'UpperLimit', 1);
ActionInfo.Name = 'actions';
% Create the actor and critic networks
statePath = [
featureInputLayer(8,'Normalization','none','Name','state')
fullyConnectedLayer(24, 'Name', 'fc1')
reluLayer('Name','relu1')
fullyConnectedLayer(24, 'Name', 'fc2')
reluLayer('Name','relu2')
fullyConnectedLayer(2, 'Name', 'output')];
actionPath = [
featureInputLayer(2,'Normalization','none','Name','action')
fullyConnectedLayer(24, 'Name', 'fc1')
reluLayer('Name','relu1')
fullyConnectedLayer(24, 'Name', 'fc2')
reluLayer('Name','relu2')
fullyConnectedLayer(1, 'Name', 'output')];
criticPath = [
concatenationLayer(1,2,'Name','concat')
fullyConnectedLayer(24, 'Name', 'fc1')
reluLayer('Name','relu1')
fullyConnectedLayer(24, 'Name', 'fc2')
reluLayer('Name','relu2')
fullyConnectedLayer(1, 'Name', 'output')];
criticNetwork = layerGraph(criticPath);
criticNetwork = addLayers(criticNetwork,statePath);
criticNetwork = addLayers(criticNetwork,actionPath);
criticNetwork = connectLayers(criticNetwork,'state','concat/in1');
criticNetwork = connectLayers(criticNetwork,'action','concat/in2');
critic = rlQValueFunction(criticNetwork, ObservationInfo, ActionInfo, ...
'ObservationInputNames', 'state', 'ActionInputNames', 'action');
actorNetwork = layerGraph(statePath);
actor = rlDeterministicActorRepresentation(actorNetwork, ObservationInfo, ActionInfo, ...
'ObservationInputNames', 'state', 'ActionInputNames', 'output');
% Create the agent
agentOptions = rlTD3AgentOptions('SampleTime',1,'TargetSmoothFactor',1e-3);
agent = rlTD3Agent(actor,critic,agentOptions);
If nothing works out, then I suggest contacting MathWorks Technical Support: https://www.mathworks.com/support/contact_us.html
Thanks,
Karan

类别

在 Help Center 和 File Exchange 中查找有关 Agents 的更多信息

标签

产品


版本

R2024a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by