Change TD3 actor network layers

Question

YQ 2022-3-20

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1675749-change-td3-actor-network-layers

回答： Shubham 2024-1-15

Hi all, I am currently using Reinforcement Learning Designer app and I am using the pre-loaded TD3 algorithm. However, I wish to remove certain layers from the actor network.

I first export the agent from the RL Designer app to my workspace, edit the actor network using the Deep Network Designer app, and then export the new network as 'layers_1'. However, I am getting the following error after excuting the last section of the code:

"Action names must match the names of the deep neural network's input or output layers. Make sure all

action names appear in the neural network"

I used this method to add more layers to the actor network before but now I can't seem to remove them.

Is there a way to remove layers from the actor network and import it back into the RL designer app?

Thanks

%%
numObs = 13; %number of observations
observationInfo = rlNumericSpec([numObs 1]); %create observation vector
numAct = 4; %number of output actions
actionInfo = rlNumericSpec([numAct 1], 'LowerLimit', -10, 'UpperLimit', 10 ); %create action vector
mdl = 'arm10';
load_system(mdl);
blk = [mdl,'/RL Agent'];
env = rlSimulinkEnv(mdl,blk,observationInfo,actionInfo);
%open RL designer app
%% After exporting agent from RL designer app to workspace
actor = getActor(agent1)
actorNet = getModel(actor)
plot(layerGraph(actorNet))
deepNetworkDesigner(actorNet) %design using deep network designer
%%
actor = setModel(actor, layers_1);
agent1 = setActor(agent1,actor);

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Shubham 2024-1-15

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1675749-change-td3-actor-network-layers#answer_1389686

在 MATLAB Online 中打开

Hi YQ,

The error message you're encountering indicates that the actor network you're attempting to import back into the Reinforcement Learning (RL) agent does not have the correct action names that match the expected input or output layer names of the network. When you modify the deep neural network for the actor, you must ensure that the names of the input and output layers align with the observation and action specifications, respectively.

Here's a step-by-step approach to ensure that you correctly modify the actor network and import it back into the RL Designer app:

Export the agent from the RL Designer app to the workspace.
Retrieve the actor network from the agent.
Modify the actor network using the Deep Network Designer app.
Make sure the input layer name matches the observation specification name and the output layer name matches the action specification name.
Export the modified network as layers_1.
Create a new actor representation with the modified network and the original action information.
Set the new actor representation back into the agent.
Import the modified agent back into the RL Designer app.

Here's how you can implement this in MATLAB code:

% After exporting agent from RL designer app to workspace
actor = getActor(agent1);
actorNet = getModel(actor);
% Modify the actor network using the Deep Network Designer app
deepNetworkDesigner(actorNet);
% After modifying the actor network in the Deep Network Designer app
% and exporting the network as 'layers_1'
% Ensure the input and output layer names match the observation and action specifications
inputLayerName = 'input'; % This should match the name in layers_1
outputLayerName = 'output'; % This should match the name in layers_1
% Create a new actor representation with the modified network
actorOptions = rlRepresentationOptions('LearnRate',1e-03,'GradientThreshold',1);
newActor = rlDeterministicActorRepresentation(layers_1, observationInfo, actionInfo, ...
    'Observation', {inputLayerName}, 'Action', {outputLayerName}, actorOptions);
% Set the new actor representation back into the agent
agent1 = setActor(agent1, newActor);
% Now you can import the modified agent back into the RL Designer app

Please replace 'input' and 'output' with the actual names of the input and output layers in your modified network layers_1. The names must match exactly, including case sensitivity.

After modifying the network in the Deep Network Designer app, you should not only export the layers but also ensure that the input and output layer names are consistent with the original network's naming convention. If you change the structure of the network, it's crucial to preserve the names of the input and output layers or update them accordingly in the code where you create the new actor representation.