Replace RL type (PPO with DPPG) in a Matlab example

Question

ali farid 2023-6-21

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1986144-replace-rl-type-ppo-with-dppg-in-a-matlab-example

编辑： ali farid 2023-6-28

There is a Matlab example about coverage path planning using PPO reinforcement learning in the following link:

https://www.mathworks.com/help/reinforcement-learning/ug/train-3-agents-for-area-coverage.html

I think the environment is fine and I only need to check the parts that there is a PPO. I am trying to replace PPO with DDPG, with the following codes

opt = rlDDPGAgentOptions(...

ActorOptimizerOptions=actorOpts,...

CriticOptimizerOptions=criticOpts,....

MiniBatchSize=64,...

SampleTime=Ts,...

DiscountFactor=0.995);

agentA = rlDDPGAgent(actor(1),critic(1),opt);

agentB = rlDDPGAgent(actor(2),critic(2),opt);

agentC = rlDDPGAgent(actor(3),critic(3),opt);

but there is error: First argument must be a rlDeterministicActorRepresentation object or an observation specification created using 'rlNumericSpec' or 'rlFiniteSetSpec' objects.

Do you have any idea?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Emmanouil Tzorakoleftherakis 2023-6-27

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1986144-replace-rl-type-ppo-with-dppg-in-a-matlab-example#answer_1263543

PPO is a stochastic agent whereas DDPG is deterministic. This means that you cannot just use actors and critics designed for PPO with DDPG and vice versa. Your best bet is to either recreate those neural nets or use the default agent feature to get an initial architecture you can iterate upon.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Emmanouil Tzorakoleftherakis 2023-6-27

I would start here and then take a look at the examples and how we set up the neural networks for each scenario

ali farid 2023-6-28

编辑：ali farid 2023-6-28

在 MATLAB Online 中打开

Thanks for your guidance. I recreated those neural nets but still there is error: matlab model input sizes must match the dimensions specified in the corresponding observation and action info specification

for idx = 1:3
    % Create actor deep neural network.
    actorNetwork = [
        imageInputLayer(obsSize,Normalization="none")
        convolution2dLayer(8,16, ...
            Stride=1,Padding=1,WeightsInitializer="he")
        reluLayer
        convolution2dLayer(4,8, ...
            Stride=1,Padding="same",WeightsInitializer="he")
        reluLayer
        fullyConnectedLayer(256,WeightsInitializer="he")
        reluLayer
        fullyConnectedLayer(128,WeightsInitializer="he")
        reluLayer
        fullyConnectedLayer(64,WeightsInitializer="he")
        reluLayer
        fullyConnectedLayer(numAct)
        softmaxLayer];
    actorNetwork = dlnetwork(actorNetwork);
    
    % Create critic deep neural network.
    statePath = [...
        featureInputLayer(prod(oinfo.Dimension),Name="NetObsInLayer")
        fullyConnectedLayer(128)
        reluLayer
        fullyConnectedLayer(200,Name="sPathOut")];
    actionPath = [
        featureInputLayer(prod(ainfo.Dimension),Name="NetActInLayer")
        fullyConnectedLayer(200,Name="aPathOut",BiasLearnRateFactor=0)];
    commonPath = [
        additionLayer(2,Name="add")
        reluLayer
        fullyConnectedLayer(1,Name="CriticOutput")];
    % Create layerGraph object and add layers
    criticNetwork = layerGraph(statePath);
    criticNetwork = addLayers(criticNetwork,actionPath);
    criticNetwork = addLayers(criticNetwork,commonPath);
    % Connect paths and convert to dlnetwork object
    criticNetwork = connectLayers(criticNetwork,"sPathOut","add/in1");
    criticNetwork = connectLayers(criticNetwork,"aPathOut","add/in2");
    criticNetwork = dlnetwork(criticNetwork);
    % create actor and critic
    actor(idx) = rlDiscreteCategoricalActor(actorNetwork,oinfo,ainfo);%(actorNetwork,oinfo,ainfo); %#ok<*SAGROW> 
    critic(idx) =  rlQValueFunction(criticNetwork, ...
    oinfo,ainfo)
end