getAction

Create an environment with a discrete action space, and obtain its observation and action specifications. For this example, load the environment used in the example Create DQN Agent Using Deep Network Designer and Train Using Image Observations.

% load predefined environment
env = rlPredefinedEnv("SimplePendulumWithImage-Discrete");

Obtain the observation and action specifications for this environment.

obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

Create a TRPO agent from the environment observation and action specifications.

agent = rlTRPOAgent(obsInfo,actInfo);

Use getAction to return the action from a random observation.

getAction(agent, ...
    {rand(obsInfo(1).Dimension), ...
     rand(obsInfo(2).Dimension)})

ans = 1x1 cell array
    {[-2]}

You can also obtain actions for a batch of observations. For example, obtain actions for a batch of 10 observations.

actBatch = getAction(agent, ...
    {rand([obsInfo(1).Dimension 10]), ...
     rand([obsInfo(2).Dimension 10])});
size(actBatch{1})

ans = 1×3

     1     1    10

actBatch{1}(1,1,7)

ans = 
-2

actBatch contains one action for each observation in the batch.

Create observation and action information. You can also obtain these specifications from an environment.

obsinfo = rlNumericSpec([4 1]);
actinfo = rlNumericSpec([2 1]);

Create a deep neural network for the actor.

net = [featureInputLayer(obsinfo.Dimension(1), ...
           "Normalization","none","Name","state")
       fullyConnectedLayer(10,"Name","fc1")
       reluLayer("Name","relu1")
       fullyConnectedLayer(20,"Name","CriticStateFC2")
       fullyConnectedLayer(actinfo.Dimension(1),"Name","fc2")
       tanhLayer("Name","tanh1")];
net = dlnetwork(net);

Create a deterministic actor representation for the network.

actor = rlContinuousDeterministicActor(net, ...
    obsinfo,actinfo,...
    ObservationInputNames={"state"});

Obtain an action from this actor for a random batch of 20 observations.

act = getAction(actor,{rand(4,1,10)})

act = 1x1 cell array
    {2x1x10 single}

act is a single cell array that contains the two computed actions for all 10 observations in the batch.

act{1}(:,1,7)

ans = 2x1 single column vector

    0.2643
   -0.2934

Create observation and action specification objects. For this example, define the observation and action spaces as continuous four- and two-dimensional spaces, respectively.

obsInfo = rlNumericSpec([4 1]);
actInfo = rlNumericSpec([2 1]);

Alternatively, you can use getObservationInfo and getActionInfo to extract the specification objects from an environment.

Create a continuous deterministic actor. This actor must accept an observation as input and return an action as output.

To approximate the policy function within the actor, use a recurrent deep neural network model. Define the network as an array of layer objects, and get the dimension of the observation and action spaces from the environment specification objects. To create a recurrent network, use a sequenceInputLayer as the input layer (with size equal to the number of dimensions of the observation channel) and include at least one lstmLayer.

layers = [ 
    sequenceInputLayer(obsInfo.Dimension(1))
    lstmLayer(2)
    reluLayer
    fullyConnectedLayer(actInfo.Dimension(1)) 
    ];

Convert the network to a dlnetwork object and display the number of weights.

model = dlnetwork(layers);
summary(model)

   Initialized: true

   Number of learnables: 62

   Inputs:
      1   'sequenceinput'   Sequence input with 4 dimensions

Create the actor using model, and the observation and action specifications.

actor = rlContinuousDeterministicActor(model,obsInfo,actInfo)

actor = 
  rlContinuousDeterministicActor with properties:

    ObservationInfo: [1x1 rl.util.rlNumericSpec]
         ActionInfo: [1x1 rl.util.rlNumericSpec]
      Normalization: "none"
          UseDevice: "cpu"
         Learnables: {5x1 cell}
              State: {2x1 cell}

Check the actor with a random observation input.

act = getAction(actor,{rand(obsInfo.Dimension)});
act{1}

ans = 2x1 single column vector

    0.0568
    0.0691

Create an additive noise policy object from actor.

policy = rlAdditiveNoisePolicy(actor)

policy = 
  rlAdditiveNoisePolicy with properties:

               Actor: [1x1 rl.function.rlContinuousDeterministicActor]
           NoiseType: "gaussian"
        NoiseOptions: [1x1 rl.option.GaussianActionNoise]
    EnableNoiseDecay: 1
       Normalization: "none"
      UseNoisyAction: 1
     ObservationInfo: [1x1 rl.util.rlNumericSpec]
          ActionInfo: [1x1 rl.util.rlNumericSpec]
          SampleTime: -1

Use dot notation to set the standard deviation decay rate.

policy.NoiseOptions.StandardDeviationDecayRate = 0.9;

Use getAction to generate an action from the policy, given a random observation input.

act = getAction(policy,{rand(obsInfo.Dimension)});
act{1}

ans = 2×1

    0.5922
   -0.3745

Display the state of the recurrent neural network in the policy object.

xNN = getRNNState(policy);
xNN{1}

ans = 2x1 single column vector

     0
     0

Use getAction to also return the updated policy as a second argument.

[act, updatedPolicy] = getAction(policy,{rand(obsInfo.Dimension)});

Display the state of the recurrent neural network in the updated policy object.

xpNN = getRNNState(updatedPolicy);
xpNN{1}

ans = 2x1 single column vector

    0.3327
   -0.2479

As expected, the state is updated.