Main Content

rlDeterministicActorPolicy

Policy object to generate continuous deterministic actions for custom training loops and application deployment

Since R2022a

expand all in page

    Description

    This object implements a deterministic policy, which returns continuous deterministic actions given an input observation. You can create an rlDeterministicActorPolicy object from an rlContinuousDeterministicActor or extract it from an rlDDPGAgent or rlTD3Agent. You can then train the policy object using a custom training loop or deploy it for your application using generatePolicyBlock or generatePolicyFunction. This policy is always deterministic and does not perform any exploration. For more information on policies and value functions, see Create Policies and Value Functions.

    Creation

    Syntax

    policy = rlDeterministicActorPolicy(actor)

    Description

    policy = rlDeterministicActorPolicy(actor) creates the deterministic actor policy object policy from the continuous deterministic actor actor. It also sets the Actor property of policy to the input argument actor.

    example

    Properties

    expand all

    Continuous deterministic actor, specified as an rlContinuousDeterministicActor object.

    Normalization method, returned as an array in which each element (one for each input channel defined in the observationInfo and actionInfo properties, in that order) is one of the following values:

    • "none" — Do not normalize the input.

    • "rescale-zero-one" — Normalize the input by rescaling it to the interval between 0 and 1. The normalized input Y is (U–Min)./(UpperLimit–LowerLimit), where U is the nonnormalized input. Note that nonnormalized input values lower than LowerLimit result in normalized values lower than 0. Similarly, nonnormalized input values higher than UpperLimit result in normalized values higher than 1. Here, UpperLimit and LowerLimit are the corresponding properties defined in the specification object of the input channel.

    • "rescale-symmetric" — Normalize the input by rescaling it to the interval between –1 and 1. The normalized input Y is 2(U–LowerLimit)./(UpperLimit–LowerLimit) – 1, where U is the nonnormalized input. Note that nonnormalized input values lower than LowerLimit result in normalized values lower than –1. Similarly, nonnormalized input values higher than UpperLimit result in normalized values higher than 1. Here, UpperLimit and LowerLimit are the corresponding properties defined in the specification object of the input channel.

    Note

    When you specify the Normalization property of rlAgentInitializationOptions, normalization is applied only to the approximator input channels corresponding to rlNumericSpec specification objects in which both the UpperLimit and LowerLimit properties are defined. After you create the agent, you can use setNormalizer to assign normalizers that use any normalization method. For more information on normalizer objects, see rlNormalizer.

    Example: "rescale-symmetric"

    Observation specifications, returned as an rlFiniteSetSpec or rlNumericSpec object or an array containing a mix of such objects. Each element in the array defines the properties of an environment observation channel, such as its dimensions, data type, and name.

    Action specifications, returned as an rlNumericSpec object. This object defines the properties of the environment action channel, such as its dimensions, data type, and name.

    Note

    For this approximator object, only one action channel is allowed.

    Sample time of the policy, specified as a positive scalar or as -1.

    Within a MATLAB® environment, the policy is executed every time you call it within your custom training loop, so, SampleTime does not affect the timing of the policy execution.

    Within a Simulink® environment, the Policy block that uses the policy object executes every SampleTime seconds of simulation time. If SampleTime is -1 the block inherits the sample time from its input signals. Set SampleTime to -1 when the block is a child of an event-driven subsystem.

    Note

    Set SampleTime to a positive scalar when the block is not a child of an event-driven subsystem. Doing so ensures that the block executes at appropriate intervals when input signal sample times change due to model variations.

    Regardless of the type of environment, the time interval between consecutive elements in the output experience returned by sim is always SampleTime.

    If SampleTime is -1, for Simulink environments, the time interval between consecutive elements in the returned output experience reflects the timing of the events that trigger the Policy block execution, while for MATLAB environments, this time interval is considered equal to 1.

    Example: SampleTime=-1

    Object Functions

    generatePolicyBlockGenerate Simulink block that evaluates policy of an agent or policy object
    generatePolicyFunctionGenerate MATLAB function that evaluates policy of an agent or policy object
    getActionObtain action from agent, actor, or policy object given environment observations
    getLearnableParametersObtain learnable parameter values from agent, function approximator, or policy object
    resetReset environment, agent, experience buffer, or policy object
    setLearnableParametersSet learnable parameter values of agent, function approximator, or policy object

    Examples

    collapse all

    Open Live Script

    Create observation and action specification objects. For this example, define the observation and action spaces as continuous four- and two-dimensional spaces, respectively.

    obsInfo = rlNumericSpec([4 1]);
actInfo = rlNumericSpec([2 1]);

    Alternatively, use getObservationInfo and getActionInfo to extract the specification objects from an environment.

    Create a continuous deterministic actor. This actor must accept an observation as input and return an action as output.

    To approximate the policy function within the actor, use a deep neural network model. Define the network as an array of layer objects, and get the dimension of the observation and action spaces from the environment specification objects.

    layers = [ 
    featureInputLayer(obsInfo.Dimension(1))
    fullyConnectedLayer(16)
    reluLayer
    fullyConnectedLayer(actInfo.Dimension(1)) 
    ];

    Convert the network to a dlnetwork object and display the number of weights.

    model = dlnetwork(layers);
summary(model)
       Initialized: true

   Number of learnables: 114

   Inputs:
      1   'input'   4 features

    Create the actor using model, and the observation and action specifications.

    actor = rlContinuousDeterministicActor(model,obsInfo,actInfo)
    actor = 
  rlContinuousDeterministicActor with properties:

    ObservationInfo: [1x1 rl.util.rlNumericSpec]
         ActionInfo: [1x1 rl.util.rlNumericSpec]
      Normalization: "none"
          UseDevice: "cpu"
         Learnables: {4x1 cell}
              State: {0x1 cell}

    Check the actor with a random observation input.

    act = getAction(actor,{rand(obsInfo.Dimension)});
act{1}
    ans = 2x1 single column vector

    0.4013
    0.0578

    Create a policy object from actor.

    policy = rlDeterministicActorPolicy(actor)
    policy = 
  rlDeterministicActorPolicy with properties:

              Actor: [1x1 rl.function.rlContinuousDeterministicActor]
      Normalization: "none"
    ObservationInfo: [1x1 rl.util.rlNumericSpec]
         ActionInfo: [1x1 rl.util.rlNumericSpec]
         SampleTime: -1

    Check the policy with a random observation input.

    act = getAction(policy,{rand(obsInfo.Dimension)});
act{1}
    ans = 2×1

    0.4313
   -0.3002

    You can now train the policy with a custom training loop and then deploy it to your application.

    Version History

    Introduced in R2022a

    See Also

    Functions

    Objects

    Blocks

    Topics