rlFiniteSetSpec

For this example, consider the rlSimplePendulumModel Simulink® model. The model is a simple frictionless pendulum that initially hangs in a downward position.

Open the model.

mdl = "rlSimplePendulumModel";
open_system(mdl)

Create rlNumericSpec and rlFiniteSetSpec objects for the observation and action specifications, respectively.

The observation is a vector containing three signals: the sine, cosine, and time derivative of the angle.

obsInfo = rlNumericSpec([3 1]) 

obsInfo = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [3 1]
       DataType: "double"

The action is a scalar expressing the torque and can be one of three possible values, -2 Nm, 0 Nm and 2 Nm.

actInfo = rlFiniteSetSpec([-2 0 2])

actInfo = 
  rlFiniteSetSpec with properties:

       Elements: [3x1 double]
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 1]
       DataType: "double"

You can use dot notation to assign property values for the rlNumericSpec and rlFiniteSetSpec objects.

obsInfo.Name = "observations";
actInfo.Name = "torque";

Assign the agent block path information, and create the reinforcement learning environment for the Simulink model using the information extracted in the previous steps.

agentBlk = mdl + "/RL Agent";
env = rlSimulinkEnv(mdl,agentBlk,obsInfo,actInfo)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : []
  UseFastRestart : on

You can also specify a reset function using dot notation. For this example, randomly initialize theta0 in the model workspace.

env.ResetFcn = @(in) setVariable(in,"theta0",randn,"Workspace",mdl)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : @(in)setVariable(in,"theta0",randn,"Workspace",mdl)
  UseFastRestart : on

If your environment has an observation space consisting of multiple channels, some continuous, some discrete, use a vector of specification objects (each defining a single channel) to define the observation space.

For example, define an observation space as consisting of four channels. The first one carries a single number labeled 7, 9, 19 or -2. The second one carries a vector over a continuous three-dimensional space. The third channel carries a two by two matrix that can be either the zero matrix (that is zeros(2)) or the identity matrix (that is eye(2)). Finally, the fourth channel carries a continuous matrix with four rows and three columns.

obsInfo = [  rlFiniteSetSpec([7 9 19 -2])
             rlNumericSpec([3 1])
             rlFiniteSetSpec({zeros(2), eye(2)})
             rlNumericSpec([4 3]) ]

obsInfo=4×1 heterogeneous RLDataSpec (rlFiniteSetSpec, rlNumericSpec) array with properties:
    Name
    Description
    Dimension
    DataType

You can access each channel specification using dot notation.

obsInfo(1)

ans = 
  rlFiniteSetSpec with properties:

       Elements: [4x1 double]
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 1]
       DataType: "double"

obsInfo(2).Name = "Velocity";
obsInfo(2).Description = "Velocity vector in m/s in body reference frame";
obsInfo(2)

ans = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: "Velocity"
    Description: "Velocity vector in m/s in body reference frame"
      Dimension: [3 1]
       DataType: "double"

Within Reinforcement Learning Toolbox™ software, agents can only have one action channel. However, if your desired design involves multiple discrete action channels, you can convert them into a single action channel with a number of elements equal to the product of the number of elements of each action channel of your original design. Specifically, each element of the single action channel corresponds to a particular combination of elements in the action channels of your original design.

For example, suppose that the valid values for a two-output system are [1 2] for the first output and [10 20 30] for the second output. Create a discrete action space specification for all possible output combinations.

actionSpec = rlFiniteSetSpec({[1 10],[1 20],[1 30],...
                              [2 10],[2 20],[2 30]})

actionSpec = 
  rlFiniteSetSpec with properties:

       Elements: {6x1 cell}
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 2]
       DataType: "double"