Network Input and Output Dimensions

The dimensions of the network input and output layers for your actor and critic must match the dimension of the corresponding environment observation and action channels, respectively. To obtain the action and observation specifications from the environment env, use the getActionInfo and getObservationInfo functions, respectively.

actInfo = getActionInfo(env);
obsInfo = getObservationInfo(env);

Access the Dimensions property of each channel. For example, get the size of the first environment and action channel:

actSize = actInfo(1).Dimensions;
obsSize = obsInfo(1).Dimensions;

In general actSize and obsSize are row vectors whose elements are the lengths of the corresponding dimensions. For example, if the first observation channel is a 256-by-256 RGB image, actSize is the vector [256 256 3]. To calculate the total number of dimension of the channel, use prod. For example, assuming the environment has only one observation channel:

obsDimensions = prod(obsInfo.Dimensions);

For rlVectorQValueFunction critics and rlDiscreteCategoricalActor actors, you need to obtain the number of possible elements of the action set. You can do so by accessing the Elements property of the action channel. For example, assuming the environment has only one action channel:

actNumElements = numel(actInfo.Elements);

Networks for value function critics (such as the ones used in AC, PG, PPO or TRPO agents) must take only observations as inputs and must have a single scalar output. For these networks, the dimensions of the input layers must match the dimensions of the environment observation channels. For more information, see rlValueFunction.

Networks for single-output Q-value function critics (such as the ones used in Q, DQN, SARSA, DDPG, TD3, and SAC agents) must take both observations and actions as inputs, and must have a single scalar output. For these networks, the dimensions of the input layers must match the dimensions of the environment channels for both observations and actions. For more information, see rlQValueFunction.

Networks for multi-output Q-value function critics (such as those used in Q, DQN, and SARSA agents) take only observations as inputs and must have a single output layer with output size equal to the number of possible discrete actions. For these networks the dimensions of the input layers must match the dimensions of the environment observations channels. For more information, see rlVectorQValueFunction.

For actor networks, the dimensions of the input layers must match the dimensions of the environment observation channels and the dimension of the output layer must be as follows.

Deep Neural Networks

Deep neural networks consist of a series of interconnected layers. You can specify a deep neural network as one of the following:

Typically, you build your neural network by stacking together a number of layers in an array of Layer objects, possibly adding these arrays to a layerGraph object, and then converting the final result to a dlnetwork object.

For agents that need multiple input or output layers, you create an array of Layer objects for each input path (observations or actions) and for each output path (estimated rewards or actions). You then add these arrays to a layerGraph object and connect them paths together using the connectLayers function.

You can also create your deep neural network using the Deep Network Designer app. For an example, see Create DQN Agent Using Deep Network Designer and Train Using Image Observations.

The following table lists some common deep learning layers used in reinforcement learning applications. For a full list of available layers, see List of Deep Learning Layers.

The Reinforcement Learning Toolbox software provides the following layers, which contain no tunable parameters (that is, parameters that change during training).

You can also create your own custom layers. For more information, see Define Custom Deep Learning Layers.

When you create a deep neural network, it is good practice to specify names for the first layer of each input path and the final layer of the output path. These names allow you to connect network paths and then later explicitly associate each network input layer with its appropriate environment channel.

The following code creates and connects the following input and output paths:

% Observation path: array of layer objects
observationPath = [
    featureInputLayer(4,Name="obsInputLayer")
    fullyConnectedLayer(24)
    reluLayer
    fullyConnectedLayer(24,Name="ObsFC2")
    ];

% Action path: array of layer objects
actionPath = [
    featureInputLayer(1,Name="actInputLayer")
    fullyConnectedLayer(24,Name="ActFC1")
    ];

% Common path: array of layer objects
commonPath = [
    concatenationLayer(1,2,Name="cct")
    reluLayer
    fullyConnectedLayer(1,Name="QValueOutputLayer")
    ];

% Assemble dlnetwork object
criticNetwork = dlnetwork();
criticNetwork = addLayers(criticNetwork,observationPath);
criticNetwork = addLayers(criticNetwork,actionPath);
criticNetwork = addLayers(criticNetwork,commonPath);

% Connect layers   
criticNetwork = connectLayers(criticNetwork,"ObsFC2","cct/in1");
criticNetwork = connectLayers(criticNetwork,"ActFC1","cct/in2");

% Initialize dlnetwork object
criticNetwork = initialize(criticNetwork);

% Display the number of learnable parameters
summary(criticNetwork)

For all observation and action input paths, you must specify a featureInputLayer as the first layer in the path, with a number of input neurons equal to the number of dimensions of the corresponding environment channel.

You can view the structure of your deep neural network using the plot function.

plot(criticNetwork)

Since the output of a network in an rlDiscreteCategoricalActor actors must represent the probability of executing each possible action, the software automatically adds a softmaxLayer as a final output layer if you do not specify it explicitly. When computing the action, the actor then randomly samples the distribution to return an action.

Determining the number, type, and size of layers for your deep neural network can be difficult and is application dependent. However, the most critical component in deciding the characteristics of the function approximator is whether it is able to approximate the optimal policy or discounted value function for your application, that is, whether it has layers that can correctly learn the features of your observation, action, and reward signals.

Consider the following tips when constructing your network.

Create and Configure Actors and Critics from a Neural Network

To create a critic from your deep neural network, use an rlValueFunction, rlQValueFunction or (whenever possible) an rlVectorQValueFunction object. To create a deterministic actor for a continuous action space from your deep neural network, use an rlContinuousDeterministicActor object. To create a stochastic actor from your deep neural network use either an rlDiscreteCategoricalActor or an rlContinuousGaussianActor object. To configure the learning rate and optimization used by the actor or critic, use an optimizer object within an agent option object.

For example, create a Q-value function critic using the neural network criticNetwork and the environment action and observation specifications. Pass as additional arguments also the names of the network input layers to be connected with the observation and action channels, respectively.

critic = rlQValueFunction(criticNetwork,obsInfo,actInfo,...
        ObservationInputNames={"obsInputLayer"}, ...
        ActionInputNames="actInputLayer");

To specify training options for the critic, use rlOptimizerOptions to create the critic optimizer object criticOpts, specifying a learning rate of 0.02 and a gradient threshold of 1.

criticOpts = rlOptimizerOptions(LearnRate=0.02,...
                                GradientThreshold=1);

Then create an agent option object, and set the CriticOptimizerOptions property of the agent option object to criticOpts. When finally you create the agent, pass the agent option object as a last input argument to the agent constructor function. Alternatively, you can create the agent first, and then access its option object, and modify the options, using dot notation.

When you create your deep neural network and configure your actor or critic, consider using the following approach as a starting point.

Also, consider the following tips when configuring your deep neural network agent.

Recurrent Neural Networks

When creating actors or critics for use with any agent except Q, SARSA, TRPO and MBPO, you can use recurrent neural networks (RNN). These networks are deep neural networks with a sequenceInputLayer input layer and at least one layer that has hidden state information, such as an lstmLayer. They can be especially useful when the environment has states that cannot be included in the observation vector.

For agents that have both actor and critic, you must either use an RNN for both of them, or not use an RNN for any of them. You cannot use an RNN only for the critic or only for the actor.

When using PG agents, the learning trajectory length (that is the sequence of input data that the network uses for learning) for the RNN is the whole episode. For an AC agent, the NumStepsToLookAhead property of its options object is treated as the training trajectory length (except when training in parallel, in which case NumStepsToLookAhead is ignored and the whole episode is used as trajectory length). For a PPO agent, the trajectory length is the MiniBatchSize property of its options object.

For DQN, DDPG, SAC and TD3 agents, you must specify the trajectory length as an integer greater than one in the SequenceLength property of their options object. These learning algorithms randomly sample MiniBatchSize experience points from all the available episodes. For each experience point, a sequence of SequenceLength consecutive experiences is used for learning. If the number of available consecutive experiences is shorter than SequenceLength (this can happen for example when an episode terminates prematurely), then the available experiences are padded to complete the sequence. The sequence is also appropriately masked so that the padded data does not affect the gradient computation.

For more information and examples on policies and value functions, see rlValueFunction, rlQValueFunction, rlVectorQValueFunction, rlContinuousDeterministicActor, rlDiscreteCategoricalActor, and rlContinuousGaussianActor.