rlQValueFunction
Q-Value function approximator with a continuous or discrete action space reinforcement learning agents
Since R2022a
Description
This object implements a Q-value function approximator that you can use as a
critic for a reinforcement learning agent. A Q-value function (also known as action-value
function) is a mapping from an environment observation-action pair to the value of a policy.
Specifically, its output is a scalar that represents the expected discounted cumulative
long-term reward when an agent starts from the state corresponding to the given observation,
executes the given action, and keeps on taking actions according to the given policy
afterwards. A Q-value function critic therefore needs both the environment state and an action
as inputs. After you create an rlQValueFunction critic, use it to create an
agent such as rlQAgent, rlDQNAgent, rlSARSAAgent, rlDDPGAgent, or rlTD3Agent. For more
information on creating actors and critics, see Create Policies and Value Functions.
Creation
Syntax
Description
critic = rlQValueFunction(net,observationInfo,actionInfo)critic. Here,
net is the deep neural network used as an approximation model,
and it must have both observation and action as input layers and a single scalar output
layer. The network input layers are automatically associated with the environment
observation and action channels according to the dimension specifications in
observationInfo and actionInfo. This
function sets the ObservationInfo and
ActionInfo properties of critic to the
observationInfo and actionInfo input
arguments, respectively.
critic = rlQValueFunction(tab,observationInfo,actionInfo)critic with discrete
action and observation spaces from the Q-value table
tab. tab is a rlTable object
containing a table with as many rows as the number of possible observations and as many
columns as the number of possible actions. The function sets the ObservationInfo and ActionInfo
properties of critic respectively to the
observationInfo and actionInfo input
arguments, which in this case must be scalar rlFiniteSetSpec
objects.
critic = rlQValueFunction({basisFcn,W0},observationInfo,actionInfo)critic using a custom basis
function as underlying approximator. The first input argument is a two-element cell
array whose first element is the handle basisFcn to a custom basis
function and whose second element is the initial weight vector W0.
Here the basis function must have both observation and action as inputs and
W0 must be a column vector. The function sets the
ObservationInfo and ActionInfo properties of
critic to the observationInfo and
actionInfo input arguments, respectively.
critic = rlQValueFunction(___,Name=Value)UseDevice property of critic using one
or more name-value arguments. Specifying the input layer names allows you explicitly
associate the layers of your network approximator with specific environment channels.
For all types of approximators, you can specify the device where computations for
critic are executed, for example
UseDevice="gpu".
Input Arguments
Name-Value Arguments
Properties
Object Functions
rlDDPGAgent | Deep deterministic policy gradient (DDPG) reinforcement learning agent |
rlTD3Agent | Twin-delayed deep deterministic (TD3) policy gradient reinforcement learning agent |
rlDQNAgent | Deep Q-network (DQN) reinforcement learning agent |
rlQAgent | Q-learning reinforcement learning agent |
rlSARSAAgent | SARSA reinforcement learning agent |
rlSACAgent | Soft actor-critic (SAC) reinforcement learning agent |
getValue | Obtain estimated value from a critic given environment observations and actions |
getMaxQValue | Obtain maximum estimated value over all possible actions from a Q-value function critic with discrete action space, given environment observations |
evaluate | Evaluate function approximator object given observation (or observation-action) input data |
getLearnableParameters | Obtain learnable parameter values from agent, function approximator, or policy object |
setLearnableParameters | Set learnable parameter values of agent, function approximator, or policy object |
setModel | Set approximation model in function approximator object |
getModel | Get approximation model from function approximator object |
Examples
Create an observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment). For this example, define the observation space as a continuous four-dimensional space, so that there is a single observation channel that carries a column vector containing four doubles.
obsInfo = rlNumericSpec([4 1]);
Create an action specification object (or alternatively use getActionInfo to extract the specification object from an environment). For this example, define the action space as a continuous two-dimensional space, so that the action channel carries a column vector containing two doubles.
actInfo = rlNumericSpec([2 1]);
A Q-value function critic takes the current observation and an action as inputs and returns a single scalar as output (the estimated discounted cumulative long-term reward for taking the action from the state corresponding to the current observation, and following the policy thereafter).
To model the parametrized Q-value function within the critic, use a neural network with two input layers (one receiving the content of the observation channel, as specified by obsInfo, and the other receiving the content of the action channel, as specified by actInfo) and one output layer (returning the scalar value).
Define each network path as an array of layer objects. Assign names to the input and output layers of each path, so you can properly connect the paths. Get the dimensions of the observation and action spaces from the environment specification (regardless of whether the observation space is a column vector, row vector, or matrix, prod(obsInfo.Dimension) is its total number of dimensions, for this example four, similarly, prod(actInfo.Dimension) is the number of dimension of the action space, for this example two).
% Observation path layers obsPath = [ featureInputLayer(prod(obsInfo.Dimension)) fullyConnectedLayer(5) reluLayer fullyConnectedLayer(5,Name="obsout") ]; % Action path layers actPath = [ featureInputLayer(prod(actInfo.Dimension)) fullyConnectedLayer(5) reluLayer fullyConnectedLayer(5,Name="actout") ]; % Common path to output layers % Concatenate two layers along dimension one. comPath = [ concatenationLayer(1,2,Name="cct") fullyConnectedLayer(5) reluLayer fullyConnectedLayer(1, Name="output") ];
Assemble dlnetwork object.
net = dlnetwork; net = addLayers(net,obsPath); net = addLayers(net,actPath); net = addLayers(net,comPath);
Connect layers.
net = connectLayers(net,"obsout","cct/in1"); net = connectLayers(net,"actout","cct/in2");
Plot network.
plot(net)
Initialize network and display the number of weights.
net = initialize(net); summary(net)
Initialized: true
Number of learnables: 161
Inputs:
1 'input' 4 features
2 'input_1' 2 features
Create the critic with rlQValueFunction, using the network as well as the observations and action specification objects. When using this syntax, the network input layers are automatically associated with the components of the observation and action signals according to the dimension specifications in obsInfo and actInfo.
critic = rlQValueFunction(net,obsInfo,actInfo)
critic =
rlQValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlNumericSpec]
ActionInfo: [1×1 rl.util.rlNumericSpec]
Normalization: ["none" "none"]
UseDevice: "cpu"
Learnables: {12×1 cell}
State: {0×1 cell}
To check your critic, use getValue to return the value of a random observation and action, given the current network weights.
v = getValue(critic, ... {rand(obsInfo.Dimension)}, ... {rand(actInfo.Dimension)})
v = single
-1.1006
Obtain values for a random batch of 5 observations.
v = getValue(critic, ... {rand([obsInfo.Dimension 5])}, ... {rand([actInfo.Dimension 5])})
v = 1×5 single row vector
-0.8916 -0.3538 -0.8732 -1.0594 -0.9641
You can now use the critic (along with an actor) to create an agent for the environment described by the given specification objects. Examples of agents that can work with continuous action and observation spaces, and use a Q-value function critic, are rlDDPGAgent, rlTD3Agent, and rlSACAgent.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Create an observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment). For this example, define the observation space as a continuous four-dimensional space, so that there is a single observation channel that carries a column vector containing four doubles.
obsInfo = rlNumericSpec([4 1]);
Create an action specification object (or alternatively use getActionInfo to extract the specification object from an environment). For this example, define the action space as a continuous two-dimensional space, so that the action channel carries a column vector containing two doubles.
actInfo = rlNumericSpec([2 1]);
A Q-value function critic takes the current observation and an action as inputs and returns a single scalar as output (the estimated discounted cumulative long-term reward for taking the action from the state corresponding to the current observation, and following the policy thereafter).
To model the parametrized Q-value function within the critic, use a neural network with two input layers (one receiving the content of the observation channel, as specified by obsInfo, and the other receiving the content of the action channel, as specified by actInfo) and one output layer (returning the scalar value).
Define each network path as an array of layer objects. Assign names to the input and output layers of each path. This assignment allows you to connect the paths and explicitly associate the network input and output layers with the appropriate environment channel.
Get the dimensions of the observation and action spaces from the environment specification (regardless of whether the observation space is a column vector, row vector, or matrix, prod(obsInfo.Dimension) is its total number of dimensions, for this example four, similarly, prod(actInfo.Dimension) is the number of dimension of the action space, for this example two).
% Observation path layers obsPath = [ featureInputLayer( ... prod(obsInfo.Dimension), ... Name="obsInLyr") fullyConnectedLayer(16) reluLayer fullyConnectedLayer(5,Name="obsPthOutLyr") ]; % Action path layers actPath = [ featureInputLayer( ... prod(actInfo.Dimension), ... Name="actInLyr") fullyConnectedLayer(16) reluLayer fullyConnectedLayer(5,Name="actPthOutLyr") ]; % Common path to output layers % Concatenate two layers along dimension one comPath = [ concatenationLayer(1,2,Name="cct") fullyConnectedLayer(8) reluLayer fullyConnectedLayer(1, Name="qvfOutLyr") ];
Create dlnetwork object and add layers.
net = dlnetwork(); net = addLayers(net,obsPath); net = addLayers(net,actPath); net = addLayers(net,comPath);
Connect layers.
net = connectLayers(net,"obsPthOutLyr","cct/in1"); net = connectLayers(net,"actPthOutLyr","cct/in2");
Plot network.
plot(net)
Initialize network and display the number of weights.
net = initialize(net); summary(net)
Initialized: true
Number of learnables: 395
Inputs:
1 'obsInLyr' 4 features
2 'actInLyr' 2 features
Create the critic with rlQValueFunction, using the network, the observations and action specification objects, and the names of the network input layers to be associated with the observation and action from the environment.
critic = rlQValueFunction(net,... obsInfo,actInfo, ... ObservationInputNames="obsInLyr",... ActionInputNames="actInLyr")
critic =
rlQValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlNumericSpec]
ActionInfo: [1×1 rl.util.rlNumericSpec]
Normalization: ["none" "none"]
UseDevice: "cpu"
Learnables: {12×1 cell}
State: {0×1 cell}
To check your critic, use getValue to return the value of a random observation and action, given the current network weights.
v = getValue(critic, ... {rand(obsInfo.Dimension)}, ... {rand(actInfo.Dimension)})
v = single
0.2703
You can now use the critic (along with an actor) to create an agent for the environment described by the given specification objects. Examples of agents that can work with continuous action and observation spaces, and use a Q-value function critic, are rlDDPGAgent, rlTD3Agent, and rlSACAgent.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Create a finite set observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment with a discrete observation space). For this example define the observation space as a scalar belonging to a finite set with four possible values.
obsInfo = rlFiniteSetSpec([7 5 3 1]);
Create a finite set action specification object (or alternatively use getActionInfo to extract the specification object from an environment with a discrete action space). For this example define the action space as a finite set with two possible values.
actInfo = rlFiniteSetSpec([4 8]);
A Q-value function critic takes the current observation and an action as inputs and returns a single scalar as output (the estimated discounted cumulative long-term reward for taking the action from the state corresponding to the current observation, and following the policy thereafter).
Since both observation and action spaces are discrete and low-dimensional, use a table to model the Q-value function within the critic. rlTable creates a value table object from the observation and action specifications objects.
qTable = rlTable(obsInfo,actInfo);
The table stores a value (representing the expected cumulative long term reward) for each possible observation-action pair. Each row corresponds to an observation and each column corresponds to an action. You can access the table using the Table property of the vTable object. The initial value of each element is zero.
qTable.Table
ans = 4×2
0 0
0 0
0 0
0 0
You can initialize the table to any value, in this case an array containing the integer from 1 through 8.
qTable.Table=reshape(1:8,4,2)
qTable =
rlTable with properties:
Table: [4×2 double]
Create the critic using the table as well as the observations and action specification objects.
critic = rlQValueFunction(qTable,obsInfo,actInfo)
critic =
rlQValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlFiniteSetSpec]
ActionInfo: [1×1 rl.util.rlFiniteSetSpec]
Normalization: ["none" "none"]
UseDevice: "cpu"
Learnables: {[4×2 dlarray]}
State: {}
To check your critic, use getValue to return the value of a given observation and action, using the current table entries.
v = getValue(critic,{5},{8})v = 6
Obtain values for a batch of 5 observations.
v = getValue(critic,{[5,3,1,7,7]},{[8,4,4,8,4]})v = 1×5
6 3 4 5 1
You can now use the critic (along with an actor) to create an agent for the environment described by the given specification objects. Examples of agents that can work with discrete action and observation spaces, and use a Q-value function critic, are rlQAgent, rlDQNAgent, and rlSARSAAgent.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Create an observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment). For this example, define the observation space as a continuous three-dimensional space, so that there is a single observation channel that carries a column vector containing three doubles.
obsInfo = rlNumericSpec([3 1]);
Create an action specification object (or alternatively use getActionInfo to extract the specification object from an environment). For this example, define the action space as a continuous two-dimensional space, so that the action channel carries a column vector containing two doubles.
actInfo = rlNumericSpec([2 1]);
A Q-value function critic takes the current observation and an action as inputs and returns a single scalar as output (the estimated discounted cumulative long-term reward for taking the action from the state corresponding to the current observation, and following the policy thereafter).
To model the parametrized Q-value function within the critic, use a custom basis function with two inputs, the first receiving the content of the observation channel (as specified by obsInfo) and the other receiving the content of the action channel, (as specified by actInfo).
Create a custom function that returns a vector of four elements, given an observation and an action as inputs.
myBasisFcn = @(myobs,myact) [ ... myobs(2)^2; ... myobs(1)+exp(myact(1)); ... abs(myact(2)); ... myobs(3) ]
myBasisFcn = function_handle with value:
@(myobs,myact)[myobs(2)^2;myobs(1)+exp(myact(1));abs(myact(2));myobs(3)]
The output of the critic is the scalar W'*myBasisFcn(myobs,myact), which represents the estimated value of the observation-action pair, under the given policy. Here W is a weight column vector which must have the same size of the custom function output. The elements of W are the learnable parameters.
Define an initial parameter vector.
W0 = [1;4;4;2];
Create the critic. The first argument is a two-element cell containing both the handle to the custom function and the initial weight vector. The second and third arguments are, respectively, the observation and action specification objects.
critic = rlQValueFunction({myBasisFcn,W0}, ...
obsInfo,actInfo)critic =
rlQValueFunction with properties:
ObservationInfo: [1×1 rl.util.rlNumericSpec]
ActionInfo: [1×1 rl.util.rlNumericSpec]
Normalization: ["none" "none"]
UseDevice: "cpu"
Learnables: {[1×4 dlarray]}
State: {}
To check your critic, use getValue to return the value of a given observation-action pair, using the current parameter vector.
v = getValue(critic,{[1 2 3]'},{[4 5]'})v = 252.3926
You can now use the critic (along with an actor) to create an agent for the environment described by the given specification objects. Examples of agents that can work with continuous action and observation spaces, and use a Q-value function critic, are rlDDPGAgent, rlTD3Agent, and rlSACAgent.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Create an observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment). For this example, define the observation space as an hybrid (that is mixed discrete-continuous) space with the continuous channel carrying a vector over a continuous two-dimensional space and the second carrying a vector over a three-dimensional space that can assume only four values.
obsInfo = [rlNumericSpec([1 2])
rlFiniteSetSpec({[1 0 -1], ...
[-1 2 1], ...
[0.1 0.2 0.3], ...
[0 0 0] ...
})
];Create an action specification object (or alternatively use getActionInfo to extract the specification object from an environment). For this example, define the action space as a discrete set consisting of three possible actions, labeled 1, 2, and 3.
actInfo = rlFiniteSetSpec({1,2,3});A Q-value function critic takes a batch of observations and a batch of actions as inputs and returns as output a corresponding batch of scalars. Each scalar represents the estimated discounted cumulative long-term reward (the value) obtained by taking the action from the state corresponding to the current observation, and following the policy thereafter.
To model the parametrized Q-value function within the critic, use a custom basis function with three inputs, the first two receiving the content of the two observation channel (as specified by obsInfo) and the third receiving the content of the action channel, (as specified by actInfo).
Create a custom function that returns a vector of four elements, given an observation and an action as inputs. Here, the third dimension is the batch dimension. For each element of the batch dimension, the output of the basis function is a vector with four elements.
myBasisFcn = @(obsA,obsB,act) [
obsA(1,1,:)+obsB(1,2,:)+obsB(1,3,:)+act(1,1,:);
obsA(1,2,:)+obsB(1,1,:)+obsB(1,2,:)-act(1,1,:);
obsA(1,1,:)+obsB(1,2,:)+obsB(1,3,:)+act(1,1,:).^2;
obsA(1,1,:)+obsB(1,1,:)+obsB(1,2,:)-act(1,1,:).^2;
];The output of the critic is the scalar W'*myBasisFcn(myobs,myact), which represents the estimated value of the observation-action pair, under the given policy. Here W is a weight column vector which must have the same size of the custom function output. The elements of W are the learnable parameters.
Define an initial parameter vector.
W0 = ones(4,1);
Create the critic. The first argument is a two-element cell containing both the handle to the custom function and the initial weight vector. The second and third arguments are, respectively, the observation and action specification objects.
critic = rlQValueFunction({myBasisFcn,W0}, ...
obsInfo,actInfo)critic =
rlQValueFunction with properties:
ObservationInfo: [2×1 rl.util.RLDataSpec]
ActionInfo: [1×1 rl.util.rlFiniteSetSpec]
Normalization: ["none" "none" "none"]
UseDevice: "cpu"
Learnables: {[1×4 dlarray]}
State: {}
To check your critic, use getValue to return the value of a given observation-action pair, using the current parameter vector.
v = getValue(critic,{[-0.5 0.6],[1 0 -1]},{3})v = -0.9000
Note that the critic does not enforce the set constraint for the discrete set elements.
v = getValue(critic,{[-0.5 0.6],[10 -10 -0.05]},{33})v = -21.0000
Obtain values for a random batch of 10 observations.
v = getValue(critic,{ ...
rand([obsInfo(1).Dimension 10]), ...
rand([obsInfo(2).Dimension 10]) ...
}, ...
{rand([actInfo.Dimension 10])});Display the seventh value of the batch.
v(7)
ans = 7.2666
You can now use the critic (along with an actor) to create an agent for the environment described by the given specification objects. Examples of agents that can work with a discrete action space, a mixed observation space, and use a Q-value function critic, are rlQAgent, rlDQNAgent, and rlSARSAAgent.
For more information on creating approximator objects such as actors and critics, see Create Policies and Value Functions.
Version History
Introduced in R2022a
