Question in short: how to design a DQN agent that uses a rlVectorQValueFunction critic based on a custom dlnetwork for an environement with observations that consists in an image and some numeric features.
The documentation of the RL toolbox needs a lot of addition to cover examples that are an epsilon more complex than the vanila ones.
The following code triggers the error. Below, my analysis so far.
Thanks for your help.
NB: I'm using 2022b, in case that matters
env = rlPredefinedEnv('SimplePendulumWithImage-Discrete');
'rlPredefinedEnv' requires Reinforcement Learning Toolbox.
obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);
initOpts = rlAgentInitializationOptions(NumHiddenUnit=64);
agent = rlDQNAgent(obsInfo,actInfo,initOpts);
criticNet = getModel(getCritic(agent));
analyzeNetwork(criticNet);
Now the fun part begin
critic = rlVectorQValueFunction(criticNet,obsInfo,actInfo);
Error using rl.internal.validate.mapNetworkInput
Model input sizes must match the dimensions specified in the corresponding observation and action info specifications.
Error in rl.internal.validate.mapFunctionObservationInput (line 24)
inputOrder = rl.internal.validate.mapNetworkInput(specDim,networkInputSize);
Error in rlVectorQValueFunction (line 101)
modelInputMap = rl.internal.validate.mapFunctionObservationInput(model,observationInfo,nameValueArgs.ObservationInputNames);
Error in exe2 (line 19)
critic = rlVectorQValueFunction(criticNet,obsInfo,actInfo);
We can see the network diagram: Input is 50x50x1x1, which is correct. Note that the image input layer add 1 dimension for chanel...
Let me emphasize that the doc of rlVectorQValueFunction only provides an example with a network with only feature input and it does not provide any information about the data dimension, and specifically how to manage the chanel dimension in data image:
"The network must have only the observation channels as inputs and a single output layer having as many elements as the number of possible discrete actions. Each element of the output vector approximates the value of executing the corresponding action starting from the currently observed state."
We can check that the network has the correct size:
>> obsInfo(1)
ans =
rlNumericSpec with properties:
LowerLimit: 0
UpperLimit: 1
Name: "pendImage"
Description: [0×0 string]
Dimension: [50 50]
DataType: "double"
>> obsInfo(2)
ans =
rlNumericSpec with properties:
LowerLimit: -Inf
UpperLimit: Inf
Name: "angularRate"
Description: [0×0 string]
Dimension: [1 1]
DataType: "double"
Where is the problem ?
I debug the rlVectorQValueFunction up to the point that it check the size of the network input data (50x50x1) versus the observation size (50x50). As the input image layer always add 1 dimension for chanel, the number of dimensions does not match.
The problems is in mapFunctionObservationInput (line 24),
inputOrder = rl.internal.validate.mapNetworkInput(specDim,networkInputSize);
specDim =
1×2 cell array
{[50 50]} {[1 1]}
K>> networkInputSize
networkInputSize =
1×2 cell array
{[50 50 1]} {[1]}
So it is clear that the network input size has a chanel dimension that is not expected....
Where does the documentation provide any explanation about that ???? How to fix this issue?
