Error with using loggedSignals in the reset function while creatin RL environment using custom functions

Question

Daniel Ramandi 2023-10-13

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2032964-error-with-using-loggedsignals-in-the-reset-function-while-creatin-rl-environment-using-custom-funct

评论： Daniel Ramandi 2023-10-16

Hi,

I'm trying to train an rlPGAgent using a dataset I have (in a cell array variable called blocks in the following code). As a result, I'm defining a custom reset function that after each episode would take the next row from my dataset as the new initial observation. But I can't get the loggedSignal property working properly. Maybe I'm understanding it wrong, but shouldn't it be initialize when the rlFunctionEnv is called?

Here's my custom reset function:

function [initialObservation, LoggedSignals] = myResetFunction(blocks)
    % Start a new trial
    LoggedSignals.currentTrial = LoggedSignals.currentTrial + 1;
    if LoggedSignals.currentTrial > numel(blocks)
        LoggedSignals.currentTrial = 1; % Loop back to the first trial if it's gone through all of them
    end
    % Reset timestep for new trial
    LoggedSignals.currentTimestep = 1;
    % Set the initial state for the new trial
    initialObservation = [blocks{LoggedSignals.currentTrial,2}(1,2),... % Initial position
                          0,... % Initial hold (always 0)
                          0,... % Initial speed (always 0)
                          blocks{LoggedSignals.currentTrial,5}]; % Required hold time for this trial
    LoggedSignals.currentState = initialObservation;
end

And I call the function as a handle as follows:

ResetHandle = @() myResetFunction(blocks);
StepHandle = @(action,LoggedSignals) myStepFunction(action,LoggedSignals,blocks);
observationInfo = rlNumericSpec([1 4], 'LowerLimit', [0, 0, -30, 0], 'UpperLimit', [31, 5, 30, 1]);
actionInfo = rlNumericSpec([1 1], 'LowerLimit', -30, 'UpperLimit', 30);
env = rlFunctionEnv(observationInfo, actionInfo, StepHandle,ResetHandle);

The myResetFunction function alone (or when running the rlFunctionEnv function) runs into an error:

Unrecognized function or variable 'LoggedSignals'.

I'm using MATLAB R2021b, but I also tried Info (which seems to be replacing LoggedSignals in 2023b), but same issue. The issue is resolved if I initialized 'LoggedSignals' outside the function, and passed it to the reset function as an input argument, but that would cause the training to get stuck on the first trial, as the variable wouldn't update between reset and step function calls during the training process! Also declaring it as global causes a lot of other issues!

Does anyone know why this is happening and how I can potentially resolve it? Thanks for your help in advance!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Emmanouil Tzorakoleftherakis 2023-10-13

编辑：Emmanouil Tzorakoleftherakis 2023-10-13

在 MATLAB Online 中打开

The very first line in our reset function is

LoggedSignals.currentTrial = LoggedSignals.currentTrial + 1;

How would the function know what LoggedSignals.currentTrial value is? Hence the error. Either pass is as an argument or make it persistent and initialize it, or make it a class variable if you create the RL environment through the class template.

LoggedSignals is not a stored/shared variable in the RL environment. You can use it like that but it's up to you to make sure it's visible where it needs to be