The error is likely occurring because the LSTM layers require explicit handling of their states, which is not just about feeding in the external states (observations) but also about managing the internal states of the LSTM layers. In the code you provided, there is a local function "localResetFcn" that is used as the environment's reset function. However, this function currently only sets the parameter 'Value' for the block 'simmodelppo/Set' in your Simulink model.
In the code snippet you've provided, there is no explicit handling of the LSTM states, so unless the "rlContinuousGaussianActor" and "rlValueFunction" objects handle this internally in a way that is not shown, you would need to add this functionality.
Here's an example of how you might modify your "localResetFcn" to include resetting the LSTM states:
function in = localResetFcn(in) %%%%%%% RANDOM INPUT GENERATOR %%%%%%%
% randomize setpoints -- ensure feasible
set_point = 1 + 0.0*rand; % Set-point [0,1]
in = setBlockParameter(in, 'simmodelppo/Set', 'Value', num2str(set_point));
% Reset the LSTM states of the actor and critic networks
% Note: The following is an example and may need adjustment
% to match the specifics of your MATLAB version and network setup.
% Reset actor LSTM states
actor = getActor(agent);
actor = resetState(actor);
agent = setActor(agent, actor);
% Reset critic LSTM states
critic = getCritic(agent);
critic = resetState(critic);
agent = setCritic(agent, critic);
end
Since the "localResetFcn" currently does not have access to the 'agent' variable, you would need to modify your training setup to ensure that the agent's states can be reset at the start of each episode. This might involve changes to how the 'agent' variable is passed around or stored.
