Train a DQN Agent: RL Simulink with Simscape (Error: Discontinuities detected within algebraic loop)

9 次查看（过去 30 天）
Reinforcement Learning
回答： MULI 2024-11-15，8:03
Hello,
I am trying to replace the PI-Controller (the highlighted area with PWM Generator of 50 kHz) for the Buck-Converter sketch with A DQN-Agent!
Here the Simulink Enviroment with the PI-Controller, which is performing well
I tried to replicate the MATLAB example "Water Tank Model" in Simulink using RL, but using DQN instead of DDPG (since the action is discrete) and replaced the environment with the buck-converter with some other minor changes (tolerance ), but I am getting various errors, when I use validateEnvironment. Here my Matlab code with the simulink environment:
Obs_Info = rlNumericSpec([1 1]);
Obs_Info.Name = 'observations'; % measured current
numObservations = Obs_Info.Dimension(1);
% specifies discrete action specifications
Act_Info = rlFiniteSetSpec([0 1]);
Act_Info.Name = 'PWM';
numActions = Act_Info.Dimension(1);
env = rlSimulinkEnv('rl_converter','rl_converter/RL Agent', Obs_Info, Act_Info);
workspace = 'Stromsteuerung_rl';
env.ResetFcn = @(in)setVariable(in,'observations',0,'Workspace',workspace);
% the agent gets executed every SampleTime seconds of simulation time
Ts = 1/50000; % (50 kHz)
Tf = (1/50000)*60; % simulation time
%% Deep Neural Network
dnn = [
featureInputLayer(1,'Normalization','none','Name','State')
fullyConnectedLayer(40,'Name','CriticStateFC1')
reluLayer('Name','CriticRelu1')
fullyConnectedLayer(40, 'Name','CriticStateFC2')
reluLayer('Name','CriticCommonRelu')
fullyConnectedLayer(2,'Name','Action')];
%% Representation of Q-Values
Critic_Opts = rlRepresentationOptions;
Critic_Opts.LearnRate = 0.001;
Critic_Opts.GradientThreshold = 1;
critic = rlQValueRepresentation(dnn, Obs_Info, Act_Info, 'Observation',{'State'},Critic_Opts);
%% Agent options
Ag_Opts = rlDQNAgentOptions;
Ag_Opts.UseDoubleDQN = true;
Ag_Opts.TargetSmoothFactor = 1;
Ag_Opts.TargetUpdateFrequency = 4;
Ag_Opts.ExperienceBufferLength = 100000;
Ag_Opts.DiscountFactor = 0.9;
Ag_Opts.MiniBatchSize = 64;
Ag_Opts.SampleTime = Ts;
Ag_Opts.EpsilonGreedyExploration.Epsilon = 0.5;
agent = rlDQNAgent(critic,Ag_Opts);
%% Training options
Train_Opts = rlTrainingOptions;
Train_Opts.MaxEpisodes = 100;
Train_Opts.MaxStepsPerEpisode = ceil(Tf/Ts);
Train_Opts.StopTrainingCriteria = "AverageReward"; % or AverageSteps
Train_Opts.StopTrainingValue = 800;
Train_Opts.Verbose = false;
Train_Opts.Plots = "training-progress";
%% Validate
validateEnvironment(env)
Warning: Discontinuities detected within algebraic loop(s), may have trouble solving
Warning: Convergence problem when solving algebraic loop containing 'rl_converter/stop simulation/Compare To Constant1/Compare'
at time 0.0. Simulink will try to solve this loop using Simulink 3 (R11) strategy.
Use feature('ModeIterationsInAlgLoops',0) to disable the strategy introduced in Simulink 4 (R12)
Why isn't this warning showing in the Water Tank Model and only in this one? Is Simscape not compatible with this application. If so, how could I change my environment to something that goes well with the RL Toolbox?
Any help is very much appreciated!

请先登录，再回答此问题。

回答（1 个）

MULI
MULI 2024-11-15，8:03
Hello,
I understand that you are trying to replace a PI controller with a DQN agent for your Buck Converter model and encountering issues with algebraic loops likely due to the usage of fast switching elements like the PWM.
In contrast, the Water Tank Model example doesn’t involve such fast dynamics or Simscape components, so it runs without these warnings.
You can follow the below suggestions to resolve this issue:
  • You could try breaking the algebraic loop by adding a small delay (e.g.,”Unit Delay” block) between the controller output and the feedback signal.
  • Since you are using DQN, a discrete-action RL algorithm, ensure that your environment runs with a discrete sample time. If the continuous-time nature of Simscape is causing issues, try setting discrete sample times in all parts of your model
For any additional insights on how to create and simulate a reinforcement learning (RL) environment using both MATLAB and Simulink you can refer below link:
https://www.mathworks.com/help/reinforcement-learning/ug/simulate-agent-and-environment-in-matlab-and-simulink.html

类别

在 Help Center 和 File Exchange 中查找有关 Reinforcement Learning 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by