Hello,
I am trying to find a RL solution to control a control moment gyroscope using the reinforcement learning designer. The control moment gyroscope is in the inner loop. The outer loop consists of a 1DOF spacecraft attitude dynamics model (but is not very relevant to the RL control problem). To give you a general idea of the dynamic system:
Inside the 'SGCMG RL control and dynamics' block, I implemented the control as well as the dynamics of the control moment gyro. In the 'SGCMG RL control law block' is the RL agent. The action is the angular velocity of the control moment gyro's gimbal angle. The observation consists of the angular velocity of the spacecraft (s/c) and the difference between commanded torque and total torque acting on the s/c. Finally, the reward function consists of the error state, cos(delta) which is a measure of proximity to singularity of the system and the use of the actuator. These are all penalties and it is therefore more of a cost function than a reward function.
As for the RL process, I initiated the environment as follows:
obsInfo=rlNumericSpec([1 2]);
obsInfo.Name='observation';
actInfo = rlNumericSpec([1 1],'LowerLimit',-10^20,'UpperLimit',10^20);
agentBlk=[mdl '/SGCMG RL control and dynamics/SGCMG RL control law/RL Agent'];
env=rlSimulinkEnv(mdl,agentBlk,obsInfo,actInfo);
plot(tout,rad2deg(delta(1,:)))
Next, I imported the environment into the reinforcement learning designer and used a default DDPG agent. My main problem at this moment is that the action will go to its maximum value, after 1 or 2 timesteps into the episode and won't change (by a lot). For instance, with a lowerlimit, upperlimit of -10^20; 10^20, the penalty for the actuator will go straight to the maximum value (deltadot*10^-5=10^15 for the current reward function). Note that this value will be subtracted before going into the reward port of the RL block. If I do not implement a lower and upper limit on the action, it will go to infinity after some timesteps and singularity problems kick in. I would say, that with this severe penalty for the actuator (action) its value wouldn't go all the way up. It's almost like my RL algorithm is trying to minimize the reward... However, if I delete the reward function alltogether (no input for the RL agent at all), the same thing happens. So it cannot be the reward function I'd say.
Would that mean there's a problem with the default DDPG agent? The default actor and critic NNs are, respectively:
Allthough I'm pretty experienced in Matlab and Simulink, RL and NNs are relatively new to me. I already tried to play with the exploration setting (stand. dev.) but this won't solve the problem. I was hoping anybody with a bit more experience in RL could spot the problem and help me on my way.
With kind regards,
Noah
) go to the infinity and beyond as well?
