Observation for Reinforcement Learning as Array

Question

Felix Windels 2022-6-2

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1732305-observation-for-reinforcement-learning-as-array

回答： Himanshu 2023-9-28

Hello,

I have created a reinforcement learning environment. It is a system in which the agent always has to make decisions (actions) for the following hour. The basis (observations) for the "decisions" (actions) are forecast values that are taken from time series. In the current version, only the forecasts for the following hour are transferred.

 ObservationInfo = rlNumericSpec([8 1]);
            ObservationInfo.Name = 'Observation';
            ObservationInfo.Description = 'Load_obs, PV_gen_obs, batt_obs, Price_obs, Heat_demand_obs, Heat_stor_obs, hour_of_day_obs,Compense_obs';

For some of the values, however, not only the forecast value for the coming hour, but also for the following hours should be transferred (e.g. 24h).

My approach was as follows:

  ObservationInfo = rlNumericSpec([8 24]); % Not 8 individual values but 8 arrays with 24 values each are transferred.
            ObservationInfo.Name = 'Observation';
            ObservationInfo.Description = 'Load_obs, PV_gen_obs, batt_obs, Price_obs, Heat_demand_obs, Heat_stor_obs, hour_of_day_obs,Compense_obs';

I have checked the step function and the arrays are created as required. However, I cannot create an executable environment. Individual values are still passed as individual values. I have passed these into an array with the value to be passed at position 1 and 23 zeros.

Does anyone have any suggestions?

BR

Felix

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Himanshu 2023-9-28

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1732305-observation-for-reinforcement-learning-as-array#answer_1320974

Hello Felix,

I understand that you are trying to modify the observation space of your reinforcement learning environment to include 24-hour forecasts for certain variables. The issue seems to be that the environment is not executing as expected, possibly due to how the observations are passed into the environment.

You can follow the below steps to fix the issue:

Your arrays should be 2D with a size of [8, 24]. Use MATLAB's "size" function to verify the correct array structure.
The step function should return observations that align with your updated specification, i.e., 2D arrays of size [8, 24].
If you're using a neural network or similar for your Q-function, it may need to be updated to handle the increased dimensionality of the observation space.
Use "validateEnvironment" function in MATLAB to check if your environment meets the requirements for reinforcement learning.

You can refer to the below documentation to learn more about Creating Custom Reinforcement Learning Environments in MATLAB. Also, please refer to the documentations to know more about the "rlNumericSpec" and the "validateEnvironment" functions in MATLAB.

Reinforcement Learning Environments - MATLAB & Simulink - MathWorks India

Create specifications object for a numeric action or observation channel - MATLAB - MathWorks India

Validate custom reinforcement learning environment - MATLAB validateEnvironment - MathWorks India