Debug mode for RL agent networks

Question

Nicolas CRETIN 2024-7-8

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2135396-debug-mode-for-rl-agent-networks

评论： Nicolas CRETIN 2024-7-19

采纳的回答： Shantanu Dixit

Is there any way to enter debug mode, to see what is happening inside the RL agent nets while training is running?

Some of my layers output NaN and I would like to know which one. I would also like to monitor to outputs of each layer.

Thanks in advance!

Nicolas

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Nicolas CRETIN 2024-7-18

It would maybe be a solution to add output layers to the network and display their result, but it's tedious.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Shantanu Dixit 2024-7-19

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2135396-debug-mode-for-rl-agent-networks#answer_1487861

编辑：Shantanu Dixit 2024-7-19

在 MATLAB Online 中打开

Hi Nicolas,

It is my understanding that you want to monitor the outputs of each layer and debug the RL agent nets while the training is running.

Similar to analyzing deep learning networks you can call the forward method for each layer of the agent's network to analyze the corresponding output during the training.

[Y1,...,YK] = forward(___,'Outputs',layerNames)

here 'layerNames' correspond to a string array, with 'layerNames(k)' representing kth layer of the agent's network, the corresponding outputs are stored in 'Yk'

Briefly you can follow the below steps:

Extract layer names from the actor network into a string array 'layerNames'
Convert Observation Buffer to dlarray for processing into the network
Forward Pass Through Each Layer using the forward method.

Refer the below code for monitoring the layer outputs corresponding to one observation after updating the actor

%% actorNetwork refers to the DNN of the actor
%% for one observation
dlX = dlarray(observationBuffer(:,:,1), 'CB'); %% format in which the network takes the input
[Y1, Y2, Y3, Y4, Y5, Y6] = forward(actorNetwork, dlX, 'Outputs', layerNames);
layerOutputs = {Y1, Y2, Y3, Y4, Y5, Y6};
for i = 1:numel(layerOutputs)
    % disp(['Layer ', layerNames(i), ' output:']);
    if any(isnan(extractdata(layerOutputs{i})), 'all')
        disp(['Layer ', num2str(i), ' output contains NaNs']);  %% check for NaNs
    end
end