Reinforcement Learning SAC Agent Critics initialise with identical weights

1 次查看（过去 30 天）
Joe Gibbs
Joe Gibbs 2024-4-4
回答： Joe Gibbs 2024-4-4
I am following the basic outline from https://uk.mathworks.com/help/reinforcement-learning/ref/rl.agent.rlsacagent.html#mw_1dfe5ced-ed0e-4ce9-a25c-38c4f163ffcb to create an RL SAC agent but I'm getting an error that the two critics I generate are identical. The help page says to use the "initialize" function to initialize the two networks separetely to ensure that the initial weights are different.
% Initialise the two critics separately
criticNetwork1 = initialize(criticNetwork);
criticNetwork2 = initialize(criticNetwork);
When I do this and input the same test values for (Observation) and (Action), I get the same value from both critic networks, implying that the initial weights are identical.
critic1 = rlQValueFunction(criticNetwork1,observationInfo,actionInfo,'ActionInputNames','action','ObservationInputNames','observation');
critic2 = rlQValueFunction(criticNetwork2,observationInfo,actionInfo,'ActionInputNames','action','ObservationInputNames','observation');
I assume this is a problem on my end and not with the "initialize" function but would welcome any suggestions as to what would be causing this problem!

请先登录，再回答此问题。

采纳的回答

Joe Gibbs
Joe Gibbs 2024-4-4
After some additional testing, the networks must be created by starting with an initial blank dlnetwork which seems to only work in 2024a.
% Create the Layer Graph
criticNetwork = dlnetwork;
criticNetwork = addLayers(criticNetwork,obsPath);
criticNetwork = addLayers(criticNetwork,actPath);
criticNetwork = addLayers(criticNetwork,commonPath);
If you create a layerGraph and convert to a dlnetwork afterwards the initialize function won't initialise with a different set of weights. If you have scripts to create networks from previous MATLAB versions you may need to change the structure to create them as blank dlnetwork objects first when using identical SAC critics!

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息

标签

产品


版本

R2024a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by