Custom DDPG Algorithm in MATLAB R2023b: Performing Gradient Ascent for Actor Network

Question

roham farhadi 2023-12-26

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2064177-custom-ddpg-algorithm-in-matlab-r2023b-performing-gradient-ascent-for-actor-network

编辑： Venu 2024-1-8

Hello MATLAB community,

I am working on implementing a custom Deep Deterministic Policy Gradients (DDPG) algorithm in MATLAB R2023b. In the DDPG algorithm, during the training of the actor network, the Q value produced by the critic network is set as the objective function for the actor network. The standard approach involves using gradient ascent to update the actor network based on these Q values.

My question pertains to the use of the gradient function from the Reinforcement Learning Toolbox to calculate gradients. Following this, how can I perform gradient ascent, as the update function from the same toolbox seems to default to gradient descent and not gradient ascent? I would appreciate any insights or examples on implementing gradient ascent in this context.

Thank you for your assistance!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Venu 2024-1-8

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2064177-custom-ddpg-algorithm-in-matlab-r2023b-performing-gradient-ascent-for-actor-network#answer_1385111

编辑：Venu 2024-1-8

Hi @roham farhadi,

Gradient ascent is the same as gradient descent except that you don't multiply your step (learning_rate * gradients) by a negative sign. So your step has the same sign as your gradient.

If the update function defaults to gradient descent, you can adjust the sign of the gradients before updating the parameters.

actorNetwork.Parameters = actorNetwork.Parameters + learningRate * -gradients; (% Perform gradient ascent by adjusting the sign of the gradients)

You can refer to example in this documentation for 'gradient' function

https://www.mathworks.com/help/reinforcement-learning/ug/train-reinforcement-learning-policy-using-custom-training.html,

Hope this helps!