References to multi-agent reinforcement learning schemes in the reinforcement learning toolbox

Question

Lin 2024-7-21

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2139171-references-to-multi-agent-reinforcement-learning-schemes-in-the-reinforcement-learning-toolbox

评论： Umar 2024-7-30

Can somebody provide several references on multi-agent reinforcement learning schemes in reinforcement learning toolbox？

2 个评论
显示无隐藏无

Umar 2024-7-22

Hi Lin,

Plese refer to following links regarding multi-agent reinforcement learning,

https://www.mathworks.com/help/reinforcement-learning/ug/train-agent-to-play-turn-based-game.html https://www.mathworks.com/help/releases/R2023b/reinforcement-learning/ref/rl.env.rlmultiagentfunctionenv.html

Please let me know if you have any further questions.

Lin 2024-7-29

Thank you for you answer!Unfortunately, I could not find any reference even after your advices.I'm looking for references & informations related to the "LearningStrategy:centralized&decentralized" property of object "rlMultiAgentTrainingOptions".Please let me know if you have further tips.Thanks again!

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Emmanouil Tzorakoleftherakis 2024-7-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2139171-references-to-multi-agent-reinforcement-learning-schemes-in-the-reinforcement-learning-toolbox#answer_1490031

The following examples are all on multi-agent reinforcement learning (centralized or decentralized):

https://www.mathworks.com/help/reinforcement-learning/ug/train-agents-for-path-following.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-3-agents-for-area-coverage.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-2-agents-to-collaborate.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-agent-to-play-turn-based-game.html

12 个评论
显示 10更早的评论隐藏 10更早的评论

Umar 2024-7-29

Hi @Lin,

The centralized critic provides feedback to all agents based on the collective experience of the entire team. This approach enables agents to coordinate their actions effectively and learn from a global perspective. Here is an example of how you can set the "LearningStrategy" property to "centralized" in MATLAB:

options = rlMultiAgentTrainingOptions('LearningStrategy', 'centralized');

Conversely, in a decentralized learning strategy, each agent learns independently based on its local observations and rewards. There is no centralized critic providing feedback to all agents. Each agent makes decisions autonomously without considering the global state of the environment. This approach is useful when agents have limited communication or when scalability is a concern.To set the "LearningStrategy" property to "decentralized" in MATLAB, you can use the following code snippet:

options = rlMultiAgentTrainingOptions('LearningStrategy', 'decentralized');

To showcase the "LearningStrategy" property of the "rlMultiAgentTrainingOptions" object in Matlab, I will create a simple example that involves setting up a multi-agent training scenario with both centralized and decentralized learning strategies and then visualize the training progress using plots. So, first setting up the environment by defining the environment and agents for our multi-agent system. I will create a simple environment with two agents.

% Define the environment

env = rlPredefinedEnv("SimpleMultiAgentEnvironment");

% Define the agents

agent1 = rlQAgent(env);

agent2 = rlQAgent(env);

Note: rlPredefinedEnv requires Reinforcement Learning Toolbox.

Then, creating the multi-agent training options by creating the "rlMultiAgentTrainingOptions" object and set the learning strategy to both centralized and decentralized.

% Create multi-agent training options

multiAgentOpts = rlMultiAgentTrainingOptions;

multiAgentOpts.LearningStrategy = ["centralized", "decentralized"];

Now, training the agents using the defined options and visualize the training progress.

% Train the agents

trainingStats = trainMultiAgent(agent1, agent2, env, multiAgentOpts);

% Plot the training progress

figure;

subplot(2,1,1);

plot(trainingStats.CumulativeReward);

title('Cumulative Reward');

subplot(2,1,2);

plot(trainingStats.ActorLoss);

hold on;

plot(trainingStats.CriticLoss);

legend('Actor Loss', 'Critic Loss');

title('Actor and Critic Loss');

Finally, observing the results after running the code, you will see two plots showing the cumulative reward and the actor/critic losses for both centralized and decentralized learning strategies. This detailed implementation demonstrates how to utilize the "LearningStrategy" property of the "rlMultiAgentTrainingOptions" object in Matlab to train agents with different strategies and visualize their training progress effectively. Feel free to customize the environment, agents, and training options to explore more complex multi-agent scenarios and further enhance your understanding of centralized and decentralized learning strategies in reinforcement learning.By understanding the nuances of the "LearningStrategy" property and its implications for multi-agent training, you can tailor your approach to suit the specific requirements of your reinforcement learning scenario. Please let me know if you have any further questions.

Umar 2024-7-30

Hi @Lin,

To delve deeper into the topic and explore relevant research papers, you may consider the following scholarly articles, hope they will provide information relevant to sentence, “ The centralized critic provides feedback to all agents based on the collective experience of the entire team ”.

Title:” Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning”

https://arxiv.org/abs/2102.04402

Title: "Enhancing Multi-Agent Coordination through Common Operating Picture Integration”

https://arxiv.org/abs/2311.04740

These papers delve into the theoretical foundations and practical implications of utilizing a centralized critic in multi-agent systems, shedding light on the benefits and challenges associated with this approach. By studying these resources, you can gain a comprehensive understanding of how centralized critics operate and their impact on the performance of multi-agent systems. Please let me know if you have any further questions.

Lin 2024-7-30

Hello @Umar @Emmanouil Tzorakoleftherakis,

I am very grateful for your help. Your answers have resolved my doubts and filled the gap of lacking references on multi-agent reinforcement learning in Matlab. Thank you again.

Umar 2024-7-30

@Lin,

Glad to help out again, please feel free to ask any questions if you still need any help.

请先登录，再进行评论。

References to multi-agent reinforcement learning schemes in the reinforcement learning toolbox

2 个评论
显示无隐藏无

采纳的回答

12 个评论
显示 10更早的评论隐藏 10更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

References to multi-agent reinforcement learning schemes in the reinforcement learning toolbox

2 个评论 显示 无隐藏 无

采纳的回答

12 个评论 显示 10更早的评论隐藏 10更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

2 个评论
显示无隐藏无

12 个评论
显示 10更早的评论隐藏 10更早的评论