It seems you want to prevent the reinforcement learning (RL) agent from taking an action that has already been taken in the current episode, which in your case is removing a node that has already been removed from the graph. To achieve this, you will need to modify your environment to keep track of the actions taken and to provide a signal or a penalty to the agent when it attempts to take an invalid action.
Here's a modified version of the step function that includes a check for whether the node has already been removed. If the node has already been removed, it provides a negative reward and sets the IsDone flag to true to end the episode.
function [Observation, Reward, IsDone, LoggedSignals] = step(this, Action)
LoggedSignals = [];
% Check if the given action is valid
if this.State(Action,1) == 0
% The node has already been removed, give a large negative reward and end the episode
Reward = -1;
IsDone = true;
Observation = this.State;
else
% Node Disconnection
A = this.A;
this.State(Action,:) = zeros(1, size(this.State, 2));
A(Action,:) = zeros(1, length(A));
A(:,Action) = zeros(length(A), 1);
this.A = A;
% Giant connected component
[gcc] = largestcomponent(A);
% Node degree
[deg, indeg, outdeg] = degrees(A);
% Local clustering coefficient
[C1, C2, C] = clustCoeff(A);
% Update system states
Observation = [deg' C];
this.State = Observation;
% Check terminal condition
disconnectedN = sum(Observation(:,1) == 0); % Count how many nodes have degree 0
IsDone = disconnectedN >= this.maxdisconnectN || gcc <= this.mingcc;
this.IsDone = IsDone;
% Get reward
Reward = 1 / length(this.A) * gcc / length(this.A);
end
% environment has been updated (e.g. to update visualization)
notifyEnvUpdated(this);
end
In this modification, before proceeding with removing a node, the step function checks if the node has already been removed by looking at the State property. If the degree of the node (first column in State) is zero, it means the node has already been removed, and the function provides a negative reward and ends the episode.
Please note that ending the episode might not be the best strategy for training an RL agent, as it could lead to the agent learning to avoid the penalty by not taking any action at all. A better approach might be to provide a negative reward but allow the episode to continue, or to implement a masking mechanism that only presents the agent with valid actions at each step.
