You need to get familiar with Sliced Variables and their restrictions. Basically, if you have an expression like state(expr,k) involving a sliced variable, then expr must remain the same if you index the same variable later in the loop. You violate this where you have state(1,k) and then later state(t-1,k). Moreover, t-1 is not a valid choice for expr. You cannot have complex expressions in the index that doesn't involve the loop variable. It has to be something like state(t,k).
Here is one way to get rid of the errors:
parfor k = 1:num_agents
for t = 1:time_steps-1
if t==1
state(t,k) = states(randi(ls),1);
if rand < epsilon
action(t,k) = actions(1,randi(la)); %#ok<*PFBNS>
else
[maxq(1,k),action(t,k)] = max(Q(state(t,k),:,k));
equalmax(1,k) = find(Q(state(t,k),:,k) == maxq(1,k));
if numel(equalmax(1,k)) > 1
idx(1,k) = randi(numel(equalmax(1,k)));
action(t,k) = idx(1,k) -1;
end
end
end
Reward_tmp(t,k) = state(t,k)*action(t,k);
end
end
reward(2:end,:)=Reward_tmp;
