Hello,
In my Simulink environment i resort to the same unit delay to avoid algebraic loops, but i can assure you my trainings converge on a complex model. The difference is that i use a continuous environment, with a discrete time agent, but in any case the agent is not learning the "wrong" policy.
The fact that "rewards are delayed" does not sound correct to me, since the reward computations follow the agent's sample time, so it should not have anything to do with the environment.
