Hi @Gaurav
Prioritized Experience Replay (PER) tends to outperform uniform experience replay in environments where rewards are sparse, delayed, or where the dynamics are non-stationary. However, in the case of quadruped robot locomotion, the rewards are consistent. We receive positive rewards at every time-step to avoid early termination and additional rewards for positive forward velocity.
Since rewards are given consistently at every time-step, the variance in the importance of different experiences is lower. This means that each experience contributes relatively equally to the learning process, making uniform sampling sufficient for effective learning. The continuous nature of the rewards also ensures that the agent receives regular feedback about its performance, reducing the need for prioritizing specific experiences.
Because of these reasons, you can observe that the reward plot with uniform experience replay is smoother compared to the reward plot using prioritized experience replay.
Hope this clears your doubt.
With regards,
Kaustab Pal
