Hi Federico,
It seems like you are working with Proximal Policy Optimization(PPO) and want to know more about the "NormalizedAdvantageMethod".
Normalization of the advantage function is beneficial in circumstances where the scale of the advantage values varies significantly across different parts of the state space or throughout training. It can help in variance reduction as well as faster training and optimal reward scaling.
For the second part, the choice between the "moving" and "current" normalization depends on the problem and environment. A few cases where "moving" might be a better option than "current" can be:
- Non-stationary environments: In cases where the environment might change over time, "moving" normalization (which is based on a restricted number of recent samples) can adapt more quickly to the changes than using "current" normalization. A common example can be stock market trading data.
- Large batches: In the case of large batches, "moving" normalization makes more sense as it can adapt faster than "current" normalization. It can also be useful in case of memory or computation constraints.
There can also be other use cases for "moving" normalization as it can smooth out the learning updates by providing a more consistent reference for advantage estimates over consecutive batches.
It might seem that "current" normalization should always perform better because it uses all available data, "moving" normalization can provide a more robust and adaptive reference for the advantage function, which can be particularly useful in dynamic or complex environments.
I hope it resolves your query!
