Hi Federico,
From your question I infer that you are concerned with the actual implementation of how the minibatch is constructed from all the updates received from the workers. During parallel synchronous training, the client collects samples from all the workers but the exact method of that is dependent on various factors like the algorithm, the settings used during the training and other specified options. The documentation does not provide implementation-level details into the process.
What you have described would be the ideal behaviour for the client, to sample equal number of experiences from each worker. But this is not always practical. Some prominent factors to consider include the following:
- Experience Variability: Not all workers may generate the same number of experiences due to the variability in episodes. Some episodes might end quicker than others based on the policy's actions and the environment dynamics.
- Sampling Strategy: The actual implementation of the minibatch sampling strategy might not enforce strict equality in the number of experiences sampled from each worker.
If you require the ideal behaviour for your application, you can simulate it manually through custom training loop. But this might not always be a good choice as the ideal behaviour is not practical in many cases.
I hope this helps.
