Hello,
I cannot answer the first part of your question as I have not tried the waterfall grid world. I can however deal with the second part.
First off, the execution time for any reinforcement learning model is extremly variable. Recall that we are not updating weights based on a ground truth like deep learning. The algorithm is using the differential equations of motion(these specify the way velocity and acceleration change based on the environment and actions) that are provided in the environment and making time steps to propagate this object through the environment. If the agent is using the maxiumum amount of steps that you allow, it needs to calculate these differential equations each time. If the algorithm takes less steps per episode it will be quicker.
In terms of having the best CPU, if you're running on a mobile(laptop) platform odds are your hardware is thermal throttling(reducing computing power because CPU is too hot) after 20-30 minutes. This could slow it down a lot. If possible, switch to a desktop platform and get a good cooling fan.
The rocket lander is a demon I will admit. I've ran it a bunch and made the following conclutions:
- lower the learning rate, I've set mine to 1e-4
- Lower the clip factor, mine is 0.1
- increase mini-batch size, mine is 128.
Also remember that this is near bleeding edge computing science and is subject to just not work sometimes. I highly reconmmend banging your head against the table to learn and get better results, it's very satisfying.
Hope this helps,
RC
