So the whole issue that made me try get PPO working, and give up on ARS for a bit, is that I’m having trouble saving the policy to file, and then loading it back up.
https://stable-baselines.readthedocs.io/en/master/guide/examples.html#continual-learning
The current problem with the PPO version is that it’s just falling over in the reward direction.