To summarize, we highlight 6 key takeaways from the success and design of OGPO: Takeaway #1: Even within the standard Actor-Critic template, simply increasing training time computation can improve sample efficiency by an order of magnitude. Takeaway #2: Better policy extraction alone can yield substantial improvements in training stability and sample efficiency. Takeaway #3: Surprisingly, full policy finetuning can increase action diversity and enhance policy exploration, despite the fact that one is only trying to maximize sparse reward (with no explicit entropy penalties!). Understanding this phenomenon is an exciting direction for future work Takeaway #4: Zero-order policy policy optimization can be incredibly effective given sufficient computation as it avoids unstable gradients through denoising steps or critics, improving performance on high-precision tasks. It also enables advantage-curation techniques, such as the conservative advantages Takeaway #5: Targeted interventions, like imitating successful trajectories and the modified “conservative advantages” are both reliable, preserve training efficiency, and ameliorate the need for task-specific hyperparameter tuning. Takeaway #6: While there are many options for fine-tuning a multi-step GCP (AWR, FPO, etc), simple PPO is the most effective, most stable, and requires the least amount of hyperparameter finetuning. (9/N)