🧬Fully Understand Evolution Strategies (ES) for LLM Reasoning ES can post-train LLMs without backprop. Our study finds that ES: 🔍 Explores higher Pass@K than GRPO 💥 Still sparse functional updates, with no catastrophic forgetting. ⚙️ Requires fewer samples as LLMs get larger We summarize the paper through 3 research questions 👇 1️⃣ Does ES exhibit the same post-training characteristics as GRPO? ES maintains broader reasoning coverage! Its population-based parameter-space exploration induces diverse reasoning behaviors: ES improves Pass@1 while achieving stronger large-K Pass@K, without the same entropy collapse observed in GRPO. The two are also complementary. 🔀 Sequential compositions—ES→GRPO and GRPO→ES—combine GRPO’s strength in Pass@1 with ES’s gains in Pass@K, yielding better Pass@1–Pass@K trade-offs. ➡️ ES preserves broader coverage; combining it with GRPO can capture both strengths. 2️⃣ Does ES necessarily cause catastrophic forgetting? No. Although ES induces substantial whole-model parameter drift, its task-relevant effects are concentrated in a small subset of larger-magnitude updates. Most parameter changes contribute little after perturbation cancellation, revealing surprising functional sparsity. Held-out capabilities also remain largely preserved under appropriate training settings. ➡️ Large parameter movement does not necessarily imply widespread functional change—or catastrophic forgetting. 3️⃣ What hyperparameter settings and estimators make ES effective and scalable? We find three practical takeaways: ✅ Z-score reward normalization is a key ingredient for effective ES training. ✅ Larger pretrained models require smaller population sizes for effective optimization. ✅ For discrete reasoning rewards, the two-point estimator commonly favored in zeroth-order SFT provides no advantage. 💡 Takeaway: ES is not merely a memory-efficient, gradient-free alternative to GRPO. Its broader reasoning coverage, functionally sparse updates, and distinct scaling behavior position it as a distinct paradigm for LLM reasoning post-training. 📄 arXiv:2608.27351

🚀 Will Agentic ESOpt Be the Future of Long-Horizon Agentic Fine-Tuning? Excited to share our new work on Arxiv. Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Memory Requirements As LLM agents become longer-horizon and more branching, conventional Agentic RL faces three growing bottlenecks: high training/memory cost and increasingly challenging long-horizon credit assignment. Our Agentic ESOpt approaches agent fine-tuning from the parameter space instead, targeting three advantages: ⚡ Model Scalability — only minimal inference-level GPU memory requirement 🔧 Flexibility — prompt/skill–parameter co-evolution ⏳ Long-Horizon Scalability — trajectory-level parameter attribution without horizon-wise decomposition There are impressive empirical results, and we believe ES may be a better-matched parameter fine-tuning paradigm as agents become larger and longer-horizon. 📄 Paper: arxiv.org/abs/2608.17310 🌐 Project: zz1358m.github.io/Project-Ag… 💻 GitHub: github.com/zz1358m/Agentic-E… 🤗 Models & Checkpoints: huggingface.co/collections/z… Code, training logs, and checkpoints have been fully open-sourced. Main takeaways ↓