wang retweeted
arxiv.org/abs/2606.29526
Reducing the problem of the training-inference gap. TIS (fengyao.notion.site/off-poli…) + rejection of checkpoint update when the gap between the rewards of training and inference is significant. The rejection requires additional rollouts (therefore it would be expensive), but the formulation could be interesting. Or it could be better to pursue true on-policy training.
wang retweeted
We would like to express our gratitude to RethinkLab of Shanghai Jiao Tong University and the Roll Team at Alibaba ATH Group for their support. Special thanks to @ProfYanJunchi and @weixunwang for their supervision and discussion😃😃
wang retweeted
Agentic RL environments are becoming critical. We integrated OpenReward (openreward.ai) into Alibaba’s ROLE (alibaba.github.io/ROLL/).
Details: github.com/alibaba/ROLL/pull…
wang retweeted
Are you also struggling with RL on long-horizon, high-difficulty agentic tasks, especially when positive rewards are sparse? Check out the latest blog from the ROLL team: warm-pajama-44a.notion.site/…
wang retweeted
My team and I hit some challenges doing RL training in terminal environments, so we wrote a blog sharing what we learned. We opened with two memes: one about researchers jumping from RLVR to Agentic RL, and another showing the chaos when RL training fails and no one knows why.
wang retweeted
The Bitter Lesson Behind Building Agentic RL in Terminal Environments
This blog post summarizes our practical experience over the past three months working on Agentic RL.
For more details, please refer to: faithful-almanac-add.notion.… #LLM #RL #Agent #AgenticRL
wang retweeted
Loved this breakdown — thanks for taking the time
It really does feel like a big step forward for open-source agentic training infrastructure!
Introducing ALE — a full-stack Agentic Learning Ecosystem that closes the loop from
execution → feedback → learning.
Three components power this loop:
• ROCK runs large-scale sandboxed execution to gather reliable trajectories.
• ROLL scales post-training with asynchronous rollouts and RL optimization.
• iFlow CLI keeps training and deployment workflows consistent end to end.
Built on ALE, we also release ROME — a production-ready agentic model trained on 1M+ real trajectories.
With its low barrier to serving a 30B model, you can build your own “super ROME” — drop your ideas, thoughts or usage feedback below
For more updates, follow us @FutureLab2025
wang retweeted
Check out our new work: “Let It Flow: Agentic Crafting on Rock and Roll” — introducing ALE, an open Agentic Learning Ecosystem with ROLL, ROCK, and iFlow CLI to streamline Agent LLM development from training to deployment, plus ROME, a production-ready agentic model trained on 1M+ real trajectories using our novel IPA algorithm that optimizes credit assignment at the semantic interaction level. Built for the community, battle-tested in practice! 🔗 arxiv.org/abs/2512.24873
wang retweeted
🚨 Chinese researchers just published a paper that destroys every AI agent startup pitch deck.
It's called ROME + ALE, and it exposes why every "AI agent company" you've heard of is building on quicksand.
Here's what nobody's talking about:
wang retweeted
🏆 Excited to share GEM received the Outstanding Paper Award @SEAWorkshop of #NeurIPS2025 . What a great way to wrap up this amazing neurips journey!
Huge thanks to the workshop committee and organizers for the recognition.
Grateful for our incredible collaborators and advisors who made this project possible. Thanks to everyone involved! 🎉
Congrats to the following paper authors attaining Outstanding Paper Awards at @SEAWorkshop!
GEM: A Gym for Agentic LLMs
Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Haotian Xu, Simon Yu, Chenmien Tan, Shaopan Xiong, Weixun Wang, Bo Liu, Hao Zhu, Weiyan Shi, Diyi Yang, Wee Sun Lee, Min Lin
wang retweeted
🧠 An Agent’s intelligence is limited only by the scale and diversity of the worlds it can explore.But traditional setups cap you at one machine—until now.
Meet ROCK🎸🎸: Reinforcement Open Construction Kit—developed by FutureLivingLab(@weixunwang ) at Alibaba Group.@AlibabaGroup
Built on @raydistributed (@robertnishihara ), it turns a massive cluster into a single elastic “environment pool.”
✅ Scale from 1 → 10,000+ sandboxes in minutes
✅ Mix homogeneous & heterogeneous environments for richer training
✅ Focus on your agent—not infrastructure
ROCK isn’t just scaling—it’s redefining what’s possible in Agentic AI.
👉 Try it: github.com/alibaba/rock
🚀Train your first Agent in just 5 minutes :
alibaba.github.io/ROCK/docs/…
#AI #ReinforcementLearning #Agent
wang retweeted
🚀 Excited to share our latest work in RL4LLM system.
🎉 ROLL Flash enables fully asynchronous overlap of generation, interaction, rewards, and training through Fine-grained Parallelism and Rollout–Train Decoupling.
1) 2.24× faster on RLVR; 2.72× faster on agentic tasks
2) Near-linear scaling: 8× GPUs → 7.6× throughput
3) Asynchronous Ratio balances utilization and sample freshness with minimal staleness cost
4) Supports off-policy algorithms (Decoupled PPO, TOPR, CISPO) with no performance loss
Join Us. Star, try, contribute—let's scale LLM RL together! 🌟
🔗 Paper: arxiv.org/abs/2510.11345
💻 Code: github.com/alibaba/ROLL
#LLMs #ReinforcementLearning #RL4LLM #SystemOptimization #AgenticAI
wang retweeted
🚀 Happy to present our new work on LLM reasoning!
We show that: (1) Attention is a structured map of the model's reasoning logic, uncovering a preplan-and-anchor reasoning rhythm. (2) Aligning RL objectives with the model's intrinsic attention rhythm yields more transparent, fine-grained, and efficient optimization.
🧠 Key Reasoning Patterns in Attention
(1) Local Chunking: Near-diagonal sawtooth patterns indicate dense intra-chunk processing. At chunk boundaries, the model performs long-range context retrieval (often with higher entropy), which guides subsequent generation.
(2) Global Anchor Planning: Sparse, high-influence anchor tokens exert broad control over later tokens. Perturbing these anchors significantly disrupts downstream reasoning.
(3) Preplan-Anchor Coupling: A stable temporal rhythm emerges: the model first emits a "preplan" token, then anchors a core semantic node, repeatedly structuring the reasoning trajectory.
⚙️ RL Innovation
We introduce a dynamic reward redistribution mechanism guided by attention-derived reasoning structure:
(1) Preplan Guidance: Boosts tokens that guide local chunks and enable long-range referencing.
(2) Anchor Enhancement: Prioritizes optimization of globally influential semantic anchors.
(3) Coupling Alignment: Reinforces the temporal coordination between preplans and anchors to solidify structured reasoning.
HuggingFace Link: huggingface.co/papers/2510.1…
arXiv Link: arxiv.org/abs/2510.13554
#LLMs #artificial_intelligence #RL4LLM
wang retweeted
#LLMs #artificial_intelligence #RL4LLM
🚀 Happy to present our work: Asymmetric Proximal Policy Optimization: Mini-Critics Boost LLM Reasoning. We re-examine PPO in LLM domain and find 3 key insights:
👀 (1) The critic might serve as a natural safeguard for stable policy training.
⚡️ (2) Training smarter reasoning agents does not require a giant critic—value estimation capability is not equivalent to model size.
😮 (3) Critic signals can further guide the reconstruction of the policy loss objective itself.
I'll go step-by-step through this thread👇.
wang retweeted
Excited to share our #RL_for_LLM paper: "Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning"
We conducted a comprehensive analysis of RL techniques in LLM domain!🥳
Surprisingly, we found that using only 2 techniques can unlock the learning capability of LLMs.😮
wang retweeted
Also has been integrated into OpenRLHF (github.com/OpenRLHF/OpenRLHF…), ROLL (github.com/alibaba/ROLL/issu…) and slime (github.com/THUDM/slime/pull/…). Now these mainstream RL frameworks (all from China!) all have supported GSPO! 🥳
🧠 For Everyone:
🎓 Researchers:Single-GPU support, modular pipeline, RL loops you can tweak in seconds. Fast iterations → fast papers.
🛠️ Engineers:Reward library, plug-and-play env routing, built-in LLM-as-judge evaluator. Fast prototype → fast deploy.
1
6
548
🏢 Enterprises:Scale to 200B+, support long-running training jobs across GPU clusters. Max out resource usage, minimize cost.
📢 Try it now →🔗 GitHub: github.com/alibaba/ROLL
📄 Tech Report: arxiv.org/abs/2506.06122
🌟 Scale smarter. Train faster. Research deeper.
🌟 Let’s roll.
