Building agents that reason, adapt, and act with RL and friends!
Joined December 2012
- Tweets637
- Following1.8K
- Followers839
- Likes1.7K
Rasool Fakoor retweeted
RL is hot and has a reputation of being difficult to control. That is, unless you have the right parametrization. Off-policy RL, in particular struggles with the fact that your roll-out looks different from what you're actually optimizing. This can happen if you roll out with a more powerful model, or if you want to roll out in FP8, while you're training in FP16, or if you update asynchronously, or in other words, almost all the time in practice.
Rasool Fakoor retweeted
Higgs Audio v3 TTS is here.
Built for voice AI that speaks, not just reads:
• 100 languages with single-digit WER/CER
• inline control over emotion, style, prosody, and sound effects
• API, Workspace, and open weights
• Blog 👉 boson.ai/blog/higgs-audio-v3…
Watch the demo 👇
Rasool Fakoor retweeted
New blog post: Effective Sample Size
Reweighting data to fix distribution shift kills bias but piles everything onto a few points. ESS measures how much data you actually have left, the dial for when a replay buffer goes stale.
alex.smola.org/posts/40-effe…
Rasool Fakoor retweeted
Shoutout to my fantastic co-organizers for making the first-ever workshop on RL Environments & Agent Evals such a success!
@rasoolfa, @anishathalye, @aagohary, @natashajaques, @TheAndiPenguin, @migballesteros, Aziza Mirsaidova, Priyaranjan Pattnayak, Ahmed Elgohary, Alina Gavrilov, Aparna Elangovan, Graham Horwood
Rasool Fakoor retweeted
Packed room to hear @alexgshaw and @ryan_marten break down how @harborframework grew into *the* framework for RL environments.
In our RLEval workshop at @CAISconf today, attendees tackled big open challenges in RLEs & Agent Evals + I shared the approach we take at @joinHandshake
Rasool Fakoor retweeted
Super useful resource, thank you for putting it together!
Researchers working on RLE design & Agent Evals might consider submitting papers / attending the first-ever Workshop in this area at the upcoming ACM Conference on AI and Agentic Systems:
rl-eval.github.io
Rasool Fakoor retweeted
Our organizing team is excited for a productive day of discussion with you on May 26:
@natashajaques, @TheAndiPenguin, @rasoolfa, @anishathalye, @migballesteros, @aagohary,
Aziza Mirsaidova, Priyaranjan Pattnayak, Ahmed Elgohary, Alina Gavrilov, Aparna Elangovan, Graham Horwood
Rasool Fakoor retweeted
This work was recently accepted by TMLR!
openreview.net/forum?id=RF6r…
Besides our main contributions in our previous post, below are our additional insights in this TMLR version when applying preference-based and unlearning-based methods to LLM math reasoning:
Can we make LLMs reason effectively without a huge inference time cost?
We show a powerful approach through learning and forgetting!
Our recipe:
1️⃣ Aggregate reasoning paths from diverse sources: Chain-of-Thought, inference-time search (Tree-of-Thought, Reasoning-via-Planning), classic algorithms (BFS, DFS)
2️⃣ Learn successful reasoning paths ✅ while forgetting failed reasoning paths ❌ at the same time, which we call Unlikelihood Fine-Tuning (UFT)
3️⃣ Small learning rate is crucial to preserve inference-time search capabilities
Results on challenging math games, Countdown & Game-of-24:
⚡180× faster inference than search-based baseline
📈Beats CoT and inference-time search (ToT, RAP)
📄 Paper: arxiv.org/abs/2504.11364
💻 Code & data: github.com/twni2016/llm-reas…
Work completed during my internship at @AmazonScience. Thank you to my co-authors @allen_a_nie @Sapana_007 @yaoliucs Huzefa Rangwala @rasoolfa!
Rasool Fakoor retweeted
Can we make LLMs reason effectively without a huge inference time cost?
We show a powerful approach through learning and forgetting!
Our recipe:
1️⃣ Aggregate reasoning paths from diverse sources: Chain-of-Thought, inference-time search (Tree-of-Thought, Reasoning-via-Planning), classic algorithms (BFS, DFS)
2️⃣ Learn successful reasoning paths ✅ while forgetting failed reasoning paths ❌ at the same time, which we call Unlikelihood Fine-Tuning (UFT)
3️⃣ Small learning rate is crucial to preserve inference-time search capabilities
Results on challenging math games, Countdown & Game-of-24:
⚡180× faster inference than search-based baseline
📈Beats CoT and inference-time search (ToT, RAP)
📄 Paper: arxiv.org/abs/2504.11364
💻 Code & data: github.com/twni2016/llm-reas…
Work completed during my internship at @AmazonScience. Thank you to my co-authors @allen_a_nie @Sapana_007 @yaoliucs Huzefa Rangwala @rasoolfa!
Rasool Fakoor retweeted
Excited to announce that our web agent paper, AgentOccam, has been accepted to ICLR 2025! 🏂🏂🏂 Huge thanks to all collaborators! 😊
Special thanks to my brilliant and considerate mentor, Yao @yaoliucs, for your constant guidance and encouragement! Sapana @Sapana_007 and Rasool @rasoolfa, your insightful support has been invaluable. Huzefa, your unwavering support as our manager has been instrumental in our success. Pratik and George, your invaluable suggestions have greatly enriched our work.
📸: a recent photo capturing some senses of the Chinese phrase "大隐隐于市".
#ICLR2025 #webagent
👾 Introducing AgentOccam: Automating Web Tasks with LLMs! 🌐 AgentOccam showcases the impressive power of Large Language Models (LLMs) on web tasks, without any in-context examples, new agent roles, online feedback, or search strategies. 🏄🏄🏄
🧙 Link: arxiv.org/abs/2410.13825
🧐 By refining the observation and action spaces, AgentOccam achieves a groundbreaking zero-shot performance, outperforming previous methods on the WebArena benchmark. This simple yet effective approach underlines the importance of aligning these spaces closely with LLM capabilities for enhanced efficiency. 📈
✨ Highlights:
- AgentOccam leads with a 29.4% improvement over state-of-the-art methods SteP, and a 161% boost in success rate compared to the vanilla agent. 🤖
- Achievements made possible without complicating the process with additional examples or strategies. 🚫
- All our replication work, prompts, and evaluator error rectifications are transparently shared in the appendix. 📚
🌟 Special thanks to my super brilliant and considerate mentor Yao and Rasool, our supportive manager Huzefa, and the invaluable suggestions and contributions from Sapana, Pratik, and George. Your guidance and support have been pivotal in this journey!
#AgentOccam #LLM #WebAutomation #AI
Rasool Fakoor retweeted
👾 Introducing AgentOccam: Automating Web Tasks with LLMs! 🌐 AgentOccam showcases the impressive power of Large Language Models (LLMs) on web tasks, without any in-context examples, new agent roles, online feedback, or search strategies. 🏄🏄🏄
🧙 Link: arxiv.org/abs/2410.13825
🧐 By refining the observation and action spaces, AgentOccam achieves a groundbreaking zero-shot performance, outperforming previous methods on the WebArena benchmark. This simple yet effective approach underlines the importance of aligning these spaces closely with LLM capabilities for enhanced efficiency. 📈
✨ Highlights:
- AgentOccam leads with a 29.4% improvement over state-of-the-art methods SteP, and a 161% boost in success rate compared to the vanilla agent. 🤖
- Achievements made possible without complicating the process with additional examples or strategies. 🚫
- All our replication work, prompts, and evaluator error rectifications are transparently shared in the appendix. 📚
🌟 Special thanks to my super brilliant and considerate mentor Yao and Rasool, our supportive manager Huzefa, and the invaluable suggestions and contributions from Sapana, Pratik, and George. Your guidance and support have been pivotal in this journey!
#AgentOccam #LLM #WebAutomation #AI
Rasool Fakoor retweeted
I’ll be presenting this work at CoRL 2024 in about a month. Let’s chat about sample-efficient robot adaptation!
Website: jessezhang.net/projects/extr…
Paper: arxiv.org/abs/2406.17768
Coauthors: @MinhoHeo, @LiuZuxin, @ebiyik_, @JosephLim_AI, @yaoliucs, @rasoolfa
Rasool Fakoor retweeted
How can robots efficiently learn **new tasks/in new settings**?
Introducing EXTRACT: a reinforcement learning (RL) framework that extracts a discrete + continuously parameterized skill library from offline data for efficient RL on new tasks!
Accepted to CoRL 2024: 🧵👇
Rasool Fakoor retweeted
Proud to release the first LLM from @boson_ai. Higgs-Llama-3-70B, built for characters and gameplay, trained on Boson-3 base. With great MMLU-Pro performance. boson.ai/higgs-opensource/
Rasool Fakoor retweeted
Offline RL is much harder than online RL or imitation learning as it needs to solve a sequence of counterfactual reasoning problems. That often gives an error of (1+\delta)^H, where delta is the one-step divergence of policy or extrapolation of Q and H is the horizon. 1/N
Rasool Fakoor retweeted
One common misconception about (deep) RL is that is was done by first defining some empirical loss as objective and then deriving model updating rules from GD, just like supervised learning. This is NOT the case for popular RL algorithms like policy gradient or TD-based. 1/N
Rasool Fakoor retweeted
'Flexible Model Aggregation for Quantile Regression', by Rasool Fakoor, Taesup Kim, Jonas Mueller, Alexander J. Smola, Ryan J. Tibshirani.
jmlr.org/papers/v24/22-0799.…
#quantile #quantiles #prediction
Rasool Fakoor retweeted
Happy to announce that our Task-Agnostic Continual RL work is accepted at CoLLAs 2023🚀
We made A LOT of changes since the last 🧵, e.g., adding 90k synthetic experiments and transformer-based approaches. So here's a new one ⬇️
5min video summary: youtube.com/watch?v=T0RYCuEC…
