Building agents that reason, adapt, and act with RL and friends!
Joined December 2012
- Tweets637
- Following1.8K
- Followers839
- Likes1.7K
Exciting time to be at Modal 🚀 i'm hiring interns & full-time researchers in RL, ML, and large models to join our **new** research team. apply on our website or reach out to me directly (contact info on my website)!
Today we're announcing that Modal has crossed $1B in annualized run rate, as well as raised a $750M Series D at a $15.75B valuation, led by @Accel.
Since our Series C, usage has tripled and we’ve grown more than 15x year-over-year.
We are deeply grateful to the developers and companies building on Modal. There's still so much more to build, and we can't think of a better time to be building software.
this is an awesome effort in the right dirextion and many congrats on starting this. Happy to help with your effort in any way that I can.
would dagger-style training offer another perspective here? would comparison make sense? check out this paper which is very relevant to yours where SFT issues r studied from exposure-bias perspective ( no expert data tho) arxiv.org/abs/2410.14655 2/2
Seems like interesting work. wouldn’t these SFT issues more of exposure bias (training != inference) problem? why use the off/on-policy framing here than exposure bias? 1/2
pretty sure, you don't mean this method 😉arxiv.org/abs/2307.12950
If you guys could fix some of the Antigravity issues, it'd help a lot. E.g, opening Codex extension closes Agent window, so you basically have to choose between Codex & Gemini Agent instead of using both at the same time. Claude Code works fine alongside it, but Codex doesn’t.
It covers other methods like SFT, DPO too. But RL in general is more complex to get working and has more pieces to it, which is why there's so much focus on it.
Predefined clipping values could be the source of this? Dropping them all together and assigning them based on training dynamics not only makes things adaptive but also addresses this to some extent. Tho haven't studied this here
arxiv.org/abs/2605.12380
Rasool Fakoor retweeted
RL is hot and has a reputation of being difficult to control. That is, unless you have the right parametrization. Off-policy RL, in particular struggles with the fact that your roll-out looks different from what you're actually optimizing. This can happen if you roll out with a more powerful model, or if you want to roll out in FP8, while you're training in FP16, or if you update asynchronously, or in other words, almost all the time in practice.
Too many ideas die at the edge of the LLM/VLM/VLA training framework. Not anymore.
Excited to release FeynRL: built to help you understand, modify, and build new RL methods without fighting the stack.
github.com/FeynRL-project/Fe…
Try it, ⭐ it, and send feedback.
tbh wish I had your patience and rigor to do something like this for RL. There’s so much to clarify about RL and where things went quite wrong: ablation studies being presented as new RL alg, old concepts getting reinvented under new names, and so on...
This is exactly why open-source frameworks like FeynRL matter more than ever. Open weights are NOT enough. Open AI needs open training recipes, alg, etc.
You should be able to understand training, modify it, & build new algorithms, optimizers, recipes.
github.com/FeynRL-project/Fe…
yeah, real pain is not just RL-training the model, but knowing why it breaks and when it actually works. Most frameworks do not help much cuz they blur the line between system and alg. FeynRL tries not to. github.com/FeynRL-project/Fe… Would love your feedback if you try it.
@ChangHao564792d
Great paper. We also studied and proposed some form of DAgger style training for LLM but for SFT using scheduled sampling.
arxiv.org/abs/2410.14655
Rasool Fakoor retweeted
Higgs Audio v3 TTS is here.
Built for voice AI that speaks, not just reads:
• 100 languages with single-digit WER/CER
• inline control over emotion, style, prosody, and sound effects
• API, Workspace, and open weights
• Blog 👉 boson.ai/blog/higgs-audio-v3…
Watch the demo 👇
I'd add github.com/FeynRL-project/Fe… to your list!
I've implemented it in very modular & clean way so system stays systems and algorithm stays alg. It helps to understand how RL training works without requiring understanding entire stack.
Take a look and you will see the difference!
"agentic" should not require fundamentally different training stacks, as long as the framework easily supports env. Checkout github.com/FeynRL-project/Fe… It is a clean & modular RL framework. We'll add env example soon, but the core training/rollout are the same.
U r welcome to contribute tho!