Arash Tavakoli (@arshtvk) header
Arash Tavakoli (@arshtvk) profile picture

Reinforcement Learning, Staff Research Scientist @RiotGames. Spent time @MPI_IS, @ImperialCollege, @UCL, @USC, @GeorgiaTech, @Microsoft (MSR), @UAlberta (RLAI).

Los Angeles, CA
Joined October 2016
Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)Photo by Arash Tavakoli (@arshtvk)
Excited to share our new TMLR paper: “Learning in Complex Action Spaces Without Policy Gradients.” A colorful collaboration between 🔴 yours truly (@RiotGames), 🟢 Sina Ghiassian (@Spotify), and 🔵 Nemanja Rakićević (@GoogleDeepMind). 📄 Paper: openreview.net/forum?id=nOL9… 💻 Code: github.com/atavakol/qmle 📌 We identify three principles that give policy gradients their effectiveness in complex action spaces and show they can equally empower value-based methods.
1
2
19
1,187
Arash Tavakoli retweeted
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node). GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100. Originally in 2019, GPT-2 was trained by OpenAI on 32 TPU v3 chips for 168 hours (7 days), with $8/hour/TPUv3 back then, for a total cost of approx. $43K. It achieves 0.256525 CORE score, which is an ensemble metric introduced in the DCLM paper over 22 evaluations like ARC/MMLU/etc. As of the last few improvements merged into nanochat (many of them originating in modded-nanogpt repo), I can now reach a higher CORE score in 3.04 hours (~$73) on a single 8XH100 node. This is a 600X cost reduction over 7 years, i.e. the cost to train GPT-2 is falling approximately 2.5X every year. I think this is likely an underestimate because I am still finding more improvements relatively regularly and I have a backlog of more ideas to try. A longer post with a lot of the detail of the optimizations involved and pointers on how to reproduce are here: github.com/karpathy/nanochat… Inspired by modded-nanogpt, I also created a leaderboard for "time to GPT-2", where this first "Jan29" model is entry #1 at 3.04 hours. It will be fun to iterate on this further and I welcome help! My hope is that nanochat can grow to become a very nice/clean and tuned experimental LLM harness for prototyping ideas, for having fun, and ofc for learning. The biggest improvements of things that worked out of the box and simply produced gains right away were 1) Flash Attention 3 kernels (faster, and allows window_size kwarg to get alternating attention patterns), Muon optimizer (I tried for ~1 day to delete it and only use AdamW and I couldn't), residual pathways and skip connections gated by learnable scalars, and value embeddings. There were many other smaller things that stack up. Image: semi-related eye candy of deriving the scaling laws for the current nanochat model miniseries, pretty and satisfying!
321
599
97
7,289
1,320,382
Arash Tavakoli retweeted
Very nice video covering the ideas and history of reinforcement learning. Love the original footage from the 1960s, 70s, and 80s.
5
55
3
462
57,282
Arash Tavakoli retweeted
Standard reinforcement learning in raw tokens is a disaster for sparse rewards! Here, we propose 𝗜𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝗥𝗟: acting on abstract actions emerging in the residual stream representation. A paradigm shift in using pretrained models to solve hard, long-horizon tasks! 🧵
23
119
30
933
257,247
Arash Tavakoli retweeted
Amazing that @SchmidhuberAI gave this talk back in 2012, months before AlexNet paper was published. In 2012, many things he discussed, people just considered to be funny and a joke, but the same talk now would be considered at the center of AI debate and controversy. Full talk:
37
236
35
1,495
365,696
Arash Tavakoli retweeted
Magic happens when these three things come together. Do seemingly useless stuff based on your research taste and view of what's the right way, look for signs of life and then scale the hell out of it, thereby crushing benchmarks you didn't directly optimize for... and even better, also develop new benchmarks along the way.
I've observed 3 types of ways that great AI researchers work: 1) Working on whatever they find interesting, even if it's "useless" Whether something will be publishable, fundable, or obviously impactful, is irrelevant to what these people work on. They simply choose something that feels interesting, weird, beautiful, or off in a way they can't ignore. For many of these people, "interestingness" is also often strong research intuition for an important problem that hasn't fully materialized yet, but their ideas often end up being meaningful during the process of exploration. The canonical example for this in physics is Richard Feynman who got intrigued by the way that plates wobbled. He followed this curiosity on something that seemed like a useless endeavor, and it ended up feeding into deeper physics (and eventually won him a Nobel prize): "It was effortless. It was easy to play with these things. It was like uncorking a bottle: Everything flowed out effortlessly. I almost tried to resist it! There was no importance to what I was doing, but ultimately there was." The AI version of this that I've observed before is when someone obsesses over a "minor" failure case, a weird training dynamic, a small theoretical mismatch, or just something that most people think is pointless to chase down. These threads end up becoming interesting and impactful more often than you'd expect. The risk is that one can spend a long time on a pointless rabbit hole, but I've observed that the best researchers often have a very good sense for when an idea is a dead end vs. whether it's promising given more effort. 2) Working on what they feel extremely strongly is the "right" way to do something These people have a clear picture of how the field *should* progress, and they're willing to work on unpopular things to prove their vision. They'll commit to something that others think is wrong, premature, or not worth it. An interesting quantitative way of measuring this is the citation graph of a paper. If you see a paper that has been around for many years but only started getting cited a lot more in recent years, that means that they were early (and right!). An obvious example is diffusion, the first paper of which was as early as 2015 (Sohl-Dickstein et al., 2015) but the ideas only started getting real traction in 2021 or later. The failure mode here is getting stuck defending a pet theory long after it's been falsified. And there's obviously many examples in our community of people who do a lot of goal post shifting or beat a dead horse for many decades. But when these ideas are legitimately undervalued, they result in paradigm shifts instead of incremental progress. 3) Crushing SOTA There's a type of researcher who isn't necessarily the most "philosophically original" or creative, but they are extremely effective at pushing a system to its limits. You can give these people a pre-existing task and benchmarks, check in on them in a month, and they will have crushed SOTA. Obviously this is not about benchmark hacking or short term wins. It's a real skill to take a combinatorial space of noisy research ideas and papers and conduct a rigorous search and ablation process. I've also found that this type of researcher has great intuition about the field: a sense for which ideas will scale, which tweaks are meaningful, good values for hyperparameters, and quickly figuring out which papers are worth paying attention to. ————— I think that these archetypes are all concrete expressions of good "research taste". (1) is a taste for interesting questions, (2) is a taste for long term worldviews, and (3) is a taste for careful execution and science. The best researchers I know often have a preference for operating in one of these modes, but frequently weave in and out of each depending on the stage of the project.
15
1
133
29,413
Here’s how I see it: • DQN (and variants) stem from the Bellman optimality equation • Actor-critic critics use Bellman expectation equations — via (Expected) SARSA targets Even with function approximation, the fixed points we’re trying to approximate are Bellman solutions.
why are the bellman equations considered foundational or important today? arent they just a straightforward application of DP to solve a problem that only arises in extremely simplified cases that never occur in practice?
1
202
Replying to @karpathy
@karpathy I think it would be good to distinguish RL as a problem from the algorithms that people use to address RL problems. This would allow us to discuss if the problem is with the algorithms, or if the problem is with posing a problem as an RL problem. 1/x
9
38
15
413
178,087
Arash Tavakoli retweeted
Wish to build scaling laws for RL but not sure how to scale? Or what scales? Or would RL even scale predictably? We introduce: The Art of Scaling Reinforcement Learning Compute for LLMs
9
98
26
554
300,665
In the same way that genAI captures fine-grained structure in images, what does the fine-grained structure of the return distribution actually look like? New work with @chongyiz1, @perryadong provides some clues!
1/ How can we model the future rewards (returns) for RL agents? While prior methods round the returns into discrete bins or predict a finite number of quantiles, we use flexible models to predict the fine-grained structure of the full return distribution: pd-perry.github.io/value-flo….
5
31
5,229
Arash Tavakoli retweeted
Excited to release new repo: nanochat! (it's among the most unhinged I've written). Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI. It weighs ~8,000 lines of imo quite clean code to: - Train the tokenizer using a new Rust implementation - Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics - Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use. - SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval) - RL the model optionally on GSM8K with "GRPO" - Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI. - Write a single markdown report card, summarizing and gamifying the whole thing. Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc. My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved. Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.
669
3,305
731
23,894
5,824,282
Replying to @makingmusic
More on the philosophy of what it means to have a goal and to be intelligent in the attached 2-page paper: incompleteideas.net/papers/S…
5
7
1
60
4,118
Arash Tavakoli retweeted
New post: Beyond Advantage in Policy Gradient fatemi.github.io/posts/pg-ba… The advantage function is not a necessity in policy gradient methods. What matters is approximating Q(s,a) and using an action-independent baseline to reduce variance. GRPO does this only halfway!
5
1
20
2,787
Arash Tavakoli retweeted
I spent the past month reimplementing DeepMind’s Genie 3 world model from scratch Ended up making TinyWorlds, a 3M parameter world model capable of generating playable game environments demo below + everything I learned in thread (full repo at the end)👇🏼
94
267
42
2,375
222,999
This👇
Most RL for LLMs involves only 1 step of RL. It’s a contextual bandit problem and there’s no covariate shift because the state (question, instruction) is given. This has many implications, eg DAgger becomes SFT, and it is trivial to design Expectation Maximisation (EM) maximum likelihood solutions that do exactly the same as RL. Of course, RL and multiagent systems will be needed as the picture illustrates.
2
261
It all started with a proposal, 60 years ago! "A Proposal for Dartmouth Summer Research Project on Artificial Intelligence". jmc.stanford.edu/articles/da…
2
1
12
1,054
We're hiring Research Scientists in Tech Research @RiotGames! If you're passionate about shaping the future of games with AI and have research experience in RL, IL, or generative methods, we'd love to hear from you. DM me if interested.
1
3
20
2,035
RLC Keynotes are now live! Covering: Wanting vs liking, Agent factories, Theoretical limit of LLMs, Pluralist value, RL teachers, Knowledge flywheels... youtube.com/playlist?list=PL…
2
35
7
241
46,253
Arash Tavakoli retweeted
Introducing DINOv3 🦕🦕🦕 A SotA-enabling vision foundation model, trained with pure self-supervised learning (SSL) at scale. High quality dense features, combining unprecedented semantic and geometric scene understanding. Three reasons why this matters…
12
137
17
1,009
136,521
What do we talk about when we talk about the Bellman Optimality Equation? If we think carefully, we are (implicitly) making three claims. #FoundationsOfReinforcementLearning #sneakpeeks
4
1
36
2,737