@_ryantlee

ML/AI researcher

 Los Angeles, CA
Joined December 2024
  • Tweets19
  • Following53
  • Followers52
Ryan Lee retweeted
Jeff Shen@jeff_shen_5h
Announcing FlashMap: a new speculative decoding algorithm based on flow maps! >6x faster inference on Qwen3-8B 🧵 Project page: jshen.net/flashmap arXiv: arxiv.org/abs/2610.24365 w/ @Jay41969851 @_ryantlee @jonathanmay @cosmo_shirley
5
16
92
3,170
Introducing GameBoyWorlds, a new challenge testbed for self-improvement in embodied video games: Paper: arxiv.org/abs/2609.32093 Website: dhananjayashok.github.io/Gam…
7
31
2
162
310,115
My first PhD paper, 'Sparse Layers are Critical to Scaling Looped Language Models,' is accepted at NeurIPS 2026! Looping & early exits enables adaptive depth; MoE makes it scale. First step toward compute-adaptive LMs. w/ @jbiloki @edwardjhu @jonathanmay arxiv.org/abs/2605.09165

Sparse Layers are Critical to Scaling Looped Language Models

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as...

arxiv.org
5
19
2
219
29,845
Ryan Lee retweeted
Ryan Lee@_ryantleeSep 2
Replying to @wangsw5653
Nice work, thanks for the cite in section 2! Would love a row in Table 1: we match (active) params for Looped-MoE architectures against baselines, so forward FLOPs match, and effective depth is fixed at 16 (8x2 vs a 16-layer Base), so KV matches too. arxiv.org/abs/2605.09165

Sparse Layers are Critical to Scaling Looped Language Models

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as...

arxiv.org
1
1
1
358
Ryan Lee retweeted
Edward Hu
@edwardjhuSep 1
How does one RL post-train a 397B model for long-horizon knowledge work? 👩‍💼 We share every step we took to bring Qwen 3.5 397B from 16.1% Pass@1 to 27.3% on APEX-Agents using DPPO, including final models weights and the full training script🚀 This is the first of many works from Mercor Research on open model training research. Full blog: mercor.com/blog/training-fro… Source code: github.com/Mercor-Intelligen…
16
117
24
791
156,892
After about 2 years, we are proud to release Gecko, an efficient architecture that improves upon Megalodon, with capability of efficiently and inherently processing sequences with unlimited context length. One of the most important idea in Gecko is Adaptive Working Memory(AWM), implemented using a linear attention mechanism with a position-aware online softmax activation. Notably, AWM globally compresses information into memory, rather than discarding historical information through forgetting. In a controlled head-to-head comparison with Llama2 and Megalodon, Gecko achieves better performance in the scale of 7B and 2T training tokens. Gecko achieves 1.68 training loss, vs. 1.67 of Llama2-13B, with half number of parameters on 2T tokens. Paper: arxiv.org/abs/2601.06463 Code: github.com/XuezheMax/gecko-l…
3
24
3
145
21,764
Load more