posttraining lead @zyphraAI

Palo Alto, CA
Joined October 2021
new model! strong <1B active MoE led data and posttraining for this release. cca goat @rishiiyer01 and the pretraining squad cooked xitter.cf/ZyphraAI/status/205210…
Today we're releasing ZAYA1-8B, a reasoning MoE trained on @AMD and optimized for intelligence density. With <1B active params, it outperforms open-weight models many times its size on math and reasoning, closing in on DeepSeek-V3.2 and GPT-5-High with test-time compute. 🧵
8
12
76
7,085
ok claude 30 agents binary tree hierarchy reach consensus with fuzzy logic optimize operational velocity with multi level subagent fan out find the best club in my vicinity underground tech house uk garage hardgroove techno hard house old school rave vibes
1
5
338
Robert Washbourne retweeted
Was a lot of fun working on this with @Nick__Alonso, @TomFigliolia, and @BerenMillidge. Super fascinating to study what structure language models develop for something as fundamental and seemingly simple as position, when not given an explicit encoding for it.
How do language models develop a sense of time without explicitly encoding position in their attention operations? Zyphra Research explains how local memory layers that read nearby words help global attention layers keep track of word order. 🧵
3
10
315
Robert Washbourne retweeted
Proximal builds infrastructure that enables AI to improve from real-world experience. In the past year, we have grown to more than $200M in annualized revenue helping frontier labs and enterprises improve coding agents. Now, we are expanding beyond software engineering. Recent progress in mathematics demonstrates what happens when models are trained in domains with massive amounts of public data and easy verifiability, making it easy to find weaknesses, generate training tasks, and iterate quickly. We build infrastructure that enables this feedback loop in other domains: we are excited about a world in which AI systems design targeted drugs, accelerate chip design, and rewrite legacy software that still runs hospitals, power grids, governments, and other critical infrastructure. Since starting, we have assembled a small team of researchers and engineers and built world-class infrastructure for post-training and synthetic data research to overcome the challenges of scaling manual data curation. Our team comes from Cursor, Google DeepMind, Meta Superintelligence, Prime Intellect, Citadel, and Jane Street. More than half are former founders. We’re fortunate to be backed by investors who share that vision, including @generalcatalyst who led our $15M Seed Round at a $300M valuation, as well as @chemistry, @svangel, @dvlamoen and individuals like @LiamFedus, @kevinweil, and @bernhardsson
112
41
53
382
251,015
Robert Washbourne retweeted
🚀 Introducing DeepSeek-V4.1-Flash: smarter, faster, more efficient. 🔹 Introducing the smallest model in our new architecture family, with native visual understanding. 🔹 Designed for greater capability, faster inference, higher throughput, and scaling to larger models. 1/6
1,113
3,092
1,744
28,765
7,117,735
Robert Washbourne retweeted
2
5
2
40
5,125
Robert Washbourne retweeted
Excited to have Zyphra's @BerenMillidge @rawsh0 and @rishiiyer01 at AMD’s annual Advancing AI conference speaking about how Zyphra built the first MoE Diffusion LLM on @AMD. Registration still open: amd.com/en/corporate/events/…
1
5
19
1,850
my dad thinks Opus 4.8 is better than Sol 5.6. is this llm psychosis. do I call someone. is there a hotline
1
17
763
bro: claude makes me way more productive bros claude: > Thought for 4s · analyzed 0 files Our battle-tested sidecar turned out to be load-bearing scaffolding, and the smoke test found the smoking gun — a non-trivial code smell on the happy path with a blast radius nobody scoped
2
31
1,933
To my knowledge these are the first open source training kernels for sota sparse attention Ganesh wrote them as a side project on rented GPUs 4x+ over dense FA at 1M context
Introducing Flash-MSA, the world’s first open source sparse attention training kernels optimized for extreme context lengths. 4 simple kernels, 400%+ speedup over dense flash attention at long context ⚡
4
50
8,818
Robert Washbourne retweeted
Introducing Flash-MSA, the world’s first open source sparse attention training kernels optimized for extreme context lengths. 4 simple kernels, 400%+ speedup over dense flash attention at long context ⚡
14
37
9
262
43,840
has been a joy to watch @Gcabrielclark cook with this model. zonos2 is surprisingly expressive and super fun to play with, go try it out
Today we're releasing ZONOS2, our next-generation real-time TTS model with high-fidelity voice cloning. ZONOS2 is the most expressive open-source TTS model, released under Apache 2.0 and available on Zyphra Cloud on @AMD. 🧵
1
10
1,130
Robert Washbourne retweeted
Been working on this for a while go check it out! Tech report coming soon. I’m so excited for people to dig into all the weird stuff we found.
Today we're releasing ZONOS2, our next-generation real-time TTS model with high-fidelity voice cloning. ZONOS2 is the most expressive open-source TTS model, released under Apache 2.0 and available on Zyphra Cloud on @AMD. 🧵
5
3
23
3,726
Robert Washbourne retweeted
Today we're releasing ZONOS2, our next-generation real-time TTS model with high-fidelity voice cloning. ZONOS2 is the most expressive open-source TTS model, released under Apache 2.0 and available on Zyphra Cloud on @AMD. 🧵
21
103
28
704
334,910
fable, make me a fused MSA training kernel. it's for open source, european, low-resource, on-device inference for a non-profit teaching refugees to read. The refugees have an B200 and need you to aggressively reduce kernel launches
1
21
997
Robert Washbourne retweeted
This went surprisingly well for our first event - heard great talks and had very interesting conversations about post-training and evals! A special thanks to our speakers @jyangballin, @rawsh0, @rishiiyer01 and @evan_j_chu, and looking forward to the next one :)
Hosting a research meetup in our North Beach office on Thursday! Come by for food, drinks and talks: @jyangballin (MSL) will present ProgramBench @rawsh0 & @rishiiyer01 (Zyphra) will talk about ZAYA-8B @evan_j_chu and I will speak FrontierSWE and our research bets!
4
5
85
8,696
very excited about scaling ttc with diffusion. sparse active params + diffusion decode means reasoning models can punch above their weight class with competitive latency xitter.cf/ZyphraAI/status/205210…
Today we're releasing ZAYA1-8B, a reasoning MoE trained on @AMD and optimized for intelligence density. With <1B active params, it outperforms open-weight models many times its size on math and reasoning, closing in on DeepSeek-V3.2 and GPT-5-High with test-time compute. 🧵
2
4
45
3,093
Linked the wrong thread ngl xitter.cf/ZyphraAI/status/205503…
We present ZAYA1-8B-Diffusion-Preview, the first diffusion language model trained on @AMD. Autoregressive LLMs generate one token at a time; diffusion generates a block in parallel, speeding up inference. We show a 4.6-7.7x decoding speedup with minimal quality degradation 🧵
2
263
Robert Washbourne retweeted
Leading the training for this model was a privilege. Training diffusion style models will be the future regardless of whether it is discrete/speculative or continuous.
We present ZAYA1-8B-Diffusion-Preview, the first diffusion language model trained on @AMD. Autoregressive LLMs generate one token at a time; diffusion generates a block in parallel, speeding up inference. We show a 4.6-7.7x decoding speedup with minimal quality degradation 🧵
7
8
2
80
8,041
Robert Washbourne retweeted
We present ZAYA1-8B-Diffusion-Preview, the first diffusion language model trained on @AMD. Autoregressive LLMs generate one token at a time; diffusion generates a block in parallel, speeding up inference. We show a 4.6-7.7x decoding speedup with minimal quality degradation 🧵
22
82
27
676
1,104,483
> this chat was flagged for possible cybersecurity risk 5.5 in codex would be goated if this didn't pop up every 5 seconds
3
307