Berkeley, CA
Joined July 2020
Vin Howe retweeted
i wish more people in ai just found the models intrinsically interesting without an agenda. i remember back when i was embedded in the linguistics crowd, i was very confused why nobody i talked to seemed to find deep learning or later GPT-3 interesting. don't these people care about language understanding and linguistic production? so why are they ignoring and dismissing this? eventually i gave up and found tpot and the rationalists and mechinterp and everything i do now, and it's - to be clear - much better. but i still sometimes look around and see so many people doing research because they care about safety and x-risk, or steering capabilities because they want the singularity to go well, or posting blog posts about ai pause because they're worried about doom, or grinding gpu kernels or yelling at claude in a loop because they want to escape the permanent underclass... those are all well and good, i can respect (at least some of) those motivations, and many of them intersect with finding the models at least instrumentally interesting - but i really just find the models interesting, terminally. it's what got me into this, and everything else is kind of a side project. i just want to know how and why the models do what they do. i'm not without my Theories and my biases - it especially frustrates me how, per @norvid_studies, "people get used to anything in two weeks" and start kvetching about things that a few years ago would have been Fantastic Science Fiction - but ultimately, when i write, it's coming from a place of wonder about how these things work.
86
88
18
1,051
45,407
Preprint 🧵! How compartmentalized are LLMs? For data in different formats (English/Chinese, Wiki/Q&A), how much transfer occurs? We provide evidence that LLMs can struggle with this sort of transfer, with consequences like sample inefficiency and capacity competition.
3
3
2
13
2,434
We build on existing work showing that frontier performance on all sorts of transfer is more inconsistent than we might hope, especially after learning from trillions of tokens: xitter.cf/NitCal/status/20263003… @NitCal xitter.cf/omerNLP/status/1907058… @omerNLP arxiv.org/abs/2408.10646 @LChoshen
Wanna check how well a model can share knowledge between languages? Of course you do! 🤩 But can you do it without access to the model’s weights? Now you can with ECLeKTic 🤯
1
1
2
448
The big labs are betting RL will unlock superhuman coding. But their infrastructure is closed, and OSS tooling doesn't support true online RL—just iterative batch optimization. We're releasing ARES to close that gap 🧵
Announcing ARES - our open-source Agentic Research and Evaluation Suite. ARES is built around 3 pillars (👇 see the thread) to make reinforcement learning for code agents easy. We’ve also found it to be incredibly useful for our own mech interp research.
9
28
14
219
38,803
Train a language model in your browser with WebGPU! I built a playground for training sequence models (Transformers, LSTMs, GRUs, vanilla RNNs) completely in your browser on synthetic tasks like sorting and simple natural language datasets like TinyStories. You can fiddle with 50+ experiment knobs to build your own model, which can be as big as you have the VRAM to accommodate. You don't have to install anything—all you need is a browser with WebGPU support. Check it out! Link to repo + blog post + features and technical details in the reply. 🧵
3
3
3
24
2,423
This project was inspired directly by: - @fleetwood___ Ratchet - @willdepue WebGPT - @dsmilkov, @shancarter TensorFlow Neural Network Playground - @kellerjordan0 Modded-NanoGPT and Muon - @xenovacom Transformers.js - @polodataclub Transformer Explainer - @brendanbycroft LLM Visualization - @karpathy ConvNetJS, micrograd, minGPT, llm.c
1
3
562
Thanks to: - @grantpitt0, who helped create the original idea, provided invaluable feedback, and helped me debug a few cursed numerical bugs. - @fleetwood___ for help with Ratchet (and pushing me to write a blog post). - @bgub_ for helpful feedback. 💜
2
385
Excited to share what I’ve been working on with @andykonwinski, @Mike_A_Merrill, and @lschmidt3 at Stanford & Laude. Introducing Terminal-Bench! A benchmark and framework to quantify how well AI agents accomplish complex tasks in a terminal environment. We believe that the terminal is a particularly powerful tool for agents because it provides a text-based low-level interface for operating a computer to an agent.
You’re unable to view this Post because this account owner limits who can view their Posts. Learn more
3
6
2
50
6,200
I'll give $1M to the first open source AI that gets 90% on this sweet new contamination-free version of SWE-bench - kprize.ai
32
115
17
638
102,279
My new piece in @HarvardBiz describes our work using AI to perform conflict mediation on social media, and how it inspired a new intervention by NextDoor which resulted in a 15% decrease in toxic content! hbr.org/2024/03/genai-could-…
3
17
1
72
7,422