Vin Howe retweeted
i wish more people in ai just found the models intrinsically interesting without an agenda.
i remember back when i was embedded in the linguistics crowd, i was very confused why nobody i talked to seemed to find deep learning or later GPT-3 interesting. don't these people care about language understanding and linguistic production? so why are they ignoring and dismissing this?
eventually i gave up and found tpot and the rationalists and mechinterp and everything i do now, and it's - to be clear - much better. but i still sometimes look around and see so many people doing research because they care about safety and x-risk, or steering capabilities because they want the singularity to go well, or posting blog posts about ai pause because they're worried about doom, or grinding gpu kernels or yelling at claude in a loop because they want to escape the permanent underclass...
those are all well and good, i can respect (at least some of) those motivations, and many of them intersect with finding the models at least instrumentally interesting - but i really just find the models interesting, terminally. it's what got me into this, and everything else is kind of a side project. i just want to know how and why the models do what they do.
i'm not without my Theories and my biases - it especially frustrates me how, per @norvid_studies, "people get used to anything in two weeks" and start kvetching about things that a few years ago would have been Fantastic Science Fiction - but ultimately, when i write, it's coming from a place of wonder about how these things work.
Preprint 🧵! How compartmentalized are LLMs?
For data in different formats (English/Chinese, Wiki/Q&A), how much transfer occurs? We provide evidence that LLMs can struggle with this sort of transfer, with consequences like sample inefficiency and capacity competition.
We build on existing work showing that frontier performance on all sorts of transfer is more inconsistent than we might hope, especially after learning from trillions of tokens:
xitter.cf/NitCal/status/20263003… @NitCal
xitter.cf/omerNLP/status/1907058… @omerNLP
arxiv.org/abs/2408.10646 @LChoshen
Preprint link: arxiv.org/abs/2605.19284
Super fun project. I'll be a fellow at @MATSProgram in Berkeley next month. Reach out!
Vin Howe retweeted
The big labs are betting RL will unlock superhuman coding. But their infrastructure is closed, and OSS tooling doesn't support true online RL—just iterative batch optimization.
We're releasing ARES to close that gap 🧵
Train a language model in your browser with WebGPU!
I built a playground for training sequence models (Transformers, LSTMs, GRUs, vanilla RNNs) completely in your browser on synthetic tasks like sorting and simple natural language datasets like TinyStories. You can fiddle with 50+ experiment knobs to build your own model, which can be as big as you have the VRAM to accommodate.
You don't have to install anything—all you need is a browser with WebGPU support.
Check it out! Link to repo + blog post + features and technical details in the reply. 🧵
This project was inspired directly by:
- @fleetwood___ Ratchet
- @willdepue WebGPT
- @dsmilkov, @shancarter TensorFlow Neural Network Playground
- @kellerjordan0 Modded-NanoGPT and Muon
- @xenovacom Transformers.js
- @polodataclub Transformer Explainer
- @brendanbycroft LLM Visualization
- @karpathy ConvNetJS, micrograd, minGPT, llm.c
Thanks to:
- @grantpitt0, who helped create the original idea, provided invaluable feedback, and helped me debug a few cursed numerical bugs.
- @fleetwood___ for help with Ratchet (and pushing me to write a blog post).
- @bgub_ for helpful feedback.
💜
Vin Howe retweeted
Excited to share what I’ve been working on with @andykonwinski, @Mike_A_Merrill, and @lschmidt3 at Stanford & Laude.
Introducing Terminal-Bench! A benchmark and framework to quantify how well AI agents accomplish complex tasks in a terminal environment. We believe that the terminal is a particularly powerful tool for agents because it provides a text-based low-level interface for operating a computer to an agent.
You’re unable to view this Post because this account owner limits who can view their Posts. Learn more
Vin Howe retweeted
I'll give $1M to the first open source AI that gets 90% on this sweet new contamination-free version of SWE-bench - kprize.ai
Vin Howe retweeted
My new piece in @HarvardBiz describes our work using AI to perform conflict mediation on social media, and how it inspired a new intervention by NextDoor which resulted in a 15% decrease in toxic content! hbr.org/2024/03/genai-could-…









