Member of Technical Staff @bespokelabsai. PhD @CSatUSC in trustworthy AI/ML, previously intern @apple @amazon, ug @UT_Dallas

Joined August 2017
Siddartha Devic ✈️ COLM 2026 retweeted
Today, we are announcing Nimble-v4 and Nimble-v4-max which are pushing the frontier of decision models, outperforming Jev and other variants. On the Decision Index public scores, nimble-v4-max and nimble-v4 score +10 and +6 against Jev 1.13, making nimble-v4-max the #1 model. Beyond benchmarks, we have done a lot of internal evals, and see meaningful improvements: 43% fewer mistakes than Jev, 50pp improvement on model routing decisions, being great at coding use cases, fact checking etc. The models are available via our API service. Go build nimbly: console.bespokelabs.ai
23
39
12
242
82,821
Siddartha Devic ✈️ COLM 2026 retweeted
We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability. benchmarks.bespokelabs.ai/au… In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve. We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours. Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets. Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)
37
94
16
560
2,035,634
Siddartha Devic ✈️ COLM 2026 retweeted
Daniel Litt
@littmathMay 27
STEM academia serves two closely intertwined purposes: the production of high quality science and the production of human capital. These two purposes feed into each other. The obvious direction is that we develop human capital by paying people to produce science. What is perhaps less obvious is that the very fact that human labor is used to produce science has historically been an important input to its quality. The goal of science is not simply to produce papers, but rather to produce good work--that a person is willing to spend months working on a paper is a (weak) witness to the fact that it has some minimum quality. If someone has a record of producing high quality work, that they wrote a paper is a stronger witness, since it was worth the opportunity cost to write it. If many people engage with it substantially, that is even stronger evidence. This is not to say that there isn't lots of low-quality work--there is, in fact a huge amount--but we have strong sorting mechanisms, admittedly using imperfect proxies (all depending on costly human labor!), to find high-quality stuff. Arguably the paper itself is not the primary product here; in many cases the primary product is actually the expertise developed over the course of producing it, which can then be applied to other questions. If you believe, as I do, that producing high quality science should be one of our fundamental goals, I think you’re obligated to embrace new tools that help one do so. Refusing to is a declaration that these outputs are not important. But I worry that we are not on track to automate the production of good work; rather, we are on track to automate the production of papers. We need new mechanisms to ensure that we are also producing good work, and to ensure that we are developing the human capital to engage with it.
26
70
12
706
76,770
Siddartha Devic ✈️ COLM 2026 retweeted
We’re thrilled to announce a $40M investment that will fuel our mission to make AI agents reliable. For the past two years, we've been heads-down doing world-class data curation research and shipping best-in-class reinforcement learning environments for training and optimizing AI agents. This funding lets us go a lot deeper on both. Thank you to our investors @Wing_VC, @MayfieldFund, @8vc, @thehousefund and our angels such as Jeff Dean, Dheeraj Pandey, Tristan Handy, and several others from Anthropic, OpenAI, Meta. And thanks to the frontier labs and enterprises we work with every day, for sharing our vision for a future where agents can run autonomously for weeks and months at a time. (more below)
10
18
14
144
101,358
Siddartha Devic ✈️ COLM 2026 retweeted
How can we train small agentic models that are highly capable of terminal use and coding? Announcing OpenThoughts-Agent + OpenThinkerAgent-32B, the strongest Qwen-3 based open-data agentic model: 44.8% avg across 7 agentic benchmarks! (1/n)
25
90
26
434
1,133,046
Siddartha Devic ✈️ COLM 2026 retweeted
I’m really excited to be @iclr_conf this week presenting our work on substructure-aware protein modeling! DM me if you want to chat about protein models, baking biological priors into ML architectures, and anything bioML for experimentalists! We built Magneton, a model-agnostic framework that distills decades of curated substructure knowledge into any pretrained encoder. We find that substructure signal is complementary to both sequence and structure as well as helps models generalize to unseen substructure types. Come say hi at our poster on Saturday from 10:30AM-1:00PM @ Pavilion 3 P3-#1006! This work is near and dear to my heart because it’s my first (co)first-author main conference paper from my undergrad. A heartfelt shoutout to my mentors Robert Calef, @manoliskellis, @marinkazitnik for all their support! Website: rcalef.github.io/magneton/ Paper: arxiv.org/abs/2512.18114
1
3
15
2,503
Siddartha Devic ✈️ COLM 2026 retweeted
New paper: Convergent Evolution: How Different Language Models Learn Similar Number Representations. Language models, classical word embeddings, and even raw token frequencies all develop the same Fourier features for numbers. But only some develop the underlying structure. 🧵
2
22
1
107
46,928
Siddartha Devic ✈️ COLM 2026 retweeted
A reward model that works, zero-shot, across robots, tasks, and scenes? Introducing Robometer: Scaling general-purpose robotic reward models with 1M+ trajectories. Enables zero-shot: online/offline/model-based RL, data retrieval + IL, automatic failure detection, and more! 🧵 (1/12)
8
104
26
409
103,396
Siddartha Devic ✈️ COLM 2026 retweeted
Los Angeles the coldest place on planet earth right now
278
2,611
536
18,728
2,553,585
Siddartha Devic ✈️ COLM 2026 retweeted
LLMs are notorious for "hallucinating": producing confident-sounding answers that are entirely wrong. But with the right definitions, we can extract a semantic notion of "confidence" from LLMs, and this confidence turns out to be calibrated out-of-the-box in many settings (!)
20
83
7
576
51,619
Siddartha Devic ✈️ COLM 2026 retweeted
Announcing 🔭✨Hubble, a suite of open-source LLMs to advance the study of memorization! Pretrained models up to 8B params, with controlled insertion of texts (e.g., book passages, biographies, test sets, and more!) designed to emulate key memorization risks 🧵
2
41
12
130
50,447
Siddartha Devic ✈️ COLM 2026 retweeted
We found a new way to get language models to reason. 🤯 No RL, no training, no verifiers, no prompting. ❌ With better sampling, base models can achieve single-shot reasoning on par with (or better than!) GRPO while avoiding its characteristic loss in generation diversity.
73
248
54
1,704
280,826
Siddartha Devic ✈️ COLM 2026 retweeted
We (Moritz Hardt, @walesalaudeen96,@joavanschoren) are organizing the Workshop on the Science of Benchmarking & Evaluating AI @EurIPSConf 2025 in Copenhagen! 📢 Call for Posters: rb.gy/kyid4f 📅 Deadline: Oct 10, 2025 (AoE) 🔗 More Info: rebrand.ly/bg931sf
2
13
1
50
5,428
Siddartha Devic ✈️ COLM 2026 retweeted
🚨 Position paper alert! 🚨 LLM uncertainty quantification (UQ) has been explored with the goal of enabling better reliance on LLMs by humans. However, we argue that common LLM UQ practices are detached from this human-centric aspiration.😭😭 arxiv.org/abs/2506.07461
2
11
1
74
7,607
Siddartha Devic ✈️ COLM 2026 retweeted
Being a grad student is peak human existence. The end goal of all political and technological progress should be allowing everyone to be a grad student forever
64
514
137
9,021
534,923
Siddartha Devic ✈️ COLM 2026 retweeted
Are LLMs correlated when they make mistakes? In our new ICML paper, we answer this question using responses of >350 LLMs. We find substantial correlation. On one dataset, LLMs agree on the wrong answer ~2x more than they would at random. 🧵(1/7)
8
45
3
201
18,283
Siddartha Devic ✈️ COLM 2026 retweeted
Sparse autoencoders (SAEs) can be used to elicit strong reasoning abilities with remarkable efficiency. Using only 1 hour of training at $2 cost without any reasoning traces, we find a way to train 1.5B models via SAEs to score 43.33% Pass@1 on AIME24 and 90% Pass@1 on AMC23.
10
55
15
497
72,546
Siddartha Devic ✈️ COLM 2026 retweeted
Textual steering vectors can improve visual understanding in multimodal LLMs! You can extract steering vectors via any interpretability toolkit you like -- SAEs, MeanShift, Probes -- and apply them to image or text tokens (or both) of Multimodal LLMs. And They Steer!
1
14
1
53
10,300
Siddartha Devic ✈️ COLM 2026 retweeted
😋 Want strong LLM reasoning without breaking the bank? We explored just how cost-effectively RL can enhance reasoning using LoRA! [1/9] Introducing Tina: A family of tiny reasoning models with strong performance at low cost, providing an accessible testbed for RL reasoning. 🧵
2
67
6
399
44,251