Member of Technical Staff @bespokelabsai. PhD @CSatUSC in trustworthy AI/ML, previously intern @apple @amazon, ug @UT_Dallas

Joined August 2017
Include

Only show posts containing:

Exclude

Hide posts containing:

Time range
-
Minimum likes
Siddartha Devic ✈️ COLM 2026 retweeted
Today, we are announcing Nimble-v4 and Nimble-v4-max which are pushing the frontier of decision models, outperforming Jev and other variants. On the Decision Index public scores, nimble-v4-max and nimble-v4 score +10 and +6 against Jev 1.13, making nimble-v4-max the #1 model. Beyond benchmarks, we have done a lot of internal evals, and see meaningful improvements: 43% fewer mistakes than Jev, 50pp improvement on model routing decisions, being great at coding use cases, fact checking etc. The models are available via our API service. Go build nimbly: console.bespokelabs.ai
23
39
12
242
82,821
yes! I'll be at the @bespokelabsai booth from Noon-6pm ish tomorrow, feel free to swing by.
1
36
Hi friends, i'll be at COLM this week, please let me know if you are around and want to meetup! Will also be at our booth @bespokelabsai, happy to chat about opportunities we have on RL envs, post-training, and understanding agent behaviors.
4
1
42
1,206
Siddartha Devic ✈️ COLM 2026 retweeted
We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability. benchmarks.bespokelabs.ai/au… In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve. We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours. Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets. Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)
37
94
16
560
2,035,636
check out our new autoresearch benchmark!
We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability. benchmarks.bespokelabs.ai/au… In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve. We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours. Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets. Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)
2
15
740
This is my first substack post after finishing my PhD. It was definitely useful to clarify my own thoughts about how things are moving! Here's the full article: siddevic.substack.com/p/some…
5
115
Lastly, as a reason for optimism, I discuss a bit about how "superintelligent" coding agents have actually _increased_ collaboration between my coworkers and I, and how this may also happen for cs theory research.
1
1
119
I believe that a good chunk of theory research (at least over the next few months) may become using these tools to understand and develop entire new branches of theory, connect seemingly disparate branches, and quickly iterate on conjectures / proofs. Lean may be important here!
1
1
109
If theorems can be proven one-shot by AI, the "unit" of work that a researcher performs must change. I propose one such vision for a subfield I worked in during my PhD called "multicalibration" within the broader area of algorithmic fairness.
1
1
94
This is already happening to an extent with the submission process at ML conferences gating based on previous authorship: xitter.cf/thegautamkamath/status…
Some are understandably upset by the latter policy. Those on the other side may get it. TMLR submissions have 4x'd, largely fueled by single authors who have no prev ML papers. Sadly, due to AI, a (short-term?) downstream effect will be additional gatekeeping and credentialism.
1
142
Next, I believe that if we place less emphasis on papers, academia may further prioritize more traditional credentialism measures: name-brand institutions, status as academic micro-celebrities, or advisor name. This is a net negative for the democratization of any field!
1
3
116
My first argument is that the depreciation of the "academic paper" will be (near-term) destabilizing, given how slow institutions are at adapting. # of papers is exploding, and it is difficult to get 👀 on your work without platforms X/twitter. Partly why I chose not to postdoc!
1
215
Like (nearly) everyone else who worked on theory at some point in their academic career, I have some thoughts on the recent developments in AI and how it may impact the way that we perform and evaluate theoretical research.
2
31
2,151
Siddartha Devic ✈️ COLM 2026 retweeted
Daniel Litt
@littmathMay 27
STEM academia serves two closely intertwined purposes: the production of high quality science and the production of human capital. These two purposes feed into each other. The obvious direction is that we develop human capital by paying people to produce science. What is perhaps less obvious is that the very fact that human labor is used to produce science has historically been an important input to its quality. The goal of science is not simply to produce papers, but rather to produce good work--that a person is willing to spend months working on a paper is a (weak) witness to the fact that it has some minimum quality. If someone has a record of producing high quality work, that they wrote a paper is a stronger witness, since it was worth the opportunity cost to write it. If many people engage with it substantially, that is even stronger evidence. This is not to say that there isn't lots of low-quality work--there is, in fact a huge amount--but we have strong sorting mechanisms, admittedly using imperfect proxies (all depending on costly human labor!), to find high-quality stuff. Arguably the paper itself is not the primary product here; in many cases the primary product is actually the expertise developed over the course of producing it, which can then be applied to other questions. If you believe, as I do, that producing high quality science should be one of our fundamental goals, I think you’re obligated to embrace new tools that help one do so. Refusing to is a declaration that these outputs are not important. But I worry that we are not on track to automate the production of good work; rather, we are on track to automate the production of papers. We need new mechanisms to ensure that we are also producing good work, and to ensure that we are developing the human capital to engage with it.
26
70
12
706
76,770
Siddartha Devic ✈️ COLM 2026 retweeted
We’re thrilled to announce a $40M investment that will fuel our mission to make AI agents reliable. For the past two years, we've been heads-down doing world-class data curation research and shipping best-in-class reinforcement learning environments for training and optimizing AI agents. This funding lets us go a lot deeper on both. Thank you to our investors @Wing_VC, @MayfieldFund, @8vc, @thehousefund and our angels such as Jeff Dean, Dheeraj Pandey, Tristan Handy, and several others from Anthropic, OpenAI, Meta. And thanks to the frontier labs and enterprises we work with every day, for sharing our vision for a future where agents can run autonomously for weeks and months at a time. (more below)
10
18
14
144
101,358
Siddartha Devic ✈️ COLM 2026 retweeted
How can we train small agentic models that are highly capable of terminal use and coding? Announcing OpenThoughts-Agent + OpenThinkerAgent-32B, the strongest Qwen-3 based open-data agentic model: 44.8% avg across 7 agentic benchmarks! (1/n)
25
90
26
434
1,133,046
Siddartha Devic ✈️ COLM 2026 retweeted
I’m really excited to be @iclr_conf this week presenting our work on substructure-aware protein modeling! DM me if you want to chat about protein models, baking biological priors into ML architectures, and anything bioML for experimentalists! We built Magneton, a model-agnostic framework that distills decades of curated substructure knowledge into any pretrained encoder. We find that substructure signal is complementary to both sequence and structure as well as helps models generalize to unseen substructure types. Come say hi at our poster on Saturday from 10:30AM-1:00PM @ Pavilion 3 P3-#1006! This work is near and dear to my heart because it’s my first (co)first-author main conference paper from my undergrad. A heartfelt shoutout to my mentors Robert Calef, @manoliskellis, @marinkazitnik for all their support! Website: rcalef.github.io/magneton/ Paper: arxiv.org/abs/2512.18114
1
3
15
2,503
Siddartha Devic ✈️ COLM 2026 retweeted
New paper: Convergent Evolution: How Different Language Models Learn Similar Number Representations. Language models, classical word embeddings, and even raw token frequencies all develop the same Fourier features for numbers. But only some develop the underlying structure. 🧵
2
22
1
107
46,928