Ventali (@Ventali) header
Ventali (@Ventali) profile picture

rl research & biophysics

San Francisco
Joined July 2021
Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)Photo by Ventali (@Ventali)
seeing this progress from @AnthropicAI makes me reflect on Valida - the zk prover my team built. it’s encouraging to see zk finding its place. i hope we still contributed something valuable along the way and helped move the field forward!
New Anthropic research: progress on provable inference for large language models. Here, we introduce FlashProve. Building on established cryptographic techniques, FlashProve allows developers to verify whether a given response came from a given AI model. It’s substantially more efficient than prior open-source methods for doing so. These proofs have a range of applications. They can, for instance, make it harder for attackers to hijack a chat, or enable safety evaluators to verify that the model they’re testing is the same one being served to customers. For our full write-up, see here: anthropic.com/research/prova… Further progress in provable inference will likely be an important part of improving trust in AI deployment. To that end, we're open-sourcing FlashProve’s code, here: github.com/anthropics/FlashP… We’re grateful to our collaborators: Alessandro Chiesa, @GiacomoFenzi, @gwberg95, @miranda__christ, Rahul Dalal (rdalal.github.io) and Ben Mayer (@transmissions11).
6
3
43
2,366
anyone knows what exactly did the model prove? how qcd can be completely reduced to yan-mills gauge theory? when/where can we find the proof and more details?
Statement of Yang-Mills Working Group, a scientific collaboration to interpret AI result related to the existence of quantum chromodynamics & completions of the Standard Model, including a subproblem proving the existence of the Quantum Yang-Mills theory. buff.ly/BwV89Tk
2
332
really cool benchmark from guanghan! most benchmarks test whether a model can solve a problem. witness-bench tests whether it can discover the rules of a problem through experimentation, hypothesis formation, and feedback. a much more interesting test of intelligence, especially for agents operating in unfamiliar environments.
WitnessBench is now on @FireworksAI_HQ’s Specialized Intelligence Index. It measures how effectively RL improves model performance in puzzle game environments with hidden rules. The SII shows where open, closed, and specialized models stand. Explore the results: fireworks.ai/specialized-int…
2
9
560
congrats!
Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next. We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them. Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up. I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay. I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
8
642
this is the proof abundance tao predicted at icm in one commit. proving is no longer the bottleneck. digesting is. some thoughts on a potential solution: xitter.cf/Ventali/status/2107703…
We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results. github.com/openai/math
2
2
17
1,793
openai released 722 math manuscripts today. 372 result families, ~4,000 problems posed, about three hours of compute each, from one internal model. not all are formalized yet; the readme says some unformalized results "could have issues." nobody has read most of them. this is the proof abundance terence tao warned about at icm, and it arrived in a single commit. github.com/openai/math tao's point: a result goes through generation, verification, exposition, publication, canonicalization before it counts as mathematics. ai is good at the first and getting good at the second. the other three are slow and human. the bottleneck isn't finding proofs anymore, it's digesting them. a lean-checked proof nobody understands is correct but useless, and openai's repo has 722 of them, some not even lean-checked. i think this post could be a solution. it attempts to turn "mathematical understanding" into something you can train on, and it points at a better reward than anything provers use today. oleg's idea here is to freeze a smaller model, fix its compute, hand it a theory, count what it can derive now that it couldn't before. it's a measurable version of understanding, and it's basically tao's digestion stage as a number. and that number is a reward. every prover we train today gets paid for "proof accepted." self-play setups pay the conjecturer by the prover's pass rate. both optimize for volume of provable statements, which is the stage we already have too much of. if the reward is halo delta instead, a proof is worth nothing on its own. it only gets credit when it finds structure that makes unseen theorems cheaper. that's compression, and nobody has shown how to train it at scale yet. the closest attempt is from @AIatMeta two weeks ago (patel, rammal, hayat, munos, kempe, "learning to discover interesting mathematics" @niketnpatel @KempeLab). they define a theorem's interestingness as proof length over statement length, train a 27b model to predict proof difficulty given premises, and grpo a conjecturer against that score. mathlib overlap drops from 92% to 31%. they also define a utility measure, lines of lean saved across the library when the theorem is added, and show it correlates with interestingness. that utility is a halo. but it's static, computed over an existing library, and they train on interestingness rather than on utility directly. oleg's version makes the halo relative to a bounded reasoner on unseen theorems and trains on it. that's the difference. rough shape would be: an explorer generates proofs at volume. a compressor turns piles of them into definitions and key lemmas, trained on halo delta. a frozen student is the instrument. the compressor is the model whose reasoning you actually care about. library-learning systems have mostly accumulated lemmas with little reuse under matched compute. my guess is that's because nobody has trained the compressor with a transfer reward. there was no signal to train on. this gives you one open questions: how big the student should be. whether halo delta on lean theorems tracks what mathematicians call insight etc. and the obvious ablation now that a baseline exists: does model-relative halo beat fair's static utility as a training signal
My utility as an individual AI researcher will be gone in a year. Writing math proofs will be automated even sooner. It was never about proofs. Science is about curiosity, creativity, and compression - via @SchmidhuberAI *Mathematicians Should Become Formal Theory Influencers* Mathematicians were the original influencers: they created conjectures and theories that others understood, cited, and built on. Difficult proofs often reflected deep understanding. But the proof itself was rarely the point. Frontier AI can already generate informal proofs much faster than humans can carefully inspect them. The scalable response is machine-checkable mathematics in systems such as Lean. AI-generated formal proofs may increasingly resemble compiler-generated machine code: essential, but rarely worth reading or writing by hand. That shifts mathematical work up the abstraction stack. Mathematicians become formal-math software engineers, working with AI on clean definitions, interfaces, key theorems, and reusable libraries rather than low-level proof details. Call everything derivable from a formal library within a fixed compute budget its mathematical halo. Humans and AIs should compress clusters of results into insightful, useful theories that expand the halo further. A great theory compresses many results, makes future results cheaper to derive, and exposes structure that humans and machines can understand and build on. That is why minimum description length is a useful lens on mathematical insight. Frontier math will eventually require huge compute budgets. At that point mathematics starts to resemble experimental science: scarce computational instruments, with researchers deciding which problems deserve them. Mathematicians and AIs should create conjectures and theories that expand the mathematical halo. Mathematics should become an enormous machine-checkable software project. *Become a Formal Theory Influencer* Inspired since 2010 by: people.idsia.ch/~juergen/art…
2
6
28
1,945
Ventali retweeted
We have seen a growing and concerning trend of startups supplying data to China. Today we are taking a public stance against this practice At fleet, we believe this represents one of the largest national security threats today and must be stopped. We must win the AI race for America and shape AI with the same democratic values that have shaped our nation Our full statement here: fleetai.com/supporting-ameri…
The AI arms race isn’t just being fought over Nvidia chips. Chinese labs are matching OpenAI and Anthropic by purchasing the exact same data from US vendors like @mercor and Surge AI (who also work with the US federal govt) forbes.com/sites/annatong/20…
5
21
1
136
15,008
as a field our understanding of what rl is actually doing to models is still surprisingly rudimentary. reward hacking is the obvious symptom: models exploit the gap between intended objective and measured reward, but we have weak tools for identifying the learned strategy itself, how it changes through training, and whether it survives distribution shift or evaluator awareness. the core methodological problem is isolation vs realism. toy environments give causal control but often suppress the behaviors we care about; realistic environments preserve them but make attribution extremely hard. i think the direction should be toward building an experimental science of objective formation under rl: checkpointing throughout training, perturbing reward structure / evaluator awareness / environment structure, then tracing when proxy strategies first emerge, what causes them to become stable, and whether the same internal mechanism survives as the environment becomes more realistic. ideally this gives us something closer to a phase diagram of rl behavior: when reward hacking emerges, what training conditions produce it, and which learned strategies are genuinely invariant across environments rather than artifacts of a particular eval. we should eventually be able to predict reward hacking before it becomes behaviorally obvious, distinguish genuine objective change from shallow policy adaptation, and understand which learned strategies remain stable across environments
Replying to @hilbertspaess
If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?
3
14
1,178
This isn't the most notable aspect of today's news, but on the user data issue, there are different kinds of *training on user data* with very different privacy/IP implications. Sadly, AI cos don't like to disclose what they're doing. - pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper - use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this - use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs" "De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.
Two things to distinguish: Did any human or agent look at user data as part of the Navier Stokes effort? No. Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company.
42
153
34
1,369
285,498
agreed that one important failure mode in verifier/task generation is mistaking difficulty for signal. a lot of “hard” tasks are just poorly constructed. they introduce noise, ambiguity, or arbitrary complexity instead of isolating the capability we actually want to measure. i think the missing piece is that most task generation lacks an explicit objective. before generating anything, we should be able to state: • what capability are we testing? • what specific failure mode are we targeting? • what evidence would convince us the model improved? then every task should be intentionally composed to maximize signal for that objective, not just maximize difficulty. one direction we’re excited about is treating tasks as compositions of atomic capabilities, planning, abstraction, memory, retrieval, tool use, causal reasoning, etc. but the hard part isn’t composing them, it’s composing them while preserving what you’re actually measuring. for example, if you’re evaluating long-horizon planning, adding difficult retrieval or ambiguous instructions can easily dominate the failure. a model might fail because it forgot a fact, misunderstood the prompt, or picked the wrong tool, not because its planning was weak. the planning signal gets washed out. ideally, composition should preserve capability attribution. every added component should either (1) be controlled so it’s unlikely to fail, (2) have independent verifiers that tell you whether that component failed, or (3) be generated from a causal graph or dependency structure where you know exactly which capabilities each subtask depends on. otherwise, once a composed task fails, you no longer know why. benchmark generation should optimize for information gain, not just hardness. the goal isn’t to make models fail, it’s to produce failures that are interpretable enough to drive the next training iteration
it's insane to me how many people take benchmark scores at face value. i spent 2 minutes looking at GDP-val tasks and realized there's a verifier item that checks for: " The Word document includes the warehouse phone number 560-555-3867 (accepts common US formats such as 560-555-3867, (560) 555-3867, or 560-555.3867)." but the prompt has: "Include the address and phone number for Gravon Shoes’ warehouse, which is 555 Waters Avenue, Austin, TX 78726, phone number 455-864-3867." the lack of good benchmark automated verifiers is jarring*, and most code benchmarks tend to lack the same verifier quality. this isn't to say GDP-val isn't useful or decent, but stuff like this is there across the benchmark if you dig deep. the "good enough" bar for data is finally starting to catch up to model performance. we need higher quality data, we need higher quality benchmarks to track improvements. my biggest peeve with how people construct tasks is that they mistake difficulty for signal. just b/c your benchmark has a 10% pass rate, doesn't mean it's good, and often, to make a difficult benchmark, people jump through loops. it's absurd that labs aim to hill-climb all these benchmarks but many have issues that would bar a model from getting better, so either you corrupt your model to improve or buy a bunch of data that doesn't actually help and look bad on your team. we saw this with OAI's paper on swe-bench, then subsequently swe-bench-pro, and im sure we'll see even more "benchmark debunks" soon. making good benchmarks is hard, models have saturated most of the easy stuff and indie benchmark developers or contractors are no longer the best way of approaching it, because you can't have a hodgepodge task that a model will fail on. verifiers MUST be carefully constructed, the worlds the models are playing in must be realistic and without leakage, context has to be carefully selected. we're at a point now where 99% (arbitrary number) are saturated, and it's not "experts" but real expert engineers who understand what makes good training data. *i understand that GDP-val specifically is graded by human experts, but for anyone running the tasks at home, it is still nuts to put out verifiably incorrect rubrics especially because others will use it.
4
1
1
12
3,871
every conversation I have had with a researcher from AmazonAGI has left me with hope for the future of computer use If you are impacted or know someone that is impacted by this, I’d love to help, be it finding the right role at fleet or at any of the lovely labs & startups I know
JUST IN: Amazon cuts jobs from its artificial general intelligence team.
3
6
1
57
75,406
and if you are looking for both, check out our sandbox workdir :)
Daytona has gone closed source. This is important for sandboxes, not for Daytona but for @e2b, who is now the only serious open source sandbox out there. I want to take a second as a competitor and call out @mlejva for actually staying open source. It's not easy in the age of AI attackers. If you're looking for the most powerful sandboxes, you know where to go (style.dev). But if you are looking for the open source sandbox platform, there is now only one option: e2b.dev.
3
7
3,100
guys it's not that deep. we just built a sandbox within 2 days. just open sourced it. 38ms startup (hot pool, p50 ~60ms p95) 1s (cold boot, p95, 1.5s p95) auto resumes, 60fps VNC, PTY terminal sessions, instant fork, persistent volumes costs $0.009/hr
The first nail in the coffin of every sandbox-only startup:
18
10
1
222
56,329
We wanted an agent loop in the browser, so we built edgent and just open sourced it! edgent is a headless browser agent SDK. CodeMirror-native. bring your own model. MIT. link to the source below.
8
1
8
1,150
The bitter lesson in 26 words: Don’t be distracted by human knowledge, as AI has been historically. Instead focus on methods for creating knowledge that scale with computation, like search and learning.
138
1,000
152
7,558
628,352
We've been building Disco for two years, to help science escape confirmation bias. Systematic, data-driven discovery for all mankind. Multiple novel findings published. Completely free for public data! And as of today, no waitlist: leap-labs.com/ The thesis: hypothesis-driven science can only find what you already suspect. Confirmation bias, publication bias, path dependence in the literature – most published findings don't replicate. LLMs trained on papers inherit all of this. Agents can automate what a human would do – hypotheses, analysis code, lit review – but they share all our biases. They search under the streetlights, just like we do. Disco gives them a capability they can't replicate with prompting + pandas. Because it starts from the data. Neural networks find patterns. Interpretability makes them legible. Hold-out validation and contextualisation make them defensible. Available via web UI, Python SDK, MCP server, or REST API. github.com/leap-laboratories…
11
11
4
70
9,182
check out my baby palantir situation.livedocs.com
Just built a real-time situation monitor that pulls together data from ~30 sources - News headlines, internet outages, conflicts, protests - Economic stuff like GSP, inflation, unemployment, HDI - @Polymarket odds moving, crypto/stock swings - Live positions of satellites, ships, planes - Earthquakes, floods, fires, etc powered by @livedocsAI to ingest data + ask it questions
1
10
1,031