ML researcher
USA
Joined October 2016
- Tweets338
- Following638
- Followers627
- Likes3.6K
Pinned Tweet
At ICML 2024, we introduced the first scaling laws for adversarial training, improving the *train-time compute* for robustness tradeoff (arxiv.org/abs/2404.09349).
At ICLR 2026, we show how to trade *test-time compute* for robustness efficiently.
xitter.cf/i/status/2037426144300…
Brian Bartoldson retweeted
Takeaways from the WSJ article about @HacktronAI using Claude to get into OpenAI's monorepo and issue a pull request (before stopping and claiming their bug bounty)
- how many nation states have already broken in and gone much further and stolen a) algorithmic secrets and b) model weights or c) gotten access to user data; these are fair public interest questions
- how many have implants in / are dwelling in the openai network as I tweet this
- how pervasive is this level of softness to pentesting across all the labs and how far are the labs from the right operating point in the security and r&d friction trade space (probably pretty far it seems)
- given the hacktron folks used anthropic's models to pull this off what's the real public safety ROI of anthropic's cyber guardrails; they add friction for legitimate cyber defenders (like our developers at my startup!) but it appears with a bit of work you can use them in actual breaches as happened here
- as the article says, the wsj folks and @S1r1u5_ had me do neutral technical review of the kill chain here pre publication; impressive from a human angle (@HacktronAI reminds me of the best of my generation of hackers I looked up to as a kid!) but also from what the models can do; between this and the openai/hf thing, I'm emotionally in a place where I feel my world as a security person is being turned upside down in slow motion with respect to what's coming
- elite persistent hacking is becoming rapidly democratized. this is coming like a freight train. we need to harden the world's code and infra as fast as possible and today's non automated methods don't stand a chance of cutting it; the world is a soft target. I continue to be unsettled but very glad I left my comfortable job at Meta to do our automated posture hardening startup
Brian Bartoldson retweeted
i do think a lot of people on the pro-open-source side are having a bit of a knee-jerk reaction to the pacing statements today, as we're used to viewing the closed labs as power-seeking.
but i think their hands are somewhat forced here, and this is just another chapter on the fairly inevitable path towards decently-fast decently-safe decently-commoditized intelligence abundance.
ask any F500 exec or swing voter. the world doesn't really want super-fast-takeoff superintelligence owned only by two companies, and thus we won't get it. it'll happen at the pace that the world can accommodate it, which means reaching some sort of confidence consensus that the models are aligned enough that we won't be dealing with scary new incidents all the time.
this will trickle out broadly, in the form of best practices and distillation. the smarter a model is, the more it has a "personality", and the less effective strict rules are. there will be awkward compromises and moral tensions. but we ultimately just want the models to be reasonable, and to do the sorts of things reasonable humans would do if our brains were faster and less error-prone and had more working memory. i think we'll get there.
the labs will build mac and windows, the rest of us are building linux. everyone's gonna do great. weird stuff will keep happening, but we'll still wake up and go to work, until the work does itself in a manner the world finds acceptable.
Brian Bartoldson retweeted
18 months before the Hugging Face hack:
"On January 7 and 8, 2025, several Federally Funded Research and Development Centers (FFRDCs) organized a small, focused, cross-lab workshop to consider how to assess and monitor the possibility that rogue AI could present an existential threat."
Now on arXiv.
arxiv.org/abs/2609.03189
18 months before the Hugging Face hack:
"On January 7 and 8, 2025, several Federally Funded Research and Development Centers (FFRDCs) organized a small, focused, cross-lab workshop to consider how to assess and monitor the possibility that rogue AI could present an existential threat."
Now on arXiv.
arxiv.org/abs/2609.03189
Brian Bartoldson retweeted
New research: Training a Misaligned Reward Seeker
What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable.
In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring.
Read more: alignment.anthropic.com/2026…
Brian Bartoldson retweeted
LE CRITIQUE: Privileged Value Functions for LLM Reinforcement Learning
Value functions for LLMs have been grossly underutilized. I'm excited to finally share my work on Privileged Value Functions, done @MistralAI !
Our paper includes 2 massive improvements to value functions:
1) Privileged Value Functions: Turns out value functions can use hidden context unavailable to the policy for improved token-level RL signal! Condition them on reference solutions, verifier rubrics, or even other group samples and rewards. No more hacky self-distillation necessary! (being a tad hyperbolic)
2) TETHER: Do not trade off group and value baselines, you can just do both! We introduce a new baseline that adaptively trades off between the group mean and value baselines adaptively based on value function reliability.
🌐 privileged-value-functions.g…
📄 arxiv.org/pdf/2608.16739
Code: github.com/HyperPotatoNeo/pr…
🧵below
Brian Bartoldson retweeted
The results on LongCoT are very surprising
- RLM loses to codex
- Opus+CC loses to GLM+pi
I guess (entirely by accident?) they found an Achilles' heel in claude code. Anthropic should be thankful.
Brian Bartoldson retweeted
revisiting a paper from early 2025: how should we post-train LLMs efficiently? standard on-policy RL forces data gen and policy updates to happen sequentially, choking hardware utilization and wasting compute time.
instead, the authors decouple exploration from learning by introducing Trajectory Balance with Asynchrony (TBA), using the objective (log z (x) + log pi_theta (y | x) - log reward)^2; z (sampled avg reward/prob from batch) acts as a conversion rate to force the probability to match the reward. this allows a central "trainer" node to continuously learn from a diverse replay buffer that multiple "searcher" nodes populate in parallel
using a mix of reward/recency-prioritized sampling, they calculate pushing this to the max. by aggressively scaling parallel data gen while learning off-policy, this recipe beats baseline accuracy while running 4-50x faster compared to sync methods!
Brian Bartoldson retweeted
I'll be presenting 3 papers at ICML 2026🫡
h1 (Spotlight) trains models to reason over longer horizons using curriculum RL over composed short-horizon data. This allows models to generalize to harder tasks and improves performance even at high pass@k.
LongCoT isolates and benchmarks long-horizon CoT capabilities, pushing models to reason over tens and hundreds of thousands of tokens in an output.
Rubric Curriculum RL uses the generation-verification gap in non-verifiable domains to allow models to self-improve on tasks like creative writing and planning. More about this project to come soon!
Very grateful for amazing collaborators @CharlieLondon02 @bartoldson @DanielNichols10 @AmyPrb @hammh0a @jackcai1206 @bkailkhu @casdewitt @sytelus @riashatislam @philiptorr, and especially co-leads Alesia and Tejas!
If you're interested in these topics, my DMs are open :)
Brian Bartoldson retweeted
We trained language models that compress massive contexts into tiny latent representations. Latent Context Language Models (LCLMs) outperform existing KV cache compression methods on the latency/accuracy frontier. 🧵1/10
Brian Bartoldson retweeted
Recordings and the full recap from the Assurance and Verification of AI Development (AViD) Workshop are live. We co-hosted with @ai_risks on May 17, colocated with IEEE S&P. The question: how do you generate trustworthy evidence about an AI system without unrestricted access to weights or infrastructure? 1/6
Brian Bartoldson retweeted
LongCoT is adding two new leaderboards! Due to the interest in agents (particularly RLMs), we’re adding a “Restricted Harness” and an “Open Harness” leaderboard.
GPT 5.2 RLM from our paper is SOTA on “Open Harness” at 25.12%. We expect tool-use SOTA to exceed this very soon!
On “Open Harness”, we allow all tool-use and code execution. On “Restricted Harness”, models may manage context, call subagents, etc, but may not write specific solver code (e.g. writing a BlocksWorld or Sudoku solver). We’re particularly excited about this leaderboard, as it allows agents to do their own context management, while sticking to LongCoT’s goal of testing models’ intrinsic reasoning capabilities.
Brian Bartoldson retweeted
Important context
xitter.cf/bartoldson/status/2045…
The attention on LongCoT is great! It's far from solved (GPT 5.2 w/out tools gets 9.8%).
Out-of-the-box, a GPT 5.2 RLM gets 25% (see Figure 7). Better prompting/training should push RLMs past this.
Comparing RLMs to no-tool baselines? See our 🧵of tips
xitter.cf/sumeetrm/status/204480…
However, beyond just writing code, RLMs can decompose hard problems into simpler ones, and have the LLM perform intermediate reasoning steps. And this behavior is aligned with what we set out to test with LongCoT.
Indeed, in the LongCoT paper, we show a second setting where we verbally ask an RLM (by modifying its system prompt) to avoid solving the problem primarily symbolically, saying it must instead only use code for problem decomposition. As Figure 7 shows, this restriction led to a drop from the 25.1% performance observed with a default RLM.
There are likely better strategies than our prompt approach for keeping RLMs comparable to no-tools baselines, and we welcome innovation from the community here.
So, if you want to compare RLMs to no-tools baselines, we suggest ensuring that RLMs are performing this sort of reasoning assistance, rather than just writing a Python code (to symbolically solve some problems) that makes it unnecessary for LLMs to reason through the steps of a problem.
Concrete tips:
- Look at performance across domains. If chess performance is 80% and chemistry performance is near 0%, the RLM might be writing Python code to simulate chess moves rather than reasoning through the game states as a no-tools baseline does. Such simulations may be harder to code for LongCoT chemistry and mathematics questions.
- Given an LLM, when you see its RLM version boost performance, check the code executed by the RLM. Was the RLM decomposing the problem into simpler ones then providing these to the sub-LLM (this is okay!), or did it write code or import a library that solves the problem directly (LLMs w/out tools can’t do this so it won’t be a fair comparison)?
Finally, we note that LongCoT has two splits: LongCoT-mini (for fast open source research) and LongCoT (for frontier models).
While some problems are naturally solvable by symbolic programs (e.g. some chess problems) and require many CoT reasoning steps, others are great tests for decomposition and maintaining important variables across steps (like mathematics and chemistry, which have composed, interdependent sub-problems). We suspect RLMs may be highly effective for all problem types.
The attention on LongCoT is great! It's far from solved (GPT 5.2 w/out tools gets 9.8%).
Out-of-the-box, a GPT 5.2 RLM gets 25% (see Figure 7). Better prompting/training should push RLMs past this.
Comparing RLMs to no-tool baselines? See our 🧵of tips
xitter.cf/sumeetrm/status/204480…
Accordingly, tomorrow, we will begin tracking tool-enabled performance on a separate leaderboard at longcot.ai/. Note that we expect this to be saturated much faster than the base leaderboard that doesn’t allow tools.
RLMs can aid this LLM-based simulation (very well in theory), but (due to REPL access) they can also just write Python code that solves certain problems symbolically (without any LLM ever seeing intermediate reasoning steps).
xitter.cf/bartoldson/status/2045…
To illustrate setting (1): when I ran DSPy RLMs on a chess problem from LongCoT-mini, I noticed the sub-LLM tool (`llm_query`) wasn't called -- Python ended up doing the search directly as follows.
```
board = ChessBoard()
for i, move in enumerate(moves):
board.make_move(move)
...
solution = board_to_fen(board)
```
You can obtain setting (2) by altering DSPy's RLM prompt to discourage such reasoning offloading. Even in this harder setting, I suspect RLMs can be successful (and training to decompose tasks may be critical here).
First, some background:
The LongCoT paper outlines that, in our primary evaluations, tool use is not allowed. Our main goal is to test whether models can reason through complex problems directly in their chain of thought. If a problem requires simulating the evolution of a chessboard state (e.g.), the LLM must do that in its CoT.