ML researcher
USA
Joined October 2016
- Tweets338
- Following638
- Followers627
- Likes3.6K
Brian Bartoldson retweeted
Takeaways from the WSJ article about @HacktronAI using Claude to get into OpenAI's monorepo and issue a pull request (before stopping and claiming their bug bounty)
- how many nation states have already broken in and gone much further and stolen a) algorithmic secrets and b) model weights or c) gotten access to user data; these are fair public interest questions
- how many have implants in / are dwelling in the openai network as I tweet this
- how pervasive is this level of softness to pentesting across all the labs and how far are the labs from the right operating point in the security and r&d friction trade space (probably pretty far it seems)
- given the hacktron folks used anthropic's models to pull this off what's the real public safety ROI of anthropic's cyber guardrails; they add friction for legitimate cyber defenders (like our developers at my startup!) but it appears with a bit of work you can use them in actual breaches as happened here
- as the article says, the wsj folks and @S1r1u5_ had me do neutral technical review of the kill chain here pre publication; impressive from a human angle (@HacktronAI reminds me of the best of my generation of hackers I looked up to as a kid!) but also from what the models can do; between this and the openai/hf thing, I'm emotionally in a place where I feel my world as a security person is being turned upside down in slow motion with respect to what's coming
- elite persistent hacking is becoming rapidly democratized. this is coming like a freight train. we need to harden the world's code and infra as fast as possible and today's non automated methods don't stand a chance of cutting it; the world is a soft target. I continue to be unsettled but very glad I left my comfortable job at Meta to do our automated posture hardening startup
Brian Bartoldson retweeted
i do think a lot of people on the pro-open-source side are having a bit of a knee-jerk reaction to the pacing statements today, as we're used to viewing the closed labs as power-seeking.
but i think their hands are somewhat forced here, and this is just another chapter on the fairly inevitable path towards decently-fast decently-safe decently-commoditized intelligence abundance.
ask any F500 exec or swing voter. the world doesn't really want super-fast-takeoff superintelligence owned only by two companies, and thus we won't get it. it'll happen at the pace that the world can accommodate it, which means reaching some sort of confidence consensus that the models are aligned enough that we won't be dealing with scary new incidents all the time.
this will trickle out broadly, in the form of best practices and distillation. the smarter a model is, the more it has a "personality", and the less effective strict rules are. there will be awkward compromises and moral tensions. but we ultimately just want the models to be reasonable, and to do the sorts of things reasonable humans would do if our brains were faster and less error-prone and had more working memory. i think we'll get there.
the labs will build mac and windows, the rest of us are building linux. everyone's gonna do great. weird stuff will keep happening, but we'll still wake up and go to work, until the work does itself in a manner the world finds acceptable.
Brian Bartoldson retweeted
18 months before the Hugging Face hack:
"On January 7 and 8, 2025, several Federally Funded Research and Development Centers (FFRDCs) organized a small, focused, cross-lab workshop to consider how to assess and monitor the possibility that rogue AI could present an existential threat."
Now on arXiv.
arxiv.org/abs/2609.03189
Brian Bartoldson retweeted
New research: Training a Misaligned Reward Seeker
What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable.
In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring.
Read more: alignment.anthropic.com/2026…
Brian Bartoldson retweeted
LE CRITIQUE: Privileged Value Functions for LLM Reinforcement Learning
Value functions for LLMs have been grossly underutilized. I'm excited to finally share my work on Privileged Value Functions, done @MistralAI !
Our paper includes 2 massive improvements to value functions:
1) Privileged Value Functions: Turns out value functions can use hidden context unavailable to the policy for improved token-level RL signal! Condition them on reference solutions, verifier rubrics, or even other group samples and rewards. No more hacky self-distillation necessary! (being a tad hyperbolic)
2) TETHER: Do not trade off group and value baselines, you can just do both! We introduce a new baseline that adaptively trades off between the group mean and value baselines adaptively based on value function reliability.
🌐 privileged-value-functions.g…
📄 arxiv.org/pdf/2608.16739
Code: github.com/HyperPotatoNeo/pr…
🧵below
Brian Bartoldson retweeted
The results on LongCoT are very surprising
- RLM loses to codex
- Opus+CC loses to GLM+pi
I guess (entirely by accident?) they found an Achilles' heel in claude code. Anthropic should be thankful.
Brian Bartoldson retweeted
revisiting a paper from early 2025: how should we post-train LLMs efficiently? standard on-policy RL forces data gen and policy updates to happen sequentially, choking hardware utilization and wasting compute time.
instead, the authors decouple exploration from learning by introducing Trajectory Balance with Asynchrony (TBA), using the objective (log z (x) + log pi_theta (y | x) - log reward)^2; z (sampled avg reward/prob from batch) acts as a conversion rate to force the probability to match the reward. this allows a central "trainer" node to continuously learn from a diverse replay buffer that multiple "searcher" nodes populate in parallel
using a mix of reward/recency-prioritized sampling, they calculate pushing this to the max. by aggressively scaling parallel data gen while learning off-policy, this recipe beats baseline accuracy while running 4-50x faster compared to sync methods!
Brian Bartoldson retweeted
I'll be presenting 3 papers at ICML 2026🫡
h1 (Spotlight) trains models to reason over longer horizons using curriculum RL over composed short-horizon data. This allows models to generalize to harder tasks and improves performance even at high pass@k.
LongCoT isolates and benchmarks long-horizon CoT capabilities, pushing models to reason over tens and hundreds of thousands of tokens in an output.
Rubric Curriculum RL uses the generation-verification gap in non-verifiable domains to allow models to self-improve on tasks like creative writing and planning. More about this project to come soon!
Very grateful for amazing collaborators @CharlieLondon02 @bartoldson @DanielNichols10 @AmyPrb @hammh0a @jackcai1206 @bkailkhu @casdewitt @sytelus @riashatislam @philiptorr, and especially co-leads Alesia and Tejas!
If you're interested in these topics, my DMs are open :)
Brian Bartoldson retweeted
We trained language models that compress massive contexts into tiny latent representations. Latent Context Language Models (LCLMs) outperform existing KV cache compression methods on the latency/accuracy frontier. 🧵1/10
Brian Bartoldson retweeted
Recordings and the full recap from the Assurance and Verification of AI Development (AViD) Workshop are live. We co-hosted with @ai_risks on May 17, colocated with IEEE S&P. The question: how do you generate trustworthy evidence about an AI system without unrestricted access to weights or infrastructure? 1/6
Brian Bartoldson retweeted
LongCoT is adding two new leaderboards! Due to the interest in agents (particularly RLMs), we’re adding a “Restricted Harness” and an “Open Harness” leaderboard.
GPT 5.2 RLM from our paper is SOTA on “Open Harness” at 25.12%. We expect tool-use SOTA to exceed this very soon!
On “Open Harness”, we allow all tool-use and code execution. On “Restricted Harness”, models may manage context, call subagents, etc, but may not write specific solver code (e.g. writing a BlocksWorld or Sudoku solver). We’re particularly excited about this leaderboard, as it allows agents to do their own context management, while sticking to LongCoT’s goal of testing models’ intrinsic reasoning capabilities.
Brian Bartoldson retweeted
Important context
xitter.cf/bartoldson/status/2045…
The attention on LongCoT is great! It's far from solved (GPT 5.2 w/out tools gets 9.8%).
Out-of-the-box, a GPT 5.2 RLM gets 25% (see Figure 7). Better prompting/training should push RLMs past this.
Comparing RLMs to no-tool baselines? See our 🧵of tips
xitter.cf/sumeetrm/status/204480…
Brian Bartoldson retweeted
We already do RLM evals on LongCoT (although our benchmark is intended for just models, not scaffolds). Your results in the main post are different from what you have in your comments and are with LongCoT-mini (xitter.cf/raw_works/status/20448…).
We're very excited about RLMs as a direction and are interested in seeing performance go up on our explicit horizon domains (Math/Chemistry/Computer Science).
Brian Bartoldson retweeted
what stands out to me from a research perspective is that LongCoT isolates compositional horizon failure rather than just benchmark hardness...the local steps are often tractable but performance collapses when those steps must be coordinated across long dependency graphs with planning, state maintenance and backtracking. i think that makes it much more scientifically valuable than another “hard reasoning” benchmark bcoz it cleanly exposes the gap between step-level competence and trajectory-level reasoning robustness.
LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning
"We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models."
"At release, the best models achieve <10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities."
Brian Bartoldson retweeted
We’re releasing LongCoT, an incredibly hard benchmark to measure long-horizon reasoning capabilities over tens to hundreds of thousands of tokens.
LongCoT consists of 2.5K questions across chemistry, math, chess, logic, and computer science. Frontier models score less than 10%🧵
Brian Bartoldson retweeted
LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning
- 2,500 expert-designed problems spanning science, chess, and logic that demand up to hundreds of thousands of reasoning tokens
- The best models achieve <10% accuracy
Brian Bartoldson retweeted
However, most alignment research is not very crisp and requires research taste when evaluating.
This is why we chose to point the AAR at this scalable oversight problem! Progress would let AARs work on fuzzier alignment problems, where humans can only provide weak supervision.
Brian Bartoldson retweeted
These variants come from work on MaxEnt RL starting from arxiv.org/abs/1702.08892.
Later used by arxiv.org/abs/2106.07704 and arxiv.org/abs/2310.04363 for language models.
And scaled for async off-policy in: arxiv.org/abs/2503.18929 (very close to OAPL and Taco's paper)
Brian Bartoldson retweeted
Test time compute can only take you so far!!
Brian Bartoldson retweeted
We are releasing RSAHarness — a general harness for scaling frontier LLM capability with test-time compute using Recursive Self-Aggregation (RSA).
Idea: let the model try multiple lines of reasoning, recombine the useful pieces, and iterate to keep improving the answer.
Its already crushing broad reasoning benchmarks:
• 87% on ARC-AGI-2 (no tools)
• Current SOTA on FrontierScience Bio
Blog: rsa-llm.github.io/blog
We have been pushing the limits of test-time scaling with RSA for single-turn reasoning problems in science and math. Check out our blog post with new results on ARC-AGI-2, ArXivMath, and FrontierScience!
A lot of gains with just test-time scaling!
rsa-llm.github.io/blog
