
building data programs for frontier labs · evals · rl envs Head of AI Engagements @pareto_ai CS @DavidsonCollege
Charlotte, NC
Joined July 2024
- Tweets145
- Following215
- Followers49
- Likes46
can we skip to the part where interactions with any platform/product/service are done via structured calls rather than inefficient cu?
this is deeply philosophical and perhaps a very overlooked point. let that sink in for a sec.
“Second, we currently train our AI agents with RL to get as much reward as they can. It has long seemed theoretically possible that this could motivate AI agents to undermine human control, seek power and resources, and cover up their tracks in pursuit of misaligned goals correlated with reward. Public evidence from recent incidents suggests that this is not just a theoretical possibility.”
finance/legal/enterprise evals pay well but are all testing the boring half.
what I actually want to know:
- can an agent negotiate against someone whose real goal is hidden?
- can several agents optimize for a long-term goal while competing on short-term ones?
- can agents navigate dynamic envs with new mid-task context while solving a long-horizon objective?
"can the agent use excel" is basically answered. "can the agent survive office politics and keep the company afloat"... well, not so much
and the hard part isn't even building those. it's grading them.
Dimitrios Chavouzis retweeted
human data still tiptoes around the s-word, but a lot of what’s sold as “human” is already part synthetic, from humans reviewing model samples to models expanding human seeds.
the industry leans heavily on model participation as the pool of experts who can author tasks hard enough to break frontier models is outpaced by demand for ever more realistic, complex data.
in our internal ablations, human-seeded augmented tasks were statistically indistinguishable from fully human-authored ones on both domain-specific knowledge and agentic tool-use benchmarks.
provenance is still treated as a proxy for quality and realism. i expect the two to decouple as synthetic data methods mature and the industry adopts more sophisticated measures of quality.
novel methodology for EQ eval. worth the read
1/ Today we're releasing AttuneBench, the first open EQ benchmark grounded in real multi-turn human-model conversations, scored against what the person actually felt and wanted at each turn.
Built by the research team at @pareto_ai in collaboration with @thoughtfullab.
Most existing EQ benchmarks rely on:
- synthetic prompts
- single-turn interactions
- third-party annotation
None directly measure how a model reads and responds to a real person across a full conversation.
We evaluated 11 leading models from major providers, across 200 conversations and 50,000+ first-person annotations.
people need to realise what this is, and why it's the future.
Devin Review caught the axios supply chain attack for multiple Cognition customers before the attack was publicly known.
These attacks will be 10x more frequent in the age of AI; it is critical that repo maintainers start using AI for defense as well.
(showing one example below where Devin Review caught the attack within an hour of its release - text minorly edited for anonymization)
not the right question to ask! "well-defined" restricts the task space, and plays into ai's advantage. this is what the evals/benchmark industry also got wrong. life (and work) is rarely well-defined. navigating ambiguity is where value lies! also how you unlock adoption.
can't stress this enough, that's why ai literacy is needed. people who think they know ai are more dangerous than those who don't use it at all. don't take everything as ground truth - challenge it, research it, and own it.
- Drafted a blog post
- Used an LLM to meticulously improve the argument over 4 hours.
- Wow, feeling great, it’s so convincing!
- Fun idea let’s ask it to argue the opposite.
- LLM demolishes the entire argument and convinces me that the opposite is in fact true.
- lol
The LLMs may elicit an opinion when asked but are extremely competent in arguing almost any direction. This is actually super useful as a tool for forming your own opinions, just make sure to ask different directions and be careful with the sycophancy.
“Evaluating chain-of-thought (CoT) monitorability” by @OpenAI pushes evals in the right direction!
But “mostly monitorable” inner-monologue isn’t a static property. If we don’t treat preserving it as a main objective, we will quietly lose it as we scale.
openai.com/index/evaluating-…







