I have a toxic relationship with agent evals
San Francisco Bay Area
Joined July 2017
- Tweets232
- Following394
- Followers108
- Likes926
Pinned Tweet
Aligning agent evals with your personal judgment is essential for building improvement loops you can trust.
Otherwise, you’ll steer your coding agents towards problems that don’t exist, and miss the ones that do.
A common approach for agent evals is LLM-as-a-Judge or Agent-as-a-Judge that labels your traces. The foundation of the eval is the prompt you define as a rubric. However, most people do not realize the first prompt they create doesn’t cover all cases that their judge will have to evaluate.
Say you want to flag instances where users are frustrated with your agent. What happens if a user is still frustrated after an issue is resolved? If your initial prompt did not include clarification on whether to flag this or not, the judge might or might not label this trace as frustrated. When this ambiguity scales to thousands of traces, you'll be making changes to your agent based on false positives or negatives.
Therefore, having a human in the loop to verify the evals is a natural prereq to creating scalable evals you can trust. This has traditionally looked like the following: hand-label agent traces, compare your labels with the judge’s, and continue iterating on the rubric until the judge aligns with your ground-truth labels.
I work directly with agent engineers who have historically found this tedious and avoided it entirely, but I’d argue that aligning against human-reviewed traces is well worth the time. Labeling even just 10-20 traces will make the quality of your rubric significantly better, allowing you to reliably grade tens of thousands of traces.
I’ve personally seen teams that neglected judge alignment and ended up spending entire sprints’ worth of engineering time making choices based on a false understanding, only to have to do a complete 180 after realizing their evals weren’t giving them accurate insights.
This is why at @JudgmentLabs, we spent a lot of time streamlining this process in our product. Once you provide the ground-truth labels, our agents iteratively refine and backtest the rubric until it matches your labels. The product also reminds you to regularly label new traces so that your judges stay accurate as your agent evolves.
Whether you’re using Judgment Labs or not, aligning your judges’ rubrics will not only improve the quality of their evals, but will eventually help create an eval system you can trust and scale.
Come work with me @JudgmentLabs!
We're hiring a Deployment/Customer Facing Engineer to help customers build evals and improve their agents.
Link to apply is below, DM w/ questions about the role or if you know someone who might be a fit.
judgmentlabs.ai/careers/d661…
Hillclimbing with @JudgmentLabs is too much fun! Currently taking inspiration from @joshjwilkin and building a realtime + reasoning agent that plays Halo through @mitchellhynes's combat lab. Will report back when 'Master Jev' escapes Installation 04.
Day 1 of building Jeeve (pronounced like Steve), a realtime + reasoning general purpose Minecraft agent, that plays with you.
You /chat Jeeve like you would message your coding agent, and Jeeve interacts with the world and learns how to perform any tasks. Being able to rewrite entire parts of its harness to hill climb on perception + learning.
Why?
1. Earlier Minecraft bots were either fast but task specific, or smart but slow (paused entire game to think).
2. Test the theory that decision models can bridge that gap to make the realtime interactions smarter, while reasoning can drive overarching actions without blocking responsiveness.
3. cool as hell (imagine: "Jeeve, build this base I saw on instagram ")
High level Design
This initial prototype of Jeeve is built on a System 1 + 2 approach.
System 1 / Decision model (Jev):
- handles real time reactivity + fight or flight decision making. Follows a current task (i.e chop down a tree). Also responds realtime to threats (mobs + hunger + health).
- can use skills / deterministic code controllers that system 2 + async tool builders create
System 2 (Opus 5.5)
- handle longer term planning tasks + handles interrupts on real time decisions needed by System 1 reacting to it's current environment.
- decides which tools Jeeve is missingand has Async Tool Builder build them as code controllers.
Async Tool Builder (Opus 5.5)
- builds low-latency code controllers that behave like control systems for small scoped tasks (e.g. efficient towering, bridge building)
- System 1 then calls this code at low latency, for the appropriate task, and it’s kept for next time.
ofc tracing, evals, hill-climbing using @JudgmentLabs
still lots of ideas to try, stay tuned!
Nick Lemoff retweeted
If the data ain't organized, every question turns into a whole damn project
Good judges are the ones that make it easy for other agents to answer the questions you have about how your agent performs.
Let’s take a prompt that you might give to an agent to analyze a set of agent traces:
“Looking across the past month, where were users frustrated with my agent across different user and request types, what kinds of problems were causing their frustration, and were their problems resolved?”
An agent analyzing these traces will struggle to answer this question. The scale of the question involves searching across millions of traces and digging through various semantic details across all of the traces on that timescale. SOTA coding agents still struggle with running analysis at this scale when traces aren’t annotated.
Getting your agent to do more accurate, deep, and token efficient analysis is conceptually the same idea as adding indexes to a database. Without indexes, queries have to do a full table scan, which is inefficient. Evaluating and analyzing agent traces is the exact same idea! The labels produced by judges become the indexes that analyzer agents can use to search across trace data more cheaply and efficiently.
Imagine looking through a million agent runs by hand. Now imagine you had a table of contents you could flip through instead!
Going back to the example question, the way that I’d approach having this question answered is to create a judge that indexes my data on each dimension I ask about in the question:
“Where were my users frustrated” → User Sentiment Judge
“across different user” → User Type
“and request types” → Request Type
“what kinds of problems were causing their frustration” → Failure mode 1, Failure mode 2, etc.
“and were their problems resolved?” → Resolution Status
With these judges labeling my traces, instead of an agent having to read every trace in full and reason over each of them, it’s searching via indexes across spans, traces, sessions and with agent/LLM judges, a citation explaining why a trace received its label. Like a database index, the labels are pre-computed once instead of every time you ask a question, which is what makes the analysis both cheaper and more accurate.
The agent we offer at @JudgmentLabs filters and groups your traces with these labels and only views cited traces when it needs specific details. Its analysis goes straight to your coding agent as context through Slack and/or MCP to ship fixes and improvements to your agent.
Regardless of where you run your evals, it’s important that you’re setting them up to answer the question that you actually care about.
Good judges are the ones that make it easy for other agents to answer the questions you have about how your agent performs.
Let’s take a prompt that you might give to an agent to analyze a set of agent traces:
“Looking across the past month, where were users frustrated with my agent across different user and request types, what kinds of problems were causing their frustration, and were their problems resolved?”
An agent analyzing these traces will struggle to answer this question. The scale of the question involves searching across millions of traces and digging through various semantic details across all of the traces on that timescale. SOTA coding agents still struggle with running analysis at this scale when traces aren’t annotated.
Getting your agent to do more accurate, deep, and token efficient analysis is conceptually the same idea as adding indexes to a database. Without indexes, queries have to do a full table scan, which is inefficient. Evaluating and analyzing agent traces is the exact same idea! The labels produced by judges become the indexes that analyzer agents can use to search across trace data more cheaply and efficiently.
Imagine looking through a million agent runs by hand. Now imagine you had a table of contents you could flip through instead!
Going back to the example question, the way that I’d approach having this question answered is to create a judge that indexes my data on each dimension I ask about in the question:
“Where were my users frustrated” → User Sentiment Judge
“across different user” → User Type
“and request types” → Request Type
“what kinds of problems were causing their frustration” → Failure mode 1, Failure mode 2, etc.
“and were their problems resolved?” → Resolution Status
With these judges labeling my traces, instead of an agent having to read every trace in full and reason over each of them, it’s searching via indexes across spans, traces, sessions and with agent/LLM judges, a citation explaining why a trace received its label. Like a database index, the labels are pre-computed once instead of every time you ask a question, which is what makes the analysis both cheaper and more accurate.
The agent we offer at @JudgmentLabs filters and groups your traces with these labels and only views cited traces when it needs specific details. Its analysis goes straight to your coding agent as context through Slack and/or MCP to ship fixes and improvements to your agent.
Regardless of where you run your evals, it’s important that you’re setting them up to answer the question that you actually care about.
Great new feature from Anthropic, getting your agent traces is going to be easier than ever! Knowing how to use them to actually improve your agent though...
Nick Lemoff retweeted
Judgment Labs is in the #Disruptors60 Class of 2026.
We're building the continuous-improvement stack for agents.
Thank you @GreenfieldVC, and to the team and customers building with us.
disruptors60.ai/?utm_source=…
ALT Judgment Labs badge for the Disruptors 60 Class of 2026, featuring Greenfield and TechCrunch Disrupt.
Nick Lemoff retweeted
@JudgmentLabs is in the #Disruptors60 Class of 2026.
We're building toward a simple goal: agents that keep getting better in production.
Proud of this team. Thanks to our customers for building with us, and @GreenfieldVC for the recognition.
ALT The Nasdaq tower in Times Square displays Judgment Labs' logo beneath the Disruptors 60 announcement.
Nick Lemoff retweeted
Day 1 of building Jeeve (pronounced like Steve), a realtime + reasoning general purpose Minecraft agent, that plays with you.
You /chat Jeeve like you would message your coding agent, and Jeeve interacts with the world and learns how to perform any tasks. Being able to rewrite entire parts of its harness to hill climb on perception + learning.
Why?
1. Earlier Minecraft bots were either fast but task specific, or smart but slow (paused entire game to think).
2. Test the theory that decision models can bridge that gap to make the realtime interactions smarter, while reasoning can drive overarching actions without blocking responsiveness.
3. cool as hell (imagine: "Jeeve, build this base I saw on instagram ")
High level Design
This initial prototype of Jeeve is built on a System 1 + 2 approach.
System 1 / Decision model (Jev):
- handles real time reactivity + fight or flight decision making. Follows a current task (i.e chop down a tree). Also responds realtime to threats (mobs + hunger + health).
- can use skills / deterministic code controllers that system 2 + async tool builders create
System 2 (Opus 5.5)
- handle longer term planning tasks + handles interrupts on real time decisions needed by System 1 reacting to it's current environment.
- decides which tools Jeeve is missingand has Async Tool Builder build them as code controllers.
Async Tool Builder (Opus 5.5)
- builds low-latency code controllers that behave like control systems for small scoped tasks (e.g. efficient towering, bridge building)
- System 1 then calls this code at low latency, for the appropriate task, and it’s kept for next time.
ofc tracing, evals, hill-climbing using @JudgmentLabs
still lots of ideas to try, stay tuned!
Haiku 5.5 in claude code is awesome, but all of my private benchmarks I've tested it on have it performing about the same as GPT 6 luna, but using significantly more tokens.
Wait what?
We’re thrilled to welcome Chris Degnan as our Chief Revenue Officer to help us scale globally and build the future of software.
Chris joined Snowflake as its first sales hire. Over the next decade, he scaled alongside the company and grew into Snowflake’s CRO, driving over $3B in ARR and building one of the most iconic sales organizations in tech.
He’s a world-class leader that builds winning teams, executes with relentless rigor, and focuses on what’s right for customers.
We’re also excited to partner with Chad Peets and the RPT team to build Cognition’s GTM team. Chad was Chris’ co-pilot in building Snowflake’s world-class sales team.
Welcome to Cognition, Chris.
