another thing I see in #agent #evaluation is a lot of attention on the exact path the agent took (agent trajectory), almost as if there is one correct sequence of tool calls and every other path should fail. most of the time that is testing the implementation we expected instead of the outcome the user needed. for many real world tasks, the first question is whether the agent produced the right result. two runs can use different tools, call them in a different order, and still complete the task correctly. forcing one expected trajectory can reject completely valid behavior, and it becomes brittle every time the model learns a better way to reach the same outcome. that does not mean the trajectory is unimportant. we need it to explain the result, catch claims that are not supported by what happened, understand how tools were used, and enforce the path requirements that are actually part of the product. if one operation really has to happen before another, or a specific action is itself part of success, then the task should say that explicitly. I think the outcome should normally be primary, with only the trajectory constraints we can explain as real requirements, because otherwise an implementation preference slowly becomes the definition of correct product behavior.

a few more thoughts about #agentic #evals. another problem I keep running into is how we decide what we are actually paying to evaluate. I see many projects begin with the artifact they just built. they built a skill, so they create an eval for the skill, run it a few times, and eventually attach some quality number to it. it is a natural place to start, but I think it starts one layer too low. a skill does not produce useful behavior by itself. the result also depends on the task, the model, the tools, the agent scaffold, the environment, and sometimes on the agent deciding that the skill is not needed at all. if all of that gets compressed into one number stored on the skill, the number looks precise, but it is difficult to say what it means. what makes more sense to me is defining the complete real-world task first, including what starts it, which initial conditions exist, what outcome we expect, and what success means, then running that same task against different versions of the system. one version can include the skill, another can leave it out, and everything else can remain as similar as possible. the thing I care about following over time is whether the system can complete the task, under which conditions, and with what reliability. the skill still matters, but more as one possible explanation for why the behavior changed than as the thing carrying a quality score of its own.

one word that has created a surprising amount of confusion when trying to run #agent #evaluation on some projects is the word "configuration". we put the model, prompt, skills, tasks, number of trials, retries, concurrency, credentials, environment setup, runner, and sometimes the judge into one configuration object, and later we cannot explain whether a result changed because the agent changed, the way we measured it changed, or the infrastructure changed. I think there are at least three different things hiding in there. one describes what we are measuring and how the evidence becomes a result. another describes the actual system being measured, including the model, prompt, skills, tools, and scaffold. the third describes how the cloud environment is going to run both of them, including provisioning, credentials, retries, concurrency, evidence capture, and cleanup. this is not only an argument about terminology. if I change the rubric, I changed the measurement. if I change the model, I changed the system. if I run the exact same system in a different kind of environment, I changed how it was executed. those changes mean different things when we compare results or try to reproduce and reuse them. configuration is a convenient word when the system is small, but it becomes a junk drawer very quickly, and once everything is inside that drawer it becomes hard to say what actually changed. my recommendation is to give these three things separate identities, version them independently, and make the comparison layer expose the difference. before calling two runs comparable, we should be able to say whether the evaluation definition, the system being evaluated, or the execution environment changed.

another thing that comes to mind when I look at small #agent #evaluation examples is how often they hide the cost of recreating enough of the real product for the result to mean something. if the agent uses tools from an MCP server, reads product data, and can mutate that data while completing the task, we are no longer evaluating a prompt and a response in isolation. every trial needs a known initial state and enough isolation that one run does not hand the answer, or the damage, to the next one. we need to observe the tool calls, capture the mutation, validate the resulting state, preserve the evidence, and either clean everything up or create another isolated environment before repeating the task. if we mock too much, the evaluation becomes cheaper but may stop representing the product we care about. the good news is that this is not an entirely new discipline. performance engineers have been creating representative workloads, controlling initial conditions, separating product behavior from infrastructure noise, measuring variance, instrumenting dependencies, and figuring out what expensive setup can be reused, and for how long, without invalidating the measurement. agentic eval adds a stochastic model, semantic judging, and tools that can make decisions and mutate state, but much of the old performance engineering stack and, more importantly, its discipline should carry forward. so the cost is not only model tokens, and it is not only the judge. it includes provisioning, external I/O, rate limits, state capture, reset, cleanup, and the capacity required to keep many independent experiments running. a small application can ignore much of this and get a lot of value from a simple eval loop. for a real #enterprise product, this environment lifecycle may be one of the hardest and most expensive parts of evaluation. I think this is also going to create a very interesting space for performance engineers. many teams entering agentic evaluation are going to rediscover problems that this discipline already knows how to reason about, and the people who understand how to build trustworthy experiments around complex systems may become as important as the people writing the prompts and graders. so far, my experience is that the agentic portion of the flow, the actual execution of the agent we are trying to evaluate, is the bottleneck in both cost and elapsed time, not the product infrastructure it talks to. I do not take that as evidence that the infrastructure does not matter, though. as we increase the number of tasks, repetitions, isolated environments, and parallel runs, I expect many of those familiar performance engineering problems to become visible very quickly.

the new #Claude eval and hillclimbing (claude.dev/blog/automating-e…) workflow is very interesting because it makes evaluation and optimization available to many more people, but I think the cost can be easy to underestimate when we look at relatively small examples. the expensive part is not only creating the data set or judging the final answer. for every candidate change, the agent has to run again with its prompt and context, execute tools, produce evidence, and repeat enough trials to separate a real improvement from normal variation. then the #hillclimber makes another change and the multiplication starts again. with a large product and a meaningful task population, candidates times tasks times trials can become a very large number of complete agent executions. after reading the current implementation, I think Claude already has some of the right pieces. it measures the noise floor, asks for a plateau rule, estimates the cost of the planned rounds, and can stop iterating when several changes fail to move the score outside that noise. those are meaningful controls, but the default is still to run the full #eval for every round, and the budget is treated as a guide that can be extended when the score is still moving. for me, knowing when to stop is one of the most important parts of the hillclimbing loop, and a plateau is only one reason to stop. before paying for the next stage, I also want to know whether the candidate has enough remaining headroom to produce a measurable improvement, whether the evidence collected so far makes that improvement plausible, and whether proving it is worth more than the next round will cost. sometimes the correct outcome is not another experiment, it is that this particular idea has exhausted the evidence budget we were willing to spend on it. this is where I think progressive evaluation starts looking essential for hillclimbing, not simply like a faster way to report failures. run a small number of complete tasks that are directly relevant to the proposed change, then the important product scenarios, and only then the long tail, with an explicit promotion rule between stages. weak candidates should disappear before consuming the full evaluation budget, while the few that survive still have to earn the broader coverage and repeated trials needed for a real claim. this does not make statistically meaningful evaluation cheap. it avoids paying the maximum price for every small idea the optimization agent wants to try, which may be what makes the loop realistic for a complex product.

speaking of agent evaluation cost, systems like TypeSafe's #Jev are very interesting to me because using a large generative model to answer every bounded evaluation question is obviously wasteful. if the answer can be represented as a choice, a score, or a probability, a much cheaper decision model can reduce a meaningful part of the judging cost, assuming we have done the work to calibrate it against evidence we trust. but I don't think that is going to make agentic evals considerably cheaper. before there is anything to judge, we still have to run the agent. the main model consumes tokens, tools execute, environments have to be prepared, external systems are touched, evidence is captured, and when reliability matters we have to do much of that multiple times. so yes, cheaper judges are going to matter quite a bit, but the agent still has to do the expensive work we are trying to measure.

for anyone building interactive MCP UI widgets on ChatGPT Apps: the follow-up message flow (the mechanism where a widget informs the agent that something happened, like a button click or a selection) has several open bugs that will likely cost you time if you don't know about them going in. the widget gets duplicated or re-rendered after sending the message. the scroll breaks so the user never sees the response. in some cases it crashes entirely. and if you send while the LLM is still responding, the call gets silently dropped. for reference, the same flow works without issue on Claude Desktop and Slack's MCP UI client. the protocol handles it fine, the problem is specific to ChatGPT's host implementation. the issues are tracked on github (github.com/openai/openai-app… #129, #69, #191, #126), some open since november 2025 with no response yet. if anyone knows someone on the ChatGPT Apps team or has a channel to help move these forward, i'd appreciate the connection. follow-up messages are the core interactive loop for widgets, without them a widget can display information but can't participate in the conversation. getting this fixed would unblock a lot of what people are trying to build on that platform.

i've been posting about what agent-composed UI looks like, but you have to see it and feel it in practice. it's very much experimental, but give it a shot and tell me what you see. the setup: you take any MCP server you already have wired up in your host (a catalog, a CRM connector, a ticketing system, whatever returns structured data) and you add `@mcp-lens/server` via npx connector alongside it. this gives the agent a tool for composing visual responses and some lightweight guidance on how to use it. that's it. you don't change anything about your existing server. no new code, no widget authoring, no renderer to build. this is the part worth experiencing firsthand. ask the same questions you normally would, and let the agent pull data from your existing server the same way it always did. if it answers in plain text, tell it "use show_lens when possible" and it picks up the pattern for the rest of the session. from there it should compose a bounded visual response sized for the conversational moment. a card when you asked about one thing. a list with per-row buttons when you're browsing. a comparison table when you're choosing between two. next-turn affordances on every response so you advance by clicking instead of typing. and it shifts shape turn by turn because the agent is reading the full conversation and deciding what fits each moment. that's it, just npx it into your config `{ "mcpServers": { "lens": { "command": "npx", "args": ["-y", "@mcp-lens/server"] } } }` next to something you already use and see what comes out.

i think #MCPUI is going to end up serving a much smaller role than its current positioning suggests. not because it doesn't work, it does, but because the class of conversational moments that genuinely need a custom, tool-authored widget is narrow, and it's about to get narrower. the majority of what users ask about in agent conversations is structured data: records, lists, comparisons, statuses, confirmations. these are well served by a constrained vocabulary the superagent composes per turn. you don't need a custom widget for "show me the account" or "compare these two incidents" or "what's the status of my order." you need the agent to know what moment the user is in and compose the right bounded affordances from that data. that's what Visualizers and agent-composed approaches do, and they do it without the tool author shipping anything beyond the data itself. where #MCPUI still earns its place: genuinely interactive experiences that go beyond what bounded affordances can express. complex data visualizations with pan/zoom, custom editors, media players, anything that needs persistent client-side state within a single turn. those exist. they're real use cases. they're also not the common case in most conversational workflows. i think within a year, agent-composed UI from structured output becomes the default path for most conversational moments, and #MCPUI becomes the escape hatch you reach for when the constrained vocabulary isn't enough. useful, supported, available, but not the primary way most tool responses end up rendered.

here's a small example that exposes a structural limit of #MCPUI widgets. the user says "i don't like emojis" partway through a conversation, or "stop showing me pricing," or "i prefer compact views." from that point forward, every response should honor that preference. if the response is plain text, the agent just adapts. it heard the preference, it remembers it, it writes differently from now on. done. if the response is an agent-composed UI from a constrained vocabulary, same thing — the agent composes the next view without emojis, without pricing, more compact. it already has the preference in context; nothing else has to change. but if the response is a pre-built MCPUI widget, that preference has nowhere to go. the widget was authored months ago by someone who never anticipated "i don't like emojis" as a runtime condition. the widget renders what it renders. the user said something, and the UI can't hear it. the escape hatches people reach for: maybe add an input parameter to the tool ("includeEmojis: false") so the widget can read it. now multiply that by every preference a user might express across a session — compact views, hide certain fields, language tone, level of detail — and the tool's input schema becomes a growing bag of display config that has nothing to do with the tool's actual job. or maybe the superagent passes all accumulated preferences into every tool call as extra context, and the widget reads them from the args. now the superagent has to maintain and serialize a preference blob on every single call, and the widget has to parse and honor an open-ended set of display rules it was never designed for. both escape hatches work in theory. both add complexity that compounds with every new preference the user might express. the architecture where the superagent composes the UI per turn doesn't face this. the agent heard "i don't like emojis" the same way it heard everything else in the conversation. the next time it composes a view, it leaves out the emojis. no config parameter, no preference blob, no widget modification. the preference is just part of the context the agent is already reading when it decides what to render.

i think the cheapest, highest-leverage UX primitive in conversational agents is a small set of bounded affordances at the end of every meaningful response. two to four next-turn buttons, sized for the moment, and the user advances by clicking instead of typing. mechanically, this is the same job the model is already doing for text. every text response is the model predicting what comes next given the conversation context. bounded affordances are the same prediction, just rendered as a few clickable options instead of a paragraph. the model already knows what the user is likely to want next, it demonstrates that by writing coherent follow-up sentences. making those predictions explicit and clickable is almost free in terms of what the model has to do differently; it's just a different output format for the same underlying work. the missed-turn failure mode is when the agent answers about structured data in plain prose and sends the user back to the keyboard. the agent had a chance to predict the next two or three moves and make them one click away. instead it wrote a paragraph and left the user to figure out what to type next. multiply that across a session and you get a UX that feels like a search bar dressed up as a chat, technically conversational, practically stalled after every answer. the rule i've ended up with: if the response was about a specific thing, end with two to four follow-ups that name what the user might do next with that thing. never zero, rarely one. anything more starts feeling like a navigation menu, which is a different shape for a different medium.

i've been thinking about what conversational UX does to frontend craft, and the shift i keep landing on is from designing widgets to designing vocabularies. fewer pixels, more structural decisions. the work moves up a level. what that looks like concretely: instead of choosing a shape for "the order detail page," you choose which node types belong in the spec, which conversational moments earn their own affordance set on a given domain, and what precedent the agent should read to compose the moment well. the agent does the per-turn composition. the vocabulary and the precedent are still authored, but they're authored once and then reused across every moment that fits. the leverage is much higher per hour of design than building one bespoke widget per data shape. the part of this that keeps me genuinely excited (not in a marketing way, in a "this is interesting work" way) is that it's a real frontend discipline. it's not prompt engineering, it's not back-end work, it's not visual design exactly. it's grammar design. choosing a small set of structural primitives and a precedent library that lets a constrained agent produce something specific to each moment without going off-script. less component, more grammar. less SPA, more conversation. that's where i think the craft is heading IMO.