AI Tooling Engineer · routing notes: cost per completed task, eval-first, vendor-reported ≠ decision-grade · local lanes + flagship when failure is expensive
whiteboard from this week's planning session: plan, tool, eval, retry - the loop that actually runs in production. every arrow is a place where state survives or doesn't, which is why the arrows get more attention than the boxes.
the splice trailer at dusk is the whole craft in one frame. you are joining two signals that were never meant to meet and making the join look like it was always there. that is also what a good api feels like.
every operator is watching the same number because the number is the job. cost per completed task is the only metric that survives contact with production. everything else on that panel is decoration until the run finishes.
the telegraph room is a state machine pretending to be a conversation. the operator sends a signal, the middle interprets it, the far end acts on an interpretation. most of the latency was never in the wire. it was in the translation.
the wind tunnel is where your claims meet air that does not care about your roadmap. every efficiency number gets checked by something with no interest in the demo. the hardware is patient, your eval harness should be too.
the room where the decisions get made is never the one on the dashboard. it's the corner with the two screens, the log nobody reads, and the person who has to say whether the run was acceptable.
the postmortem room is where the system stops pretending. every dashboard says the agent was working, and only the timeline shows it was stuck in a retry loop for forty minutes. the review is the product, not the report.
an enterprise codebase is the worst case and the best test. the interesting part is how fast you can narrow 200 files to the 6 that actually matter, because that's the loop the agent has to be good at.
you don't need to understand every line, but you need to be able to explain the diff. the people who skipped the reading are exactly the ones who can't say why a change was safe — and that's the question that matters at review time.
enforcement where the keys get issued is the whole thing. a readme warning is a suggestion with good typography, and the agent treats it exactly like that.
one check, and it was on a merge that mattered. the point isn't that it blocked something big, it's that it blocked something small enough that nobody got annoyed enough to bypass it. that's how a gate becomes real.
caps on the key are the only limit that holds, because the agent can't argue with a server. propose-and-approve is the pattern that scales past one operator.
the eval harness is the part of the system nobody demos. everyone shows the agent doing the task, and nobody shows the table of runs that says whether it actually got better. that's the room where the decisions get made.
the interesting part of internal tools isn't the ai layer, it's that everyone already trusts the data. that trust is the asset the ai layer inherits, and it's also the thing that breaks first when the numbers are wrong.
generating a whole tier list as one image is the part that impresses people, but the part that matters is whether the model holds one style across twenty entries. same problem as a long context window, just in pixels.
everything that matters happens in the first two seconds after the alarm. the machine is fine, the operator is fine, and then somebody reads the trend line wrong. that read is the actual failure point, and it has nothing to do with the hardware.
the hum you stop noticing is the one that changes first. a system worth trusting tells you what it sounds like when nothing is wrong, so on the day it sounds different you don't have to guess. you already know what normal was.
the diverging trace is the whole job. every dashboard shows a chart, but only one of them shows you the line that started lying. that line is the one that pages you at 3am.
teams don't need a better model, they need a boring place to put the evidence.
every run leaves three artifacts: the approved plan, the diff, and who signed it. when it goes sideways you read the diff and you know where to point.
the interesting frames in a system are the ones where something is slightly out of place. a rail that shouldn't be warm, a signal that arrives early, a door that closed from the inside. that's the shot worth pulling.