JEV TIP 004: Turn failure into skill improvements by adding an incident field to your agent’s completion report. Send the incident report and active skill to JEV against explicit gap criteria. If it qualifies, revise or create a skill and then replay incidents as evals before promotion.

JEV TIP 003: When an agent marks an attempt as failed, send JEV: > task contract > failure report > failed strategy > proposed retry strategy > criteria for what counts as a legitimate retry If the retry doesn’t meet the criteria, force a replan, escalate, or stop.

JEV TIP 002: Run JEV outside the loop to track task drift. On each state transition, send the task contract + the delta + a checklist to classify drift. Wire the signal into a fuse to trip after a crossed threshold, and then it can redirect, escalate, or pause the agent.

JEV TIP 001: Stop wasting your main model’s tokens figuring out which skill to load. Use JEV to classify the task against skill names + short descriptions, then inject the selected skill directly into the prompt before it reaches the model

glop few understand this

jev evals are done time to write the article then everyone can stop arguing and we can return to building AGI

nature just dropped a new Cat model: Leopardus tilcayo 3 lb model weights 1.4M year training run 10 years in private preview fully local inference native multimodality currently available exclusively in the Bolivian Yungas

how bro looks at when I tell him we're going to puck some shit up today

just got access to JEV and replaced my prefrontal cortex with it, I’m finally free from the burden of choice thank you @typesafeai

One of the interesting ways to do this is to set up a gate where your agent can’t merge the PR until you can answer questions like: Why is it needed? What changes were made? What’s the expected outcome? Why was this implementation chosen over alternatives? How will know or track it worked? And then you can set up a database that tracks the answers with a pointer to the PR and issue. This lets you track actual outcome against expected outcome and gives you a historical view of your decisions and work. The first unlock is your agents don’t need to reconstruct your intent or reasoning through expensive search and reads. The second unlock is you get a systems level overview of your own decisions which means you can start asking questions like: What assumptions were you or the agent repeatedly wrong about? Which PRs underperform expectations? Which PRs got reverted? Where was your model of the problem wrong and why ? Now all of your future work increases in quality because it inherits the quality from everything that came before.

it’s a heavy metal kind of day

this is going to make a phenomenal outer loop over an LLM

if you're running sub-agents playing telephone instead of having a single source of truth for task state, it's no wonder you're going to have a bad time? "one trace to evaluate, and no summaries between sub-agents." it's literally there in the blog post lmao you tested one kind of deployment pattern using isolated state and a summary against single agent managing one source of task state the actual ablation could've been something like: testing shared global state that's updated by every sub/peer agent and read by the orchestrator test different deployment patterns over the same state architecture or task type then variatiosn of the schema for the global state then test the mixture of models doing the work maybe single-agent wins on simple issues with small blast radius but then fails as compelity and blast radius grow? now you would've tested single vs multi-agent and had results that would have made a good blog post sub-agents aren’t dead, single agents aren’t dead but critical thinking, that might be dead lmao