The methodological choice in this paper from the university of Munich released yesterday, should rewire how you evaluate LLMs in production: the surface fidelity gap. It measures the distance between how on-task a model sounds and whether it actually respected the brief. Llama-70B has the largest gap. +0.80. The output reads aligned. The constraints are gone. The knows-but-violates rate measures non-compliance despite the model correctly recalling the rule. It ranges from 8% to 99% across seven models. Structured checkpointing reduces it. It does not close the gap. For anyone designing business processes on top of LLMs, the lesson is methodological. Reading the output is not evaluation. Asking the model 'what were the rules?' is not evaluation. Evaluation is checking the output against a structured brief with binary-decidable constraints. That was the only mechanism the paper found that reliably catches violations. If your QA layer is a human skim or a model self-check, you are measuring how aligned the output sounds. Not whether it complied.

Software, data, and agents. One platform.

Tenacity is also building the tools that keep you on the path. xitter.cf/GPallocca/status/20493…

Semgrep runs the pattern rules across the stack. Models have tells. They repeat the same bad habits — 2021 Stack Overflow auth patterns, deprecated APIs, the same three SQL injection shapes. Plus PHPMD on the Laravel side and ESLint-security on the React side — each catches the dialect-specific bad habits the generic rules miss. Fails the build before a human ever sees it.

KPIs that matter need no explanation.

Got an extremely charismatic character from my Midjourney experimentation, what should I do with her? 🧐

Hustling weekend. What are you building ?

L'Oms fa brute force su tutti gli scenari, così potrà sempre dire che l'aveva detto. #poverinoi

Amazon AWS developer day, the world is flying to the Cloud, are you still on the ground?