a brand new model paradigm! introducing 𝚌𝚕𝚊𝚒𝚛𝚒𝚌: a system one model for extracting, computing, estimating, and predicting values from unstructured inputs. blog.cloudflare.com/clairic-…

Closed frontier LLMs are dangerous In most post-training reasoning efforts, a pre-trained base model trains on its own chains of thought (CoTs) when a final-answer verifier says the answer is correct. This is the basic principle of RL methods based on outcome rewards, used for STEM reasoning, coding tasks and even ethical alignment. Different algorithms have weigh the CoTs differently, but they all reinforce CoTs. As we've shown, training on one's own data, while not knowing how to distinguish it from someone else's data, causes delusions. See love4all.ai/files/why-it-is-… Everyone knows the different between something they did and something someone else did. This is essential for safety. If I write "Kevin is unfriendly" on a napkin, go for a walk, and then read the message on the napkin not knowing who wrote it, then my belief that "Kevin is unfriendly" is reinforced. These delusions are essentially confabulations, over-confidence that results from confusing P(o | a) with P(o | do(a)), where o is an outcome and a the action (CoT tokens). Unlike in past work, here I'm alerting everyone to this danger when training models to think, a typical step in the process of building agents. When the verifier supplies one bit about the answer, algorithms like ReST or GRPO spread that bit over every token of the CoT trace, so a reasoning step that is wrong but happens to precede a correct answer (a false lemma, a shortcut that only works on the training distribution, a sycophantic or test-gaming rationale, a subtle bias) is reinforced. Traits that are merely neutral are sharpened by mode-seeking self-distillation, and traits that are spuriously correlated with success grow exponentially. The damage may be invisible to the researchers and metrics, and in-distribution accuracy rises, but it shows up off-distribution. There has been some work on this. For example, Appendix G of the STaR paper arxiv.org/pdf/2203.14465 touches on this potential bias amplification but only on the surface. Some researchers have proposed process rewards as the solution arxiv.org/pdf/2211.14275 However, self-delusion by mistaking ones actions with observations produced by a third party highlights that this problem can be much more serious. Something subtle, invisible to the metrics and to the researchers, could lurk in the synthetic data. When deploying the AI out of distribution, that something could be fatal. Don't trust RL, RLHF, etc, blindly unless you really understand how the verification has been done. Corollary, don't trust any of the closed frontier models.

Lots of non-tech posts from me lately, so here's proof I still write code. I swapped the decision model inside my root-cause agent from @typesafeai Jev to @TypeLLM. Same code, 60 incidents. Same accuracy. TypeLLM proved its answer 10x more often, for 36% less. Jev was faster.