Session Recap | Bangalore, India 🇮🇳 | 26th Sept | Topic: Beyond Autoregression and Backprop Another amazing session just completed in Bangalore by @ocdbytes & @BLRxZo! 📖 Summary - drive.google.com/file/d/1zEk… @typesafeai's Jev answers typed questions in a single forward pass and returns probabilities, so the whole session came down to whether those probabilities can be trusted. 📝 Topic Overview: A chatbot generates an answer token by token and then sounds confident about it. Jev does neither. You give it state plus typed questions: Noul for yes/no, Choice for one option from your list, Score for ordered levels. It returns a probability over only the options you allowed. That makes the output type-safe, not correct. The "0% hallucination" figure comes from guaranteed schema matching, so Jev can still pick the wrong option from your list with high confidence. ✍🏻 @typesafeai has not published the architecture, weights or training method. So we rebuilt the idea from public papers and labeled every guess as a guess. ✍🏻 Better models got worse at knowing when they are wrong. In Guo et al., ResNet beat LeNet on CIFAR-100 error (30.6% vs 44.9%) but kept saying 0.9 when it was right far less often. ✍🏻 Calibration means that of all answers given at 0.8, about 80% should be right. GPT-4's pre-trained model scored an ECE of 0.007 on an MMLU subset, and post-training pushed it to 0.074. 💡 ECE is the average gap between stated confidence and actual accuracy. ✍🏻 Most "calibration" in the open Jev clones is a single temperature fitted after training, which is a 2017 trick - What does RLCD add beyond a strong base model plus one number? ✍🏻 This is the backprop problem. Jev outputs full probability distributions, which are differentiable, so plain supervised training with a Brier loss already rewards calibration. 💡 So why is RLCD reinforcement learning at all? The candidates are a reward based on downstream outcomes instead of labels, or decisions chained over steps as in the Doom demo?