🚨BREAKING: Princeton just proved that AI agents are throwing away the most valuable data they'll ever collect. And nobody noticed because it looks like normal conversation. Every time an AI agent takes an action, it receives what researchers call a "next-state signal." A user reply. A tool result. A terminal output. A test verdict. Every existing system takes that signal and uses it as context for the next response. Then discards it forever. The Princeton team just proved this is one of the most expensive mistakes in AI engineering. Because that signal contains two things nobody was extracting. First: an implicit score. A user who re-asks a question is telling you the agent failed. A passing test is telling you it succeeded. A detailed error trace is scoring every step that led to it. This is a live, continuous reward signal hiding inside every interaction. Free. Universal. Completely ignored. Second: a correction direction. When a user writes "you should have checked the file first," they're not just saying the response was wrong. They're specifying which tokens should have been different and how. That's not a scalar reward. That's token-level supervision. And scalar rewards throw every single bit of it away. They built a system called OpenClaw-RL around recovering both. Then they ran the experiment that changes everything. An agent started with a personalization score of 0.17. After just 36 normal conversations, with no new training data, no labeled dataset, and no human annotations, the combined method hit 0.81. The agent didn't get retrained. It got used. That's the part nobody is talking about. The model was serving live requests at the same time it was being trained on them. Four completely decoupled loops running simultaneously. Policy serving. Rollout collection. Reward judging. Weight updates. None waiting for the others. The agent gets smarter every time someone talks to it. And the deeper the task, the more it matters. On long-horizon agentic tasks, outcome-only rewards give you a signal at the very end of a trajectory and nothing in between. Their process reward model scores every single step using the live next-state signal as evidence. Tool-call accuracy jumped from 0.17 to 0.30. GUI accuracy improved further on top of that. This creates a shift nobody has fully reckoned with yet. The current paradigm: collect data offline, train in batches, deploy, hope it works. The new paradigm: deploy, extract training signal from every interaction, update continuously, improve automatically. Every conversation is training data. Every correction is a gradient. Every re-query is a reward signal. The agents that figure this out first won't need bigger datasets. They'll just need more users.