Looped Transformers should get better the more they loop. In practice, more loops often make them worse. We call this latent overthinking.
We found why, and how to fix it. Introducing InfiLoop 🧵
📄 arxiv.org/abs/2610.11570
Our fix: loop-native residual connections.
Instead of only the latest state, InfiLoop keeps a running summary of past states, weighted by content + learned decay—downweighting noisy updates while memory stays constant as loops grow.
With just 7M params:
• 97.9% on Sudoku-Extreme
• 13.6% pass@2 on ARC-AGI-2
• Still improving beyond 20,000 test-time loops
More loops → better reasoning.
💻 github.com/pixeli99/InfiLoop
w/ @pumpkinnnnne @luu_yinn @Shiwei_Liu66
