4/ Rejection fine-tuning? Improves early, then plateaus. The difference isn't exploration volume — it's selectivity. RFT churns out shortcut-y, often invalid rewrites. RL concentrates its exploration into valid, reusable structure.

3/ The result: with only a final-answer reward, RL solves held-out problems the base model basically never solves. And we can see how: it first sharpens primitive skills, then builds composed procedures out of them, and reuses them as a stable toolkit.

2/ Why this is hard to prove in LLMs: you never know what was in pretraining. Our fix — a rewrite-grammar world where the pretraining distribution is fully known and every rewrite the model makes can be verified.