that definition makes sense if you look at examples reported by others: xitter.cf/TaraViswanathan/status…

not a question but relevant -- you can directly ask gpt5.5 what it means by "goblin"; it's more or less an incentive or "sub-agent"

(too late to edit, config matching mentioned results is: "16 epochs on a random subset of 3.2K* samples") 16 epochs on 400 samples yields: 83% -- AIME'24, 63% -- AIME'25, 66% -- GPQA.

So when does repetition stop helping? Turns out token accuracy on training dataset is a pretty reliable signal for this. Once the model hits ~100% token accuracy on the training set, additional epochs don't bring further gains, which makes it a nice practical stopping criterion.

Part of the story here is termination rate. Models trained with more epochs are much better at concluding their reasoning. With 1 epoch on 51K samples and a token limit of 32K, the model only concludes 24% of the time. With 32 epochs on 1600 samples, that jumps to 89%.

When training Olmo-3 7B on Dolci SFT dataset, 16 epochs on a random subset of 400 samples leads to: 80% (pass@16) on AIME'24, 63% on AIME'25, 62% on GPQA; while one epoch on over 51K samples yields: 47% -- AIME'24, 50% -- AIME'25, 24% -- GPQA.

Common knowledge in ML: more unique training data → better generalization. Turns out this doesn't hold for long-CoT SFT. Under a fixed update budget, repeating a small dataset multiple times beats training on more unique samples. And it's not even close.

a bit late reply, but, doing multiple epochs in SFT seems to solve this issue: arxiv.org/abs/2602.11149

The approach you mentioned is equivalent to the "Naive Bidir." variant from the ablation study in the paper - we can see that it improves results over the baseline (fully causal decoder-only model):

Then we have the Flan-T5 paper (arxiv.org/abs/2210.11416) that tested different architectures and objectives, with prefix-lm (Flan-U-PaLM) being the strongest model. (3/3)