oai uses self-play for prompt injection robustness arxiv.org/abs/2607.26115

Exactly -- we evaluated untrained RLMs on LongCoT and think training to decompose tasks would boost performance. Two eval settings to consider: (1) Python may be used to avoid reasoning (dotted bars) (2) LLM/RLM does all the reasoning (solid bars) xitter.cf/a1zhang/status/2045211…

I second the suggestion to learn about async RL by using prime-RL, their code is very clear and hackable. But if you prefer to learn about async RL for LLMs via papers, I recommend these early ones: - arxiv.org/abs/2410.18252 - arxiv.org/abs/2503.18929 xitter.cf/i/status/2038308395997…

On 3: with a similar objective and off-policy buffer setup, we found the answer is task-dependent. - For math reasoning (MR), being closer to the on-policy setting helped (e.g. m=0.9). - For preference fine-tuning (PFT) of a different model, it was actually optimal to sample older, high-reward samples from the replay buffer as much as half the time (m=0.5), rather than be as on-policy as possible (m=1). I think the answer's dependence on the model and data arises for the reason you gave: if the older policy samples interesting/diverse trajectories you can exploit, going highly off-policy is fine. If/when fresher rollouts help, exploration may be the bottleneck. Caveat: OAPL's setup has notable differences (like using the inference policy as the reference), and it could perform better in these settings where we found more off-policy data hurts.

If we reduce the white-box attack's radius to 16/255, then even less robust VLMs can resist white-box attacks better as TTC is scaled. Consistent with the RICH, TTC's benefits don't require a special model -- they require enough base robustness to reason about a given input.

We explore various TTC strategies (e.g. prompt repetition) and attacks (e.g. PGD). Even in white-box settings like this one, the addition of a spec and a TTC scaling strategy does wonders for resisting the attack -- provided there's sufficient base robustness.

Robustness from Inference Compute Hypothesis (RICH): test-time-compute (TTC) benefits grow as the model gains the initial robustness needed to unlock instruction following. The rich get richer.

Give an LLM a spec: more reasoning ➡️ better spec satisfaction. Even on adversarially attacked data. But reasoning benefits fade if attacks are stronger (e.g. white-box or multimodal). Our hypothesis suggests reasoning can stop such attacks. Toy example in the video. 🧵

Sonnet 4.6 also exceeds Opus 4.5 on FrontierScience's Biology Olympiad questions. Notably, scaling Sonnet 4.6's test-time compute with RSA led to the best performance of any model tested. See rsa-llm.github.io/blog for details and more results! xitter.cf/mgdurrant/status/20238…

By step 300 for this Dr. GRPO run, train accuracy is already collapsing (top plot) before a lot of the change in the IS ratio, entropy, etc. (bottom plots). We did try two-sided masking with IcePop, but the CISPO-style clipping had no lower bound because that's what related work does (MiniMax, ScaleRL). If you have a clipping approach in mind (e.g. values for the args in github.com/bbartoldson/tba-p…), I'll run the experiment and reply with what happens.

We explored a few values. As shown below, if 𝜏 is too small, no stability. if 𝜏 is too large, no task optimization. K2 and TBA both work well with 0.005. BTW, if you want to get this result, you should use the K2 KL implementation in this PR: github.com/PrimeIntellect-ai….

🔬 Ablation 4: IcePop Masking instead of Clipping However, we can use TBA with IcePop-style masking instead of CISPO-style clipping, attaining the same performance as our default TBA approach with CISPO-style clipping.

🔬 Ablation 3: Stability approach How does TBA compare to AntGroup's IcePop on highly off-policy data? Both aim to address instability caused by off-policyness, but IcePop was designed to address forward-pass backend mismatches. Our experiments show TBA has greater stability than IcePop when high off-policyness is induced by async training.

🔬 Ablation 2: Reference policy resetting Resetting prevents the stability-adding KL penalty from harming task-reward optimization. This has an effect on performance almost as large as importance sampling's!

🔬 Ablation 1: KL reference policy Why are both TBA and Kimi K2 performant on highly off-policy data? Both use KL regularization. Both compute KL against a moving reference policy: TBA resets every ρ=50 steps (flexible), Kimi K2 uses the inference policy as the reference.