Reading through these papers has given me a better understanding of why RL scaling laws are so messy compared to those from pretraining. Pretraining scaling laws and RL scaling laws are two completely different things for several reasons: 1. Defining compute: Pretraining has a very clean compute footprint of C = 6ND. RL compute is more complex to capture due to the presence of both sampling and policy updates. Some papers try to maintain the same FLOP estimate for compute, while others measure compute in terms of GPU hours. The efficiency of our training framework can cause the relationship between FLOPs / GPU hours to vary pretty drastically. 2. Intra versus inter-model extrapolation: Pretraining scaling laws fit trends across many model training runs with different settings to understand how model / data size (and compute) impact results. This allows us to extrapolate teh results of future training runs. In RL, we fit scaling laws both within an individual training run (intra-model extrapolation) and across training runs (inter-model extrapolation). Intra-model extrapolation is not necessary for pretraining because it is more stable, while RL is extremely sensitive to the exact training configuration being used. 3. Measuring performance: Pretraining scaling laws predict a very particular performance metric: the cross entropy loss (or some other related entropy metric) measured over an in-domain, held-out validation set. This is a stable performance metric that is typically computed over a very diverse dataset (i.e., some random sample from the pretraining corpus). RL scaling laws maintain the practice of computing performance over an in-domain validation set. However, the performance metric that they predict is reward (or accuracy) on a validation set. This is a downstream performance metric, and it can fluctuate drastically depending on the benchmark being used or the composition of data in that benchmark. 4. Lack of standardization: There are generally just more knobs that we can change in RL compared to pretraining. The design space is massive, and we are not sure (yet) which design decisions impact the scaling properties of RL. Several papers have focused on this topic and made meaningful progress on understanding what changes actually impact RL scaling. However, this does not change the fact that slight differences in the RL training setup can completely change observed scaling trends for RL. For this reason, many papers are comparing apples to oranges in terms of their recommendations for RL scaling, making progress on the topic difficult. There are even some papers that have completely opposite findings from each other, and this is likely do to slight differences in their exact GRPO formulation.