Llama 3 degrades much more than Llama 2 when quantized. 🤔 📌 Most possible reason because Llama 3, trained on a record 15T tokens, captures extremely nuanced data relationships, utilizing even the minutest decimals in BF16 precision fully. Making it more sensitive to quantization degradation. 📌 So sensitive that even the smallest decimal points of each parameter offered by BF16 precision were filled and had a purpose. Other LLMs were trained for far less (2T), and thus did not have time to saturate smaller precision ranges of the parameters like Llama did, and thus are not affected by quantization as much. 📌 It's pretty well established that bigger models are more data efficient / absorb more new information per exposure. They've got more unused information-storing free capacity. But a model that is trained to within an inch of its life does not have that capacity. And because it is jammed packed, that lack of capacity also results in a lack of redundancy within the network. So at a fixed, small parameter count, a model that's been pretrained for far longer—like Llama 3 8b, in comparison to earlier similarly-sized models—might have begun to saturate the capacity of its trainable params, and thus not absorb much from continued training. Therefore, modifying the network is more damaging to overall performance with a smaller model (Llama 3 8B) that has been trained to freakishly high levels of that saturation. i.e. altering one node causes a cascade of brokenness because of that lack of redundancy.