[LG] ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization H You, Y Guo, Y Fu, W Zhou… [Georgia Institute of Technology] (2024) arxiv.org/abs/2406.05981 - The paper proposes ShiftAddLLM, a method to accelerate pretrained large language models (LLMs) via post-training shift-and-add reparameterization, creating efficient multiplication-less models. - ShiftAddLLM quantizes each weight matrix into binary matrices paired with group-wise scaling factors. The associated multiplications are then reparameterized into 1) bitwise shifts between activations and scaling factors, and 2) queries and adds based on the binary matrices. - To mitigate accuracy loss from reparameterization, a multi-objective optimization method is presented to minimize both weight quantization error and output activation error. - An automated bit allocation strategy is also introduced to assign a mixed number of bits to each layer based on their vulnerability, further improving accuracy and efficiency. - Experiments on 5 LLM families and 8 tasks show ShiftAddLLM achieves higher accuracy (5.6 and 22.7 lower perplexity on average) at comparable or even lower latency than state-of-the-art quantized LLMs at 3 and 2 bits respectively. It also reduces over 80% memory and energy vs original LLMs.