So, "overlooked for 8+ years" turned out to be not quite right, it has been implemented in Google's Flaxformer library for at least 2 years Now the question is why it didn't catch on. If it had no significant effect on performance, this idea is still worth revisiting for the purpose of facilitating quantization of large models with minimal performance impact The fact that 97% of outlier activation LLMs occur in whitespace and punctuation positions does, indeed, hint at the ability of "attending to nothing" in order to get rid of outlier weights being potentially valuable Flaxformer implementation of "softmax with extra logit", which is exactly the same as the one suggested by Evan: github.com/google/flaxformer…