Mattia Villani

[LG] The Topos of Transformer Networks M J Villani, P McBurney [King’s College London] (2024) arxiv.org/abs/2403.18415 - The transformer neural network architecture has achieved great success in natural language processing tasks. This paper provides a theoretical analysis of the transformer through the lens of topos theory. - Many common neural network architectures like convolutional and recurrent networks can be embedded in a pretopos of piecewise-linear functions. The transformer lives in the topos completion of this pretopos. - This suggests transformers are higher-order reasoners, while other architectures are first order. The attention mechanism selects parameters for a neural network, then evaluates the network. - The decomposition into 'choose' and 'evaluate' morphisms shows transformers require exponentials and evaluation maps only found in the topos completion. - The internal logic of transformers is richer and more expressive than other architectures. This provides a new perspective on their unprecedented success.