Abstract: Transformer neural networks have emerged as the state-of-the-art in AI across text, audio, image, and video processing tasks. However, the attention mechanism that is core to Transformers ...