llms.blog
Linear Attention2 articles

Linear Attention

Articles

  • Performers and FAVOR+: How Positive Orthogonal Random Features Linearize Transformer Attention

    The quadratic complexity of standard self-attention has remained a central computational ceiling in Transformer architectures. Because standard attention computes pairwise similarity across all token pairs in a sequence of length $L$, memory consumption and compute scale as $O(L^2)$. For long contexts, high-resolution visual tokens, and biological sequence modeling, this quadratic bottleneck forces strict sequence truncation or aggressive hardware partitioning. In Rethinking Attention with Perf

    1 min

  • Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck

    Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ

    1 min