llms.blog
LLM Inference2 articles

LLM Inference

Articles

  • Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck

    Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ

    1 min

  • Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving

    Tree-Structured Speculative Decoding: How Multi-Candidate Trees and Tree Attention Accelerate LLM Serving Large language model inference is fundamentally constrained by memory bandwidth during the auto-regressive decoding phase. Because each token generation step requires loading billions of model parameters from high-bandwidth memory (HBM) to compute units for a single token, standard auto-regressive generation operates at low arithmetic intensity. Speculative decoding addresses this bottlene

    1 min