llms.blog
Attention4 articles

Attention

Articles

  • Cross-Layer Attention: How Sharing Key-Value Tensors Across Transformer Layers Slashes KV Cache Memory

    In autoregressive large language models, the primary operational ceiling for serving long sequences and high batch concurrency is the Key-Value (KV) cache. During standard generation, every transformer layer computes and stores key and value activations for every token in the sequence to prevent quadratic recomputation during subsequent autoregressive decoding steps. While this mechanism transforms inference time complexity from $O(N^2)$ to $O(N)$ per generated token, it introduces a massive me

    1 min

  • Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces

    Multi-Head Attention in Large Language Models: How Query, Key, and Value Projections Route Information Across Subspaces The attention mechanism serves as the computational core of modern Transformer architectures, providing a mechanism for tokens to dynamically exchange information across arbitrary sequence positions without recurrent state transitions or fixed convolutional receptive fields. While single-head attention computes a single set of attention weights per token pair, modern large lan

    1 min

  • Sliding Window Attention in Large Language Models: How Bounded Receptive Fields, Interleaved Layers, and Rolling KV Buffers Scale Contexts

    Standard causal multi-head attention imposes two severe computational constraints as sequence lengths expand into tens or hundreds of thousands of tokens. First, calculating pairwise query-key dot products scales quadratically with sequence length, requiring $O(N^2)$ floating-point operations. Second, autoregressive generation requires caching key and value projections for all preceding tokens, causing the key-value (KV) cache to grow linearly with sequence length $O(N)$ across all layers and at

    1 min

  • Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    1 min