llms.blog
Transformers12 articles

Transformers

Articles

  • The Residual Stream in Transformers: How Linear Additive State, Subspace Projections, and Layer Accumulation Power Modern LLMs

    In standard descriptions of transformer architectures, multi-head attention and feedforward networks receive the vast majority of analytical focus. However, the operational backbone coordinating these sublayers is the residual stream. First introduced in convolutional computer vision models by He et al. (2016) to mitigate vanishing gradients in deep networks, residual skip connections operate in large language models as a persistent, high-dimensional linear communication bus. Rather than proces

    1 min

  • Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training

    Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training In deep transformer architectures, weight initialization is the primary determinant of whether a trillion-token pre-training run converges smoothly or diverges during the first thousand steps. When training networks with 80 to 120 layers (such as Llama 3 70B, GPT-4, or deep mixture-of-experts models), naive application of classical Gaussian or uniform

    1 min

  • Multi-Query Attention: How Single Key-Value Head Sharing Slashed Transformer Serving Bottlenecks

    Multi-query attention (MQA) is an architectural modification to the Transformer attention mechanism designed to resolve the memory bandwidth bottleneck during autoregressive token generation. First proposed by Noam Shazeer in the 2019 paper Fast Transformer Decoding: One Write-Head is All You Need, MQA alters the ratio of query, key, and value heads by sharing a single key head and a single value head across all query heads in each Transformer layer. While standard multi-head attention (MHA) pr

    1 min

  • Cross-Attention and Latent Bottlenecks: How Asymmetric Queries Fuse External Modalities in Transformer Architectures

    Standard self-attention mechanisms in Transformer architectures process tokens symmetrically: every token in a sequence queries every other token within the same sequence. While this symmetric formulation drives autoregressive sequence generation and dense contextual representations, it exhibits a quadratic compute and memory profile of O(N^2) relative to sequence length. When scaling to high-resolution sensory inputs (such as raw video frames, high-density audio waveforms, or millions of point

    1 min

  • Chain-of-Thought Reasoning in Large Language Models: How Intermediate Rationales Expand Computational Expressivity and Transformer Capacity

    Autoregressive large language models operate by predicting the next token in a sequence conditioned on preceding context. When presented with complex multi-step problems in mathematics, symbolic manipulation, algorithmic planning, or formal logic, standard standard models tasked with providing immediate answers often fail. This failure stems from a fundamental structural constraint: a model forced to output a final answer immediately must compress the entire multi-step derivation into a single f

    1 min

  • State Space Duality and Mamba-2: How Semiseparable Matrices Unify Attention and SSMs

    For years, deep sequence modeling operated under a structural divide. On one side stood the Transformer architecture, anchored by softmax attention. Softmax attention scales quadratically in compute and memory with sequence length (O(T^2)), but its operations map cleanly to dense General Matrix Multiplications (GEMMs), maximizing utilization of GPU Tensor Cores. On the other side stood Structured State Space Models (SSMs), such as S4 and Mamba-1. Structured SSMs scale linearly in time (O(T · N))

    1 min

  • Diffusion Transformers (DiT): How Patchification and adaLN-Zero Replaced U-Nets in Generative AI

    Generative visual models relied for years on convolutional U-Net architectures to execute iterative denoising. From Denoising Diffusion Probabilistic Models (DDPM) and Ablated Diffusion Models (ADM) to Latent Diffusion Models (LDMs) behind Stable Diffusion, convolutional backbones served as the default engine for image synthesis. While convolutional inductive biases provided translation equivariance and local spatial hierarchies, they imposed architectural rigidities that resisted compute scalin

    1 min

  • The Logit Lens: How Direct Unembedding Decodes Intermediate Representations in Large Language Models

    Autoregressive large language models process input tokens through deep stacks of transformer blocks, generating a final probability distribution over the vocabulary at the final layer. For years, the intermediate computations occurring within these hidden layers were treated as opaque black boxes. Mechanistic interpretability research has demonstrated that modern transformer architectures operate through a structured, iterative refinement process across their residual stream. The logit lens and

    1 min

  • Layer Normalization and RMSNorm in Large Language Models: How Pre-LN, Scaling Invariance, and QK-Norm Stabilize Deep Transformers

    Training deep autoregressive Transformers requires maintaining numerical stability across dozens or hundreds of stacked attention and feed-forward blocks. As models scale from 7 billion to hundreds of billions of parameters, uncontrolled variance growth along the residual stream or unbounded attention logits can trigger catastrophic loss spikes, gradient underflow, or numerical divergence. Normalization layers act as the primary stabilizing mechanism in modern Large Language Models (LLMs). Whil

    1 min

  • Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck

    Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ

    1 min

  • Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads

    Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads Training and serving modern large language models requires navigating severe hardware memory and compute constraints. While standard Distributed Data Parallelism (DDP) replicates the entire model across multiple accelerators, modern frontier architectures containing tens or hundreds of billions of parameters exceed the physical memory capacity of any single GPU. Even with 80 GB o

    1 min

  • Grouped-Query Attention: How Head Sharing Slashed KV Cache Memory and Scaled LLM Serving

    Serving large language models at scale presents a fundamental hardware bottleneck: autoregressive decoding is bound by memory bandwidth rather than compute. While the initial prompt processing phase (prefill) operates as compute-bound matrix multiplications, token-by-token generation requires loading billions of cached attention states from GPU High-Bandwidth Memory (HBM) to on-chip SRAM for every single generated token. Multi-Head Attention (MHA), introduced in the foundational Vaswani et al.

    1 min