llms.blog
Distributed Training6 articles

Distributed Training

Articles

  • Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures

    Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures Scaling dense Large Language Models (LLMs) requires activating every parameter in the network for every token in a sequence. While techniques like Tensor Parallelism, Pipeline Parallelism, and Fully Sharded Data Parallelism distribute billions of dense parameters across clusters, computational cost scales linearly with parameter count. Mixture-of-Experts (Mo

    1 min

  • Activation Checkpointing in Large Language Models: How Selective Recomputation Eliminates Memory Bottlenecks

    Large language model pre-training and fine-tuning are fundamentally constrained by GPU memory (VRAM). While distributed techniques such as Fully Sharded Data Parallel (FSDP), ZeRO, and Tensor Parallelism successfully shard model parameters, optimizer states, and gradients across hundreds or thousands of GPUs, activation memory presents a distinct scaling bottleneck. During the forward pass of a transformer model, intermediate tensor outputs must be preserved in GPU memory so that backpropagatio

    1 min

  • Sequence Parallelism in Large Language Models: How Megatron-SP, DeepSpeed Ulysses, and RingAttention Distribute Long Contexts

    Sequence Parallelism in Large Language Models: How Megatron-SP, DeepSpeed Ulysses, and RingAttention Distribute Long Contexts Training and serving frontier large language models on context windows spanning hundreds of thousands to millions of tokens introduces a fundamental memory barrier. While model parameters can be distributed across GPUs using Tensor Parallelism (TP) or Fully Sharded Data Parallelism (FSDP / ZeRO), activation memory scales directly with sequence length $S$. For sequence le

    1 min

  • Pipeline Parallelism in Large Language Models: How GPipe, 1F1B Scheduling, and Interleaving Tame Memory and Bubbles

    Training frontier large language models with tens or hundreds of billions of parameters exceeds the physical memory capacity of any individual GPU. While intra-node sharding strategies such as Tensor Parallelism partition individual matrix multiplications across accelerators over high-speed NVLink interconnects, scaling across multi-node clusters encounters strict hardware boundaries. Tensor Parallelism requires multiple collective All-Reduce communications per transformer layer. Across standar

    1 min

  • Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training

    Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training Training large language models across distributed GPU clusters introduces a fundamental memory bottleneck. In traditional Distributed Data Parallel (DDP) setups, every GPU maintains an identical copy of model weights, optimizer states, and gradients while processing independent data batches. As models scale from billions to hundreds of billions of parameters, static model s

    1 min

  • Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads

    Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads Training and serving modern large language models requires navigating severe hardware memory and compute constraints. While standard Distributed Data Parallelism (DDP) replicates the entire model across multiple accelerators, modern frontier architectures containing tens or hundreds of billions of parameters exceed the physical memory capacity of any single GPU. Even with 80 GB o

    1 min