Pre-Training6 articles

Pre-Training

Articles

Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training
Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training In deep transformer architectures, weight initialization is the primary determinant of whether a trillion-token pre-training run converges smoothly or diverges during the first thousand steps. When training networks with 80 to 120 layers (such as Llama 3 70B, GPT-4, or deep mixture-of-experts models), naive application of classical Gaussian or uniform
August 21, 20261 min
Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics
Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics In large language model pre-training, data composition is as consequential as parameter count and compute budget. While early foundation models relied on raw natural frequencies or manual heuristic filtering to construct training corpora, empirical scaling laws have shown that arbitrary domain ratios cause severe compute inefficiencies. Over-sampling redundant te
August 21, 20261 min
Loss Spikes and Training Stability in Large Language Models: How Attention Logit Drift, z-loss, and QK-Norm Prevent Gradient Explosions
During the pre-training of modern large language models, few operational failures are as costly as loss spikes. When training clusters containing thousands of GPUs run for weeks across trillions of tokens, a sudden, discontinuous surge in cross-entropy loss can corrupt optimizer momentum buffers, induce numerical overflow in half-precision representations, and permanently degrade downstream model capabilities. In severe cases, models experience catastrophic divergence, forcing engineering teams
August 21, 20261 min
Adaptive Optimizers in Large Language Model Pre-Training: How AdamW, Adafactor, and Lion Scale Gradient Updates Across Billions of Parameters
Adaptive Optimizers in Large Language Model Pre-Training: How AdamW, Adafactor, and Lion Scale Gradient Updates Across Billions of Parameters Large language model pre-training requires optimizing billions of parameters over trillions of tokens across distributed GPU clusters. Standard stochastic gradient descent (SGD) fails in this regime because Transformer loss landscapes are severely ill-conditioned, with gradient magnitudes differing by orders of magnitude across layers and token positions.
August 20, 20261 min
Data Filtering and Deduplication in Large Language Models: How Heuristics, MinHash LSH, SemDeDup, and Quality Classifiers Curate Pre-Training Corpora
Data Filtering and Deduplication in Large Language Models: How Heuristics, MinHash LSH, SemDeDup, and Quality Classifiers Curate Pre-Training Corpora Pre-training data quality governs the downstream reasoning, factual accuracy, and sample efficiency of foundation large language models (LLMs). While early pre-training efforts relied on uncurated or lightly processed dumps from web repositories like Common Crawl, modern state-of-the-art models depend on multi-stage data curation pipelines that fi
August 19, 20261 min
Warmup-Stable-Decay (WSD): How Decoupled Annealing Replaced Cosine Decay in Modern LLM Pre-Training
For years, foundation model pre-training adhered to a standard optimization convention: linear learning rate warmup followed by a full-horizon cosine decay. Adopted across GPT-3, PaLM, Chinchilla, and LLaMA, cosine annealing provided stable convergence across diverse parameter scales. However, it introduced a severe structural limitation: the learning rate schedule is rigidly tied to a fixed, upfront token budget. If a team decides to extend pre-training mid-run, branch into domain-specific vari
August 19, 20261 min

Pre-Training

Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training

Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics

Loss Spikes and Training Stability in Large Language Models: How Attention Logit Drift, z-loss, and QK-Norm Prevent Gradient Explosions

Adaptive Optimizers in Large Language Model Pre-Training: How AdamW, Adafactor, and Lion Scale Gradient Updates Across Billions of Parameters

Data Filtering and Deduplication in Large Language Models: How Heuristics, MinHash LSH, SemDeDup, and Quality Classifiers Curate Pre-Training Corpora

Warmup-Stable-Decay (WSD): How Decoupled Annealing Replaced Cosine Decay in Modern LLM Pre-Training