llms.blog
Scaling Laws2 articles

Scaling Laws

Articles

  • Maximal Update Parametrization (muP): How Tensor Programs Enable Zero-Shot Hyperparameter Transfer in LLM Pre-Training

    Pre-training a frontier large language model requires hundreds of thousands of GPU hours and millions of dollars in compute. At that scale, traditional hyperparameter tuning is financially and operationally impossible: teams cannot sweep learning rates, weight initializations, or optimizer betas across multiple 70B parameter runs to find the loss minimum. Historically, practitioners relied on ad-hoc heuristic extrapolation or manual guesses from small runs, often leading to sub-optimal loss curv

    1 min

  • Chinchilla Scaling Laws: How Compute-Optimal Pre-Training and Inference Economics Reshaped LLM Architecture

    Before 2022, frontier artificial intelligence labs scaled large language models primarily by increasing parameter counts while keeping training dataset sizes relatively static. OpenAI's 2020 GPT-3 model trained 175 billion parameters on 300 billion tokens. DeepMind's 2021 Gopher trained 280 billion parameters on 300 billion tokens, and Megatron-Turing NLG scaled to 530 billion parameters on 270 billion tokens. These allocation decisions followed early empirical scaling laws that suggested parame

    1 min