llms.blog
Educational10 articles

Educational

Articles

  • Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics

    Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics In large language model pre-training, data composition is as consequential as parameter count and compute budget. While early foundation models relied on raw natural frequencies or manual heuristic filtering to construct training corpora, empirical scaling laws have shown that arbitrary domain ratios cause severe compute inefficiencies. Over-sampling redundant te

    1 min

  • Self-Consistency Decoding: How Sampling Diverse Reasoning Paths Eliminates Single-Trajectory Fragility in LLMs

    Self-Consistency Decoding: How Sampling Diverse Reasoning Paths Eliminates Single-Trajectory Fragility in LLMs Standard inference pipelines in large language models rely on greedy decoding or low-temperature sampling to generate answers. For tasks requiring direct factual recall or basic classification, greedy token selection generates fluent and accurate outputs. However, when large models are tasked with complex multi-step reasoning, greedy generation exhibits severe structural fragility. In

    1 min

  • Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity

    Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity Rejection sampling fine-tuning (RAFT) has emerged as the practical workhorse of LLM alignment. While PPO-based RLHF dominated early literature, production systems from Llama 2 to DeepSeek-R1 rely on a simpler loop: generate multiple completions per prompt, score them with a reward model, keep the best, and fine-tune on the filtered data. The technique converts the reinforcement learning problem into s

    1 min

  • SwiGLU and Gated Linear Units: How Bilinear Gating Replaced Standard FFNs in Modern LLMs

    Every modern open-weight and frontier large language model, from Meta's LLaMA 3 and Mistral to Alibaba's Qwen 2.5 and DeepSeek-V3, has abandoned the standard two-layer Feed-Forward Network (FFN) originally introduced in the 2017 Transformer architecture. In its place, model architectures have converged almost universally on Gated Linear Units (GLU), specifically the Swish-Gated Linear Unit (SwiGLU). While the original Transformer relied on standard non-linear activations like ReLU or Gaussian E

    1 min

  • Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck

    Linear Attention and Retentive Networks: How Recurrent Duals and Chunkwise Tiling Eliminate the Quadratic Bottleneck Autoregressive large language models built on standard multi-head self-attention face two fundamental scaling ceilings: quadratic compute and memory complexity during pre-training, and linearly expanding key-value (KV) cache memory footprints during autoregressive generation. While optimizations such as FlashAttention reduce memory access overheads and Grouped-Query Attention (GQ

    1 min

  • Context Window Extension in Large Language Models: How Position Interpolation, YaRN, and LongRoPE Scale Sequence Lengths

    Large language models are bounded during pretraining by a fixed sequence length, typically between 2,048 and 8,192 tokens. When standard autoregressive transformers attempt to process sequences beyond this pretraining context window, performance degrades immediately. Perplexity rises sharply and the model loses coherence within a few dozen tokens past the training boundary. Extending this context window by training from scratch on long sequences is computationally prohibitive due to the quadrat

    1 min

  • Attention Sinks in Large Language Models: How StreamingLLM Prevents Perplexity Explosion in Infinite Sequences

    Autoregressive large language models are trained on fixed context windows, yet real-world applications (such as continuous coding agents, live conversation servers, and document streaming pipelines) require models to process unbounded token sequences. When standard LLMs operate on sequences longer than their pre-training context length, computational complexity and key-value (KV) cache memory scale quadratically and linearly, respectively. A seemingly natural workaround is sliding window attent

    1 min

  • In-Context Learning in Large Language Models: How Induction Heads and Attention Circuits Learn Without Weight Updates

    Autoregressive large language models demonstrate the ability to adapt to new tasks, follow few-shot demonstrations, and execute algorithmic patterns entirely within their context windows. Unlike traditional fine-tuning, in-context learning occurs at inference time with frozen model parameters, leaving weights completely unchanged ($\Delta \theta = 0$). For several years following the scaling demonstrations in GPT-3, the internal mechanism governing in-context learning remained an empirical blac

    1 min

  • State Space Models in Large Language Models: How Mamba, S4, and Selective Recurrence Challenge Transformer Attention

    Modern large language models rely almost universally on the Transformer architecture. However, the core mechanism powering Transformers, softmax multi-head self-attention, exhibits fundamental scaling limitations. Specifically, standard self-attention requires quadratic time and memory complexity relative to sequence length during prefilling, alongside a linear memory expansion for the key-value (KV) cache during autoregressive token generation. To circumvent these computational bottlenecks, re

    1 min

  • Rotary Position Embeddings: How Geometry Solved Long Context in Modern LLMs

    Rotary Position Embedding (RoPE) has become the standard positional encoding mechanism across modern large language models, including Meta's Llama series, Mistral, Qwen, and DeepSeek. Unlike earlier techniques that added positional vectors directly to token representations or modified attention matrices with relative distance penalties, RoPE encodes position through geometric rotations in the complex plane. This design enables models to compute relative token distances while processing individu

    1 min