llms.blog
Deep Learning28 articles

Deep Learning

Articles

  • Neural Ordinary Differential Equations: How Continuous-Depth Dynamics and Adjoint Sensitivity Solve the Memory Bottleneck in Deep Learning

    Neural Ordinary Differential Equations: How Continuous-Depth Dynamics and Adjoint Sensitivity Solve the Memory Bottleneck in Deep Learning Deep neural networks are traditionally structured as a discrete sequence of layers. An input tensor passes through layer after layer, transforming its representation at fixed, integer time steps. In standard architectures like Residual Networks (ResNets), each successive block computes an additive update: h_{t+1} = h_t + f(h_t, \theta_t) In 2018, researche

    1 min

  • Masked Autoencoders: How Asymmetric Encoders, High Masking Ratios, and Pixel Reconstruction Scaled Vision Transformers

    Masked Autoencoders: How Asymmetric Encoders, High Masking Ratios, and Pixel Reconstruction Scaled Vision Transformers Self-supervised pre-training transformed natural language processing through masked language modeling, popularized by BERT (Devlin et al., 2018). By hiding a subset of input tokens and training a bidirectional Transformer to predict the missing words from context, models learned rich, generalizable linguistic representations without manual annotations. Adapting this masked pre

    1 min

  • Gradient Low-Rank Projection (GaLore): How Dynamic Subspaces Enable Full-Parameter LLM Pre-Training with Low-Rank Memory

    Pre-training foundation large language models has historically required massive compute clusters, largely due to the memory footprint of optimizer states rather than the model weights themselves. While parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) freeze weights and introduce small adapter matrices, they fail when applied to pre-training from scratch because they restrict parameter updates to a fixed, static low-rank manifold. Gradient Low-Rank Projection (GaLore),

    1 min

  • Variational Autoencoders: Mathematical Derivation of the ELBO, the Reparameterization Trick, and Mitigating Posterior Collapse

    Variational Autoencoders: Mathematical Derivation of the ELBO, the Reparameterization Trick, and Mitigating Posterior Collapse Traditional autoencoders map high-dimensional data into deterministic latent vectors. While effective for dimensionality reduction and non-linear feature compression, deterministic autoencoders fail as generative models because their latent representations lack continuous probabilistic structure. Unregularized latent spaces contain wide regions of empty space and severe

    1 min

  • Vector Quantization and VQ-VAEs: How Discrete Codebooks, Straight-Through Estimators, and Commitment Losses Power Multimodal Tokenization

    Autoregressive sequence models excel at discrete token prediction. In natural language processing, words and subwords map onto categorical vocabularies where token identity is exact and cross-entropy loss provides direct likelihood optimization. Continuous multi-dimensional signals—such as images, video frames, raw audio waveforms, and robotic sensorimotor trajectories—present a fundamental mismatch for standard transformer architectures. Historically, variational autoencoders (VAEs) bridged ra

    1 min

  • The Gumbel-Softmax Trick: How Continuous Relaxations Enable Differentiable Discrete Sampling

    The Gumbel-Softmax Trick: How Continuous Relaxations Enable Differentiable Discrete Sampling In modern deep learning, end-to-end training depends on reverse-mode automatic differentiation. When an architecture operates on continuous tensors, computing gradients via the chain rule is straightforward. However, many foundational artificial intelligence problems involve discrete choices: selecting tokens from a fixed vocabulary, routing tokens to expert networks in a Mixture-of-Experts (MoE) archit

    1 min

  • The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations

    The Information Bottleneck in Deep Learning: How Mutual Information Compression Shapes Generalization and Neural Representations Classical statistical learning theory struggles to explain why overparameterized deep neural networks generalize well to unseen test data. Traditional complexity measures such as Vapnik-Chervonenkis (VC) dimension and Rademacher complexity scale with the raw count of tunable weights, predicting severe overfitting when parameter counts exceed dataset sample sizes. Yet

    1 min

  • The Lottery Ticket Hypothesis in Large Language Models: How Sparse Subnetworks and Iterative Magnitude Pruning Retain Transformer Capacity

    The Lottery Ticket Hypothesis in Large Language Models: How Sparse Subnetworks and Iterative Magnitude Pruning Retain Transformer Capacity Modern large language models operate under extreme overparameterization. Frontier architectures allocate tens or hundreds of billions of parameters to achieve low perplexity and robust generalization across reasoning, code generation, and factual retrieval. Yet empirical pruning consistently demonstrates that post-training models can lose 30% to 50% of their

    1 min

  • Linear Mode Connectivity in Deep Neural Networks: How Permutation Symmetries, Git Re-Basin, and the Single-Basin Hypothesis Unify Model Checkpoints

    title: "Linear Mode Connectivity in Deep Neural Networks: How Permutation Symmetries, Git Re-Basin, and the Single-Basin Hypothesis Unify Model Checkpoints" slug: "linear-mode-connectivity-in-deep-neural-networks-how-permutation-symmetries-git-re-basin-and-the-single-basin-hypothesis-unify-model-checkpoints" feature_image: "https://cms.llms.blog/content/images/2026/08/linear-mode-connectivity-cover.png" excerpt: "Linear Mode Connectivity reveals how neural network checkpoints connect along flat

    1 min

  • The Softmax Bottleneck in Large Language Models: Mathematical Foundations, Matrix Rank Limits, and Mixture of Softmaxes

    title: "The Softmax Bottleneck in Large Language Models: Mathematical Foundations, Matrix Rank Limits, and Mixture of Softmaxes" slug: "the-softmax-bottleneck-in-large-language-models-mathematical-foundations-matrix-rank-limits-and-mixture-of-softmaxes" status: "published" feature_image: "https://cms.llms.blog/content/images/2026/08/softmax-bottleneck-cover-1.png" excerpt: "A standard linear projection followed by Softmax caps the rank of predicted log-probability distributions to the hidden dim

    1 min

  • The Curse of Multilinguality in Large Language Models: Capacity Dilution, Tokenizer Fertility, and Representation Interference

    The Curse of Multilinguality in Large Language Models: Capacity Dilution, Tokenizer Fertility, and Representation Interference Training a single transformer foundation model to process dozens or hundreds of languages is one of the central goals of modern natural language processing. In theory, massive multilingual pre-training unlocks positive cross-lingual transfer: low-resource languages gain syntactic, factual, and reasoning capabilities from the rich supervision available in high-resource l

    1 min

  • Logit Soft-Capping in Large Language Models: How Tanh-Based Bounds Prevent Attention Saturation and Numerical Instability

    In modern autoregressive Transformer training, maintaining numerical stability across trillions of tokens requires strict control over intermediate activation magnitudes. As models grow deeper and wider, pre-softmax logits in self-attention mechanisms and final vocabulary projection layers frequently drift toward extreme values. When logit values grow unconstrained, standard softmax normalization enters a saturation regime where output probabilities collapse into near one-hot distributions, caus

    1 min

  • Reversible Transformers: How Invertible Residual Blocks Eliminate Activation Memory in Deep Networks

    Training deep transformer models is primarily bounded by activation memory rather than parameter storage. During the forward pass of standard backpropagation, automatic differentiation engines cache intermediate activations across every attention head, layer normalization, and feed-forward sublayer so they can be referenced during the backward pass to evaluate gradients. For a transformer with N layers, sequence length L, batch size B, and hidden dimension d_model, storing these activations requ

    1 min

  • Weight Tying in Large Language Models: Mathematical Foundations, Geometric Bottlenecks, and Modern Architectural Trade-Offs

    Weight Tying in Large Language Models: Mathematical Foundations, Geometric Bottlenecks, and Modern Architectural Trade-Offs In autoregressive language models, the embedding layer at the input and the unembedding projection layer at the output serve as the two bridges between discrete vocabulary tokens and the continuous hidden representation space. In the foundational Transformer architecture (Vaswani et al., 2017) and early generative models like GPT-2 (Radford et al., 2019), the weights of th

    1 min

  • Rank Collapse in Transformer Architectures: Why Pure Attention Decays Doubly Exponentially and How Residuals Restore Expressivity

    Rank Collapse in Transformer Architectures: Why Pure Attention Decays Doubly Exponentially and How Residuals Restore Expressivity The foundational premise of the Transformer architecture is that self-attention acts as a dynamic routing engine, computing pairwise affinities across sequence positions to contextualize representations. However, theoretical analysis reveals an underlying instability: stripped of its auxiliary components, a pure self-attention network rapidly destroys its own express

    1 min

  • Arithmetic Intensity and the Roofline Model: Why LLM Generation Is Memory-Bound and Prefill Is Compute-Bound

    Arithmetic Intensity and the Roofline Model in Large Language Models: Why Generation Is Memory-Bound and Prefill Is Compute-Bound Every optimization in modern large language model (LLM) serving exists to solve a single hardware reality: running a Transformer model is not one computational problem, but two completely distinct workloads operating on opposite sides of a physical performance barrier. When an LLM processes a prompt during the prefill phase, it saturates modern GPU Tensor Cores, ope

    1 min

  • Emergent Outlier Features in Large Language Models: Why Hidden Dimension Spikes Arise at Scale and How They Reshape Quantization

    Emergent Outlier Features in Large Language Models: Why Hidden Dimension Spikes Arise at Scale and How They Reshape Quantization When language models scale past approximately 6.7 billion parameters, their internal representations undergo a sharp qualitative phase transition. In smaller models (125M to 2.7B parameters), hidden state activations remain relatively compact, bounded within predictable normal distributions across all embedding dimensions. However, as demonstrated by Dettmers et al. (

    1 min

  • The Linear Representation Hypothesis in Large Language Models: How Neural Geometry Organizes Concepts, Latent Truth, and Steerable Features

    The Linear Representation Hypothesis in Large Language Models: How Neural Geometry Organizes Concepts, Latent Truth, and Steerable Features Deep neural networks are composed of alternating non-linear activation functions, multi-head attention operations, and high-dimensional matrix projections. Despite this architectural non-linearity, empirical research across mechanistic interpretability reveals a striking geometric regularity: within the intermediate representation spaces of large language m

    1 min

  • Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training

    Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training In deep transformer architectures, weight initialization is the primary determinant of whether a trillion-token pre-training run converges smoothly or diverges during the first thousand steps. When training networks with 80 to 120 layers (such as Llama 3 70B, GPT-4, or deep mixture-of-experts models), naive application of classical Gaussian or uniform

    1 min

  • Cross-Attention and Latent Bottlenecks: How Asymmetric Queries Fuse External Modalities in Transformer Architectures

    Standard self-attention mechanisms in Transformer architectures process tokens symmetrically: every token in a sequence queries every other token within the same sequence. While this symmetric formulation drives autoregressive sequence generation and dense contextual representations, it exhibits a quadratic compute and memory profile of O(N^2) relative to sequence length. When scaling to high-resolution sensory inputs (such as raw video frames, high-density audio waveforms, or millions of point

    1 min