Superposition and Sparse Autoencoders: Mathematical Foundations, the Polysemanticity Bottleneck, and Dictionary Learning in Language Models

For years, attempts to interpret transformer neural networks by inspecting individual neurons encountered an obstinate barrier: polysemanticity. A single neuron in an intermediate multi-layer perceptron (MLP) or residual stream layer rarely corresponds to a single human-interpretable concept. Instead, the same neuron frequently fires on a disparate mixture of inputs, such as Python syntax errors, discussions of Renaissance art, and Spanish verbs. This phenomenon prevents mechanistic interpretability from treating the neuron basis as the fundamental unit of computation.

Recent work in mechanistic interpretability formalized the explanation for this phenomenon through the superposition hypothesis. Because the number of real-world features a language model must represent far exceeds the dimensional capacity of its hidden states, the network projects an overcomplete set of features into almost-orthogonal directions across shared vector spaces. To disentangle these compressed representations post hoc, researchers train Sparse Autoencoders (SAEs) using overcomplete dictionary learning. This article details the mathematical geometry of superposition, the formulation and loss objectives of SAEs, modern architectural variants designed to eliminate shrinkage bias, and their application to model steering and circuit analysis.

1. The Polysemanticity Bottleneck and Linear Representations

To understand why polysemanticity emerges, one must first define what a feature is in the context of neural representations. Under the linear representation hypothesis formalized by Elhage et al. (2022) and Park et al. (2023), a feature is a property of an input that is represented by a specific one-dimensional direction in activation space. If an input contains feature $i$ with scalar intensity $f_i \ge 0$, it contributes a vector component $f_i v_i$ to the activation vector $x \in \mathbb{R}^D$, where $v_i \in \mathbb{R}^D$ is a unit-norm direction vector ($\|v_i\|_2 = 1$).

If features were strictly mapped to the canonical standard basis vectors $e_1, e_2, \dots, e_D$, the model could represent at most $D$ features simultaneously without interference. Each neuron activation would correspond to exactly one feature, yielding pure monosemanticity. However, language modeling requires tracking millions of concepts, syntactic rules, and factual entities ($M$), whereas modern hidden state dimensions range between $D = 2{,}048$ and $D = 12{,}288$.

Because $M \gg D$, mapping each feature to an orthogonal standard basis vector is mathematically impossible. The network must choose between two regimes:

• Dedicated Allocation: Retain orthogonal features, discarding all features beyond the first $D$.
• Superposition Allocation: Pack $M$ features into $D$ dimensions by assigning each feature a distinct direction $v_i$ that is not strictly orthogonal to other directions, tolerating small amounts of cross-feature interference.

2. The High-Dimensional Geometry of Superposition

The viability of superposition relies on high-dimensional geometry. In a $D$-dimensional Euclidean space $\mathbb{R}^D$, the maximum number of mutually orthogonal vectors is exactly $D$. However, as $D$ grows into hundreds or thousands, the number of almost-orthogonal vectors grows exponentially.

According to the Johnson-Lindenstrauss lemma and spherical concentration of measure, if one randomly samples unit vectors on the sphere $\mathbb{S}^{D-1}$, the inner product between any pair $v_i, v_j$ ($i \neq j$) is tightly concentrated around zero with variance $\mathcal{O}(1/D)$:

$$\mathbb{P}(|\langle v_i, v_j \rangle| \ge \epsilon) \le 2 \exp\left(-\frac{D \epsilon^2}{2}\right)$$

When a model computes a representation from a set of active features $\mathcal{S} \subset \{1, \dots, M\}$, the total activation vector $x$ is:

$$x = \sum_{i \in \mathcal{S}} f_i v_i$$

To read out the presence and magnitude of target feature $j$, a downstream linear probe or attention projection computes the projection $\langle v_j, x \rangle$:

$$\langle v_j, x \rangle = \langle v_j, f_j v_j \rangle + \sum_{i \in \mathcal{S}, i \neq j} f_i \langle v_j, v_i \rangle = f_j + \sum_{i \in \mathcal{S}, i \neq j} f_i \langle v_j, v_i \rangle$$

The second term represents cross-talk interference noise. The magnitude of this noise depends on two factors:

• Geometric Coherence: The maximum pairwise inner product $\mu = \max_{i \neq j} |\langle v_i, v_j \rangle|$.
• Feature Sparsity: The cardinality of active features $k = |\mathcal{S}|$. In natural language, most semantic concepts are inactive at any single token position ($k \ll M$).

When $k$ is small and $D$ is large, the aggregate interference $\sum_{i \neq j} f_i \langle v_j, v_i \rangle$ remains small relative to the true signal $f_j$. Non-linear activation functions such as ReLU, GeLU, or SwiGLU then filter out this sub-threshold interference noise. In synthetic toy models, Elhage et al. (2022) demonstrated that neural networks naturally arrange non-orthogonal feature vectors into structured geometric configurations, such as antipodal pairs, regular pentagons, and high-dimensional cross-polytopes, maximizing capacity while minimizing destructive interference.

3. Sparse Autoencoders as Overcomplete Dictionary Learning

Because features are stored as linear directions in superposition rather than aligned with individual neuron axes, standard neuron-level inspection fails. Bricken et al. (2023) and Cunningham et al. (2023) proposed recovering the true underlying feature basis through unsupervised dictionary learning using Sparse Autoencoders (SAEs).

An SAE is trained on intermediate model activations (such as the residual stream $x \in \mathbb{R}^D$ or MLP layer outputs). It maps $x$ to a high-dimensional latent space $f(x) \in \mathbb{R}^F$ where $F \gg D$ (typically $F = 16D$ to $F = 128D$), enforcing that only a minimal subset of latents activate for any single input.

Standard SAE Architecture

The standard autoencoder consists of a linear encoder, a non-linear threshold, and a linear decoder:

$$f(x) = \text{ReLU}(W_{\text{enc}}(x - b_{\\text{dec}}) + b_{\\text{enc}})$$

$$\hat{x} = W_{\text{dec}} f(x) + b_{\text{dec}}$$

Where:

• $x \in \mathbb{R}^D$ is the model activation vector.
• $b_{\text{dec}} \in \mathbb{R}^D$ is the decoder bias (often initialized to the geometric median of the activation dataset).
• $W_{\text{enc}} \in \mathbb{R}^{F \times D}$ is the encoder weight matrix.
• $b_{\text{enc}} \in \mathbb{R}^F$ is the encoder bias vector.
• $W_{\text{dec}} \in \mathbb{R}^{D \times F}$ is the decoder dictionary matrix, where each column $d_j = W_{\text{dec}, :, j}$ represents a unit-norm feature direction ($\|d_j\|_2 = 1$).
• $\hat{x} \in \mathbb{R}^D$ is the reconstructed activation vector.

Training Objective

The SAE loss function balances reconstruction fidelity against feature sparsity:

$$\mathcal{L}(x) = \underbrace{\|x - \hat{x}\|_2^2}_{\mathcal{L}_{\text{recon}}} + \lambda \underbrace{\sum_{j=1}^F |f_j(x)|}_{\mathcal{L}_{\text{sparse}}}$$

The $\mathcal{L}_{\text{recon}}$ term penalizes reconstruction error, ensuring that the decomposed dictionary vectors capture the full information content of the activation space. The $\mathcal{L}_{\text{sparse}}$ term applies an $L_1$ penalty weighted by hyperparameter $\lambda$, driving the majority of latent activations $f_j(x)$ to zero.

Activation Space x in R^D
        |
        v  (Subtract b_dec)
   x - b_dec
        |
        v  (W_enc @ . + b_enc)
   Affine Projection
        |
        v  (ReLU / TopK / JumpReLU)
Latent Features f(x) in R^F  [Sparse: >99% zeros]
        |
        v  (W_dec @ f(x) + b_dec)
Reconstruction x_hat in R^D

4. Overcoming Failure Modes in Sparse Autoencoders

While standard $L_1$-regularized SAEs successfully isolate monosemantic features, they suffer from two major structural failure modes: shrinkage bias and dead latents. Recent research has introduced architectural modifications to resolve both issues.

1. The L1 Shrinkage Problem and TopK SAEs

In standard SAEs, the $L_1$ loss applies a constant gradient penalty $\lambda$ to all positive latent activations:

$$\frac{\partial \mathcal{L}_{\text{sparse}}}{\partial f_j(x)} = \lambda \quad \text{for } f_j(x) > 0$$

To minimize this penalty, the network systematically reduces the magnitude of $f_j(x)$ below its true value. As a result, the reconstructed vector $\hat{x}$ has a systematically underestimated norm, degrading downstream model performance when $\hat{x}$ replaces $x$ in substitution experiments.

To eliminate shrinkage, Gao et al. (2024) introduced TopK SAEs. Instead of adding an $L_1$ loss term, TopK SAEs replace the standard ReLU activation with a hard $\text{TopK}$ operator that directly keeps only the $k$ largest pre-activations and zeroes out the rest:

$$z(x) = W_{\text{enc}}(x - b_{\text{dec}}) + b_{\text{enc}}$$

$$f(x) = \text{TopK}(z(x), k)$$

$$\mathcal{L}_{\text{TopK}}(x) = \|x - W_{\text{dec}} f(x) - b_{\text{dec}}\|_2^2$$

Because no $L_1$ penalty is applied to the active latents, TopK SAEs exhibit zero shrinkage bias while achieving Pareto-superior trade-offs between reconstruction error and $L_0$ sparsity.

2. Gated SAEs

An alternative approach developed by Rajamanoharan et al. (2024) is the Gated SAE, which decouples feature detection (determining which features are present) from feature estimation (determining their magnitude). Gated SAEs employ two parallel encoder pathways:

• A gating pathway with a Heaviside step or sharp activation to produce a binary mask $\pi(x) \in \{0, 1\}^F$.
• A magnitude pathway that estimates positive scalar intensity $r(x) \in \mathbb{R}^F_+$.

The latent representation is computed as the elementwise product $f(x) = \pi(x) \odot r(x)$. The $L_1$ sparsity penalty is applied strictly to the gating pre-activations, preserving unbiased gradient flow to the magnitude pathway.

3. Dead Latent Mitigation and Ghost Gradients

During SAE training, a subset of dictionary features can suffer from initialization traps where their pre-activations never exceed the threshold for any sample in the training batch. Once a latent is dead, its gradient remains zero permanently.

To prevent latent extinction, practitioners use three primary techniques:

• Geometric Re-initialization: Detecting latents that have not fired for $N$ consecutive steps and re-initializing their encoder and decoder vectors toward high-error residual vectors $x - \hat{x}$.
• Ghost Gradients: Rajamanoharan et al. (2024) introduced auxiliary ghost gradients that propagate a fractional reconstruction error through dead latents without altering the forward-pass activations, pulling dead directions back into the active data manifold.
• Learning Rate Warm-Up and Decay: Decaying the sparsity coefficient $\lambda$ early in training to allow the full dictionary to establish initial receptive fields.

5. Architectural Comparison: Standard vs. Modern SAEs

Modern SAE research has yielded distinct architectural trade-offs:

• Standard L1 SAE: Uses soft thresholding via $\lambda \|f(x)\|_1$. Exhibits high shrinkage bias proportional to $\lambda$. Baseline $1\times$ computational overhead. Offers simple implementation and stable convergence.
• TopK SAE: Uses hard selection via $\text{TopK}(z, k)$. Completely eliminates shrinkage bias. Low $1\times$ computational overhead. Delivers strict $L_0$ control and optimal reconstruction fidelity.
• Gated SAE: Uses a decoupled binary gating subnet. Completely eliminates shrinkage bias. Incurs medium $1.3\times$ computational overhead. Provides unbiased magnitude scaling and smooth optimization dynamics.
• JumpReLU SAE: Uses parametric thresholding $z \cdot \mathbb{I}(z > \theta)$. Exhibits minimal shrinkage bias. Incurs low $1.05\times$ computational overhead. Supports continuous threshold learning with straight-through gradient estimators.

6. Mechanistic Interpretability, Monosemanticity, and Feature Steering

Once an SAE is trained and its dictionary columns $d_j$ are extracted, several quantitative and qualitative evaluations confirm that polysemanticity has been resolved.

Automated Monosemanticity Scoring

To measure feature interpretability at scale, researchers evaluate automated interpretability using an external LLM judge:

1. Top-Activating Token Extraction: Collect the top 20 text snippets that produce the highest activation $f_j(x)$ for a specific latent $j$.
2. Hypothesis Generation: The evaluator LLM analyzes the snippets and generates a natural language description of what concept latent $j$ represents.
3. Falsification / Prediction Phase: The evaluator LLM is presented with a held-out mixture of high-activation and low-activation snippets and must predict the activation level based solely on its generated hypothesis.

Features discovered by SAEs consistently achieve high prediction accuracy across diverse domains, capturing highly specific concepts such as uppercase acronyms, legal liability clauses, code indentations, and emotional tone shifts.

[Residual Stream x] ---> [SAE Encoder] ---> Feature #4812: "Legal disclaimer clauses"
                                       ---> Feature #9210: "Python syntax error"
                                       ---> Feature #1402: "Golden Gate Bridge reference"

Model Steering via Feature Clamping

Beyond passive inspection, SAEs enable precise causal intervention. In standard models, modifying a single neuron vector induces collateral interference across all concepts sharing that neuron. In contrast, modifying an SAE latent acts on a single monosemantic direction.

To steer a model toward or away from a concept, the residual stream activation $x$ during the forward pass is modified using feature dictionary vector $d_j$:

$$\tilde{x} = x + \alpha \cdot d_j$$

Where $\alpha \in \mathbb{R}$ is a steering coefficient. Setting $\alpha > 0$ forces the model to express the concept (for example, causing the model to repeatedly reference a specific entity or adopt a specific persona), while setting $\alpha < 0$ suppresses the concept without degrading unrelated language modeling capabilities.

Circuit Discovery Across Layers

By replacing dense layer activations across multiple transformer blocks with SAE latents, researchers can trace discrete causal circuits. The interaction between feature $f_j^{(l)}$ at layer $l$ and feature $f_k^{(l+1)}$ at layer $l+1$ is quantified by calculating the gradient of the downstream feature with respect to the upstream feature:

$$C_{j \to k} = \frac{\partial f_k^{(l+1)}}{\partial f_j^{(l)}}$$

This maps neural network computation from an intractable black-box weight matrix into an interpretable directed acyclic computational graph composed of discrete semantic nodes.

Sources

• Toy Models of Superposition (Elhage et al., 2022) - Foundational theoretical framework establishing the geometry of superposition and feature packing in high dimensions.
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learning (Bricken et al., Anthropic, 2023) - First scaled application of overcomplete Sparse Autoencoders to language model residual streams.
• Scaling and Evaluating Sparse Autoencoders (Gao et al., OpenAI, 2024) - Mathematical analysis of TopK SAEs, scaling laws for dictionary learning, and elimination of shrinkage bias.
• Gated Sparse Autoencoders for Neural Network Interpretability (Rajamanoharan et al., Google DeepMind, 2024) - Formulation of Gated SAEs and ghost gradient mechanics for dead latent mitigation.
• Sparse Autoencoders Find Highly Interpretable Features in Language Models (Cunningham et al., ICLR 2024) - Independent demonstration of monosemantic dictionary recovery and feature steering in open-weight models.
• The Linear Representation Hypothesis and the Geometry of Concept Spaces (Park et al., 2023) - Mathematical verification of concept representation as linear subspaces in deep neural networks.