Latent Reasoning in Large Language Models: How Continuous Thoughts and Recurrent Hidden States Bypass Discrete Tokenization

Standard autoregressive language models solve multi-step reasoning tasks by generating explicit verbal scratchpads. Under the Chain-of-Thought (CoT) paradigm formalized by Wei et al. (2022), a Transformer expands its effective computational depth by emitting intermediate natural language tokens into the prompt context. Each emitted token provides an additional forward pass through the network's layers, transforming reasoning into a sequence of left-to-right text predictions.

While language-based reasoning has enabled significant performance gains across arithmetic, symbolic manipulation, and algorithmic benchmarks, forcing internal computational steps through a discrete vocabulary imposes strict theoretical and operational constraints. In response, recent research into latent reasoning investigates architectures where intermediate reasoning occurs directly within continuous hidden state trajectories, bypassing tokenization entirely.

The Discrete Token Bottleneck in Chain-of-Thought

In standard autoregressive Transformers, the transition between reasoning steps is constrained by an information-collapsing discrete bottleneck.

Standard Chain-of-Thought:
Input x ───► [Transformer Layers] ───► Hidden State h_t ───► LM Head (W_u) ───► Discrete Token y_t
                                                                                      │
               Input x_{t+1} ◄─── Embedding Layer (W_e) ◄──────────────────────────────┘

Continuous Latent Reasoning:
Input x ───► [Transformer Layers] ───► Hidden State h_t ───► Continuous Thought c_t
                                                                   │
               Input x_{t+1} ◄─── RMSNorm(h_t) ◄───────────────────┘

At step $t$, the top-layer hidden activation $h_t \in \mathbb{R}^d$ encapsulates a high-dimensional continuous representation of the model's current computational state. To produce the next token, this representation is projected through the unembedding matrix $W_u \in \mathbb{R}^{|V| \times d}$ to generate vocabulary logits, passed through a softmax operator to yield categorical probabilities $P(y_t \mid x_{<t})$, and sampled to produce a single discrete token index $y_t \in \{1, \dots, |V|\}$. The subsequent forward pass embeds $y_t$ back into continuous space via the input embedding matrix $W_e \in \mathbb{R}^{d \times |V|}$.

This discrete projection cycle introduces three fundamental bottlenecks:

1. Information Quantization and Rank Collapse: The vocabulary projection maps a dense vector in $\mathbb{R}^d$ (where $d$ typically spans 2,048 to 8,192 dimensions) onto a single discrete category. This forces the model to collapse internal superpositions of alternative hypotheses, discarding uncertainty and subtle relational bindings that cannot be mapped onto isolated dictionary words.
2. Premature Deterministic Commitment: Autoregressive decoding commits greedily to a specific token sequence. If a chosen reasoning branch encounters a logical dead end, the model cannot seamlessly backtrack or reweight alternative paths without executing external search wrappers such as Tree of Thoughts (Yao et al., 2023) or Monte Carlo Tree Search, which multiply inference latency.
3. KV Cache and Compute Inefficiency: Articulating step-by-step logic in natural language requires substantial lexical overhead: syntax tokens, connectives, and formatting boilerplate. These verbose tokens consume key-value (KV) cache slots, increase memory bandwidth pressure, and scale attention computation quadratically with sequence length.

Continuous Latent Reasoning: The Coconut Framework

To overcome the discrete token bottleneck, Hao et al. (Meta / COLM 2025) introduced Coconut (Chain of Continuous Thought). Coconut removes the unembedding and re-embedding projection loop during intermediate reasoning stages, allowing the model to reason directly in continuous latent space.

Continuous Thought Recurrence

In Coconut, the model alternates between two operating modes: Language Mode and Latent Mode.

During Language Mode, the model functions as a standard autoregressive language model, generating tokens for prompt ingestion and final answer emission. When entering Latent Mode, the unembedding head is bypassed. The final layer hidden state $h_t^{(L)} \in \mathbb{R}^d$ at sequence position $t$ is designated as a continuous thought vector $c_t$:

$$c_t = h_t^{(L)}$$

For the subsequent generation step $t+1$, the continuous thought $c_t$ is fed directly as the input embedding $x_{t+1}$, replacing the standard token embedding lookup:

$$x_{t+1} = \text{RMSNorm}(c_t)$$

The continuous thought vector is processed through the full stack of Transformer attention and feed-forward layers, generating the next continuous thought $c_{t+1} = h_{t+1}^{(L)}$. This recurrent feedback loop allows the model to perform continuous multi-step computation across arbitrary latent horizons without generating discrete text tokens.

Multi-Stage Curriculum Training

Training an LLM to reason continuously presents an optimization challenge: pre-trained models are specialized in discrete text prediction. Directly replacing language chains with continuous vectors from scratch leads to catastrophic optimization collapse.

To establish continuous reasoning, Hao et al. developed a multi-stage curriculum learning strategy that progressively internalizes discrete reasoning steps into latent thoughts:

• Stage 0 (Baseline CoT): The model is fine-tuned on standard question-answer pairs with full natural language reasoning chains $(q, r_1, r_2, \dots, r_k, a)$, where each $r_i$ denotes a reasoning step composed of discrete tokens.
• Stage $j$ (Progressive Replacement): For each reasoning step $i \le j$, the sequence of discrete tokens in $r_i$ is replaced by a fixed number of continuous thought vectors $\mathbf{c}_i = (c_{i,1}, \dots, c_{i,m})$. The subsequent reasoning steps $(r_{j+1}, \dots, r_k)$ and final answer $a$ remain in natural language. The model is trained with standard cross-entropy loss applied exclusively to the remaining discrete tokens:

$$\mathcal{L}_{\text{stage } j} = -\sum_{t \in \text{Tokens}(r_{>j}, a)} \log P(y_t \mid q, \mathbf{c}_1, \dots, \mathbf{c}_j, y_{<t})$$

• Final Stage: All intermediate natural language reasoning steps are fully replaced by continuous thoughts $(\mathbf{c}_1, \dots, \mathbf{c}_k)$. The model takes question $q$, computes continuous trajectories in latent space, and directly outputs final answer $a$.

Emergent Breadth-First Search and Path Exploration

A critical finding in continuous latent reasoning is the emergence of parallel search dynamics. In discrete Chain-of-Thought, a language model is forced to commit to a single discrete token at each step, defining a single path in the reasoning graph (depth-first progression).

Because continuous thought vectors $c_t \in \mathbb{R}^d$ exist in high-dimensional continuous space, a single vector can encode a superposition of multiple potential reasoning states simultaneously. Mathematical probes conducted by Hao et al. (2024) demonstrate that continuous thoughts effectively execute an implicit Breadth-First Search (BFS):

1. Multi-Hypothesis Encoding: At step $t=1$, the continuous thought activation assigns non-zero projection components along multiple valid candidate directions in latent space.
2. Implicit Value-Guided Pruning: Over subsequent latent steps $(t=2, 3)$, attention layers compute cross-positional alignments that dampen invalid branches while amplifying trajectories consistent with the target objective.
3. Backtracking Without Token Regeneration: In graph search tasks (such as finding paths in complex networks), Coconut outperforms standard Chain-of-Thought specifically on problems requiring extensive backtracking, while using significantly fewer thinking steps. The model avoids getting trapped in local greedy choices because alternative paths remain partially activated within the continuous vector.

Alternative Latent Reasoning Formulations

Continuous-space computation is explored across several distinct structural paradigms:

Latent Reasoning Taxonomy:

1. Recurrent Continuous Thoughts (Coconut)
   Prompt ──► [Continuous Thought 1] ──► [Continuous Thought 2] ──► Output Tokens

2. Token-Level Parallel Deliberation (Quiet-STaR)
   Token_t ──► [Parallel Thought Branches] ──► Thought-Weighted Prediction ──► Token_{t+1}

3. Step-by-Step Internalization (Implicit CoT)
   Horizontal layer-wise recurrence removes explicit intermediate tokens via distillation.

4. Layer-Recurrent Universal Models (Recurrent Depth)
   Fixed-parameter weights looped across variable iteration steps per token.

1. Quiet-STaR: Token-Level Deliberation

While Coconut replaces macro-level reasoning steps with latent thoughts, Quiet-STaR (Zelikman et al., 2024) introduces fine-grained token-level deliberation. Quiet-STaR enables a language model to generate internal rationales before predicting arbitrary future text.

At each token position, the model generates $n$ parallel thought traces of length $T_{\text{thought}}$ using special start-of-thought <|startthought|> and end-of-thought <|endthought|> markers. A mixing head computes a dynamic interpolation weight $\alpha \in [0, 1]$ between the base language model prediction and the rationale-augmented prediction:

$$P(y_{t} \mid x_{\le t}) = (1 - \alpha) P_{\text{base}}(y_t \mid x_{\le t}) + \alpha P_{\text{thought}}(y_t \mid x_{\le t}, \text{thought})$$

The thought generation policy is optimized using the REINFORCE algorithm with a baseline reward computed from how substantially the internal thoughts improve next-token cross-entropy on unstructured pre-training corpora.

2. Implicit Chain-of-Thought

Deng et al. (2024) demonstrated that language models can internalize multi-step explicit reasoning chains through progressive layer-wise distillation. By removing intermediate tokens one by one and training the model to predict subsequent tokens via horizontal hidden state transfer, the network learns to condense multi-step derivations into internal representations across intermediate Transformer layers.

3. Universal Transformers and Recurrent Depth

Prior to modern LLMs, Universal Transformers (Dehghani et al., 2018) and recurrent depth networks (Schwarzschild et al., 2021) demonstrated that looping input representations through shared Transformer layers expands computational depth without adding parameter memory. Modern continuous thought methods combine recurrent depth with causal sequence processing, separating reasoning compute from lexical output length.

Positional Encodings and KV Cache Mechanics

Integrating continuous thoughts into existing autoregressive architectures requires careful management of sequence metadata:

Rotary Position Embedding (RoPE) Indexing

Modern decoder-only LLMs rely on Rotary Position Embeddings (Su et al., 2024). When a continuous thought $c_t$ is inserted into the sequence, it must be assigned a position index $p_t$:

• Sequential Indexing: Assigning $p_{t+1} = p_t + 1$ treats continuous thoughts identically to discrete tokens in the causal attention graph. The model attends to prior discrete prompt tokens and previous continuous thoughts with standard positional decay.
• Index Freezing: Certain architectures hold the position index constant across a continuous thought burst ($p_{\text{latent}} = p_{\text{prompt}} + 1$), forcing intra-thought attention to operate as permutation-invariant iterative refinement before resuming sequential indexing for final token generation.

Serving Economics and VRAM Footprints

Continuous thoughts offer significant inference efficiency advantages:

• KV Cache Slot Compression: In multi-step mathematical reasoning, verbalizing a single arithmetic step in natural language typically consumes 20 to 50 tokens (e.g., "Subtracting 14 from both sides gives 3x = 42, then dividing by 3 yields x = 14."). In Coconut, that same derivation is represented by 1 to 2 continuous thought vectors. This achieves a 10x to 25x reduction in KV cache allocation for intermediate computation.
• Memory Bandwidth Reduction: In autoregressive generation, memory bandwidth is the primary bottleneck during decoding. By reducing total sequence length, continuous thoughts reduce DRAM-to-SRAM KV cache transfers, increasing decoding throughput on memory-bound workloads.

Challenges and Failure Modes

Despite strong theoretical advantages, continuous latent reasoning introduces distinct technical trade-offs:

1. Representation Drift and Activation Norm Explosion: Without the regularizing constraint of language vocabulary projection, recurrent continuous thoughts can drift away from the manifold of natural language representations. Unbounded feedback loops risk numerical instability or activation saturation, requiring strict pre-layer normalization (RMSNorm or QK-Norm) to maintain stability.
2. Loss of Interpretability: Natural language Chain-of-Thought produces human-readable, auditable reasoning traces. Continuous latent thoughts are high-dimensional vector trajectories. Detecting hallucinations, auditing safety boundaries, or debugging logical flaws requires auxiliary probing classifiers or projection decoders.
3. Rigid Capacity per Step: A single continuous vector $c_t \in \mathbb{R}^d$ has a fixed representational capacity bounded by model dimension $d$. For highly dense symbolic operations, a single continuous thought may lack the capacity to execute complex transformations, necessitating calibrated multi-vector thought allocation.

Production Outlook

Latent reasoning represents an architectural bridge between rigid discrete token generation and unconstrained continuous computation. As frontier models increasingly scale inference-time compute, hybrid architectures that combine continuous internal state exploration with selective discrete text generation offer a path toward higher computational efficiency, broader planning capabilities, and reduced memory overhead.

Sources

• Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903.
• Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., & Tian, Y. (2024). Training Large Language Models to Reason in a Continuous Latent Space. arXiv:2412.06769.
• Zelikman, E., Harik, G., Shao, Y., Jayasiri, V., Haber, N., & Goodman, N. D. (2024). Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking. arXiv:2403.09629.
• Deng, Y., Choi, Y., & Shieber, S. (2024). From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step. arXiv:2405.14838.
• Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601.
• Dehghani, M., et al. (2018). Universal Transformers. arXiv:1807.03819.
• Su, J., et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.