Cross-Model KV Cache Transfer in Production: Architecture, Closed-Form Ridge Projections, and Cascaded Serving Economics

Modern enterprise LLM serving architectures frequently rely on multi-model pipelines to balance inference cost, generation latency, and output quality. In model routing cascades, lightweight 8B models triage incoming queries and escalate complex reasoning tasks to 70B or MoE models. In speculative decoding pipelines, smaller draft models propose token sequences verified by larger target models. In long-horizon AI agent swarms, sub-agents frequently switch between specialized models across multi-turn workflows.

However, moving active conversation sessions between different models has historically imposed a severe latency and compute penalty: the re-prefill bottleneck. Because each language model maintains distinct parameter dimensions and latent representations, the receiving target model cannot ingest the source model's Key-Value (KV) cache. Instead, the target model must re-process the entire accumulated prompt history from scratch. In a 32,768-token context window, re-prefilling a 70B model adds 1,500ms to 3,000ms of Time-to-First-Token (TTFT) latency, burning high-cost GPU compute on redundant prompt evaluation.

Recent breakthroughs in representation geometry demonstrate that cross-model KV cache transfer is possible without re-running prefill. By exploiting linear correlations across models within the same architectural family, serving systems can project KV tensors from a small source model directly into a large target model using a closed-form, training-free ridge regression mapping.

The Re-Prefill Penalty in Cascaded Architectures

The operational cost of multi-model pipelines scales quadratically with sequence length during the prefill phase. For an input sequence of length $N$, computing the self-attention Key and Value states requires $O(N^2)$ FLOPs across all transformer layers:

$$Q = X W_Q, \quad K = X W_K, \quad V = X W_V$$

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$

When an inference gateway evaluates an incoming query with an 8B model and decides to escalate the request to a 70B model, the 8B model's KV cache is discarded. The 70B model allocates fresh GPU memory and executes a complete forward pass over all $N$ tokens before generating its first token.

Latency Overhead by Context Length on NVIDIA H100 GPUs

• 4,096 Tokens: Standalone 70B prefill takes 185 ms versus 28 ms on an 8B model, incurring 157 ms of redundant latency.
• 16,384 Tokens: Standalone 70B prefill takes 720 ms versus 98 ms on an 8B model, incurring 622 ms of redundant latency.
• 32,768 Tokens: Standalone 70B prefill takes 1,850 ms versus 245 ms on an 8B model, incurring 1,605 ms of redundant latency.
• 65,536 Tokens: Standalone 70B prefill takes 4,200 ms versus 560 ms on an 8B model, incurring 3,640 ms of redundant latency.

In agentic loops where context accumulates over dozens of tool calls and conversational turns, this re-prefill delay occurs on every escalation, destroying interactive responsiveness and multiplying infrastructure spend.

The Geometry of Cross-Model KV Representations

The feasibility of direct KV transfer relies on a key geometric property of foundation models: within a shared architectural lineage (such as the Llama, Qwen, or Mistral model families), intermediate Key and Value representations occupy closely aligned affine subspaces.

While models of different parameter counts differ in total depth, hidden dimension $d_{\text{model}}$, and intermediate feed-forward width, many families utilize matched Key-Value configurations. In a matched-KV pair:

1. The source and target models share the same number of KV heads $H_{kv}$ (governed by Grouped-Query Attention).
2. The source and target models share the identical per-head dimension $d_k$ (typically 128 dimensions).

Token-level Ordinary Least Squares (OLS) regression across matched-KV pairs reveals that the transformation between source Key vectors $K_{\text{src}} \in \mathbb{R}^{N \times d_k}$ and target Key vectors $K_{\text{tgt}} \in \mathbb{R}^{N \times d_k}$ exhibits high linear determination ($R^2 > 0.85$ across mid-to-deep layers). The models organize semantic attention patterns into similar geometric manifolds, enabling direct vector translation.

Source Model (8B/14B)                Target Model (32B/70B)
[ Layer L_s Prefill ]                [ Layer L_t Decode ]
        │                                    ▲
        ▼                                    │
 [ Raw K, V Cache ]                   [ Projected K, V ]
        │                                    │
        ▼                                    │
 [ Invert RoPE (R_-m) ]              [ Reapply RoPE (R_m) ]
        │                                    ▲
        ▼                                    │
 [ Content Vectors ] ──► [ Ridge W ] ────────┘
                      (d_k x d_k Matrix)

The Closed-Form Ridge Mapping Pipeline

Rather than training deep neural network adapters or training specialized distillation fusers with gradient descent, cross-model KV transfer can be executed using an analytical, closed-form Ridge Regression fit solved on a lightweight calibration dataset.

The translation pipeline operates across four discrete stages:

1. Positional Frequency Inversion (RoPE Stripping)

Modern autoregressive LLMs encode relative token distances using Rotary Position Embeddings (RoPE). RoPE applies an orthogonal rotation matrix $\mathbf{R}_m$ to Key vectors at sequence position $m$:

$$\tilde{K}_m = \mathbf{R}_m K_m$$

Because $\mathbf{R}_m$ introduces position-dependent sinusoidal modulation, attempting to fit a linear mapping directly on $\tilde{K}$ causes spatial interference across different token positions. The transfer pipeline isolates pure semantic content by multiplying the cached Key tensors by the inverse rotation matrix:

$$K_m^{\text{content}} = \mathbf{R}_{-m} \tilde{K}_m$$

Value vectors $V$ do not undergo RoPE rotation and pass directly to the projection stage without inversion.

2. Per-Head Ridge Regression Projection

For each attention head $h \in \{1, \dots, H_{kv}\}$, the serving engine maintains a precomputed projection matrix $W_K^{(h)} \in \mathbb{R}^{d_k \times d_k}$ and $W_V^{(h)} \in \mathbb{R}^{d_k \times d_k}$.

The projection matrix is calculated offline using standard L2-regularized ridge regression across a calibration dataset $X \in \mathbb{R}^{M \times d_k}$ (source activations) and $Y \in \mathbb{R}^{M \times d_k}$ (target activations):

$$W = (X^T X + \lambda I)^{-1} X^T Y$$

Because $d_k$ is small ($128 \times 128$), computing $(X^T X + \lambda I)^{-1}$ involves inverting a $128 \times 128$ matrix. This operation takes less than 10 milliseconds on a single CPU core and requires no GPU backpropagation or hyperparameter tuning. A calibration corpus of only 256 to 512 tokens from general pre-training text is sufficient to achieve optimal matrix conditioning.

3. Layer Selection and Alignment Mapping

Because target models typically have more layers than source models (for example, Llama 3.1 8B has 32 layers while Llama 3.1 70B has 80 layers), the system maps source layers $L_s$ to target layers $L_t$.

Research by NVIDIA (arXiv:2608.03893) demonstrates that uniform depth interpolation or greedy cosine-similarity layer pairing preserves the majority of representational fidelity. When mapping a 32-layer source to an 80-layer target, target layers are assigned the projected representation of the nearest proportional source layer:

$$L_s = \text{round}\left(L_t \times \frac{\text{Layers}_{\text{src}}}{\text{Layers}_{\text{tgt}}}\right)$$

4. Target Positional Reapplication

Once the unrotated Key vectors are projected through $W_K$, the target model's rotary frequencies $\mathbf{R}_m^{\text{tgt}}$ are applied to the resulting tensor:

$$\tilde{K}_{m, \text{tgt}} = \mathbf{R}_m^{\text{tgt}} \left( K_m^{\text{content}} W_K \right)$$

The projected Key and Value tensors are inserted directly into the target model's PagedAttention KV cache slots, allowing the target model to begin decoding immediately at position $N+1$.

Production Architecture and Serving Topologies

Integrating cross-model KV transfer into production inference engines modifies the cluster request lifecycle:

[ Inbound Query ] ──► [ AI Gateway / Router ]
                             │
                             ▼
               [ Fast Prefill Node (8B / L40S) ]
                             │ (Computes K,V at low cost)
                             ▼
              [ Ridge Projection Kernel (<5ms) ]
                             │
            (Projected KV Tensors via RoCEv2 / RDMA)
                             │
                             ▼
              [ Target Decode Node (70B / H100) ]
                             │
                 (Immediate First Token Output)

1. Disaggregated Heterogeneous Prefill

In traditional Disaggregated Prefill and Decode (PD separation), prefill nodes and decode nodes run the identical model architecture, requiring identical high-memory GPU hardware across both tiers.

Cross-model KV transfer enables heterogeneous PD separation:

• Prefill Pool: Clusters of power-efficient, commodity GPUs (such as NVIDIA L40S or A100-40GB) run compact 8B or 14B models to ingest massive context windows at low cost.
• Projection Layer: A fused CUDA kernel executes the $128 \times 128$ matrix multiplication on the generated KV cache in under 3ms.
• Decode Pool: High-throughput NVIDIA H100/H200 clusters receive the projected KV tensors over high-bandwidth InfiniBand or RoCEv2 interconnects, dedicating 100% of their compute to autoregressive token generation.

2. Speculative Escalation in Multi-Turn Agents

When an autonomous agent initiates a multi-step task, the gateway assigns the task to a lightweight model. The lightweight model generates thoughts and executes initial tool calls.

If the agent encounters an exception, complex logic puzzle, or high-ambiguity output:

1. The orchestrator halts execution on the small model.
2. The accumulated KV cache (containing system prompts, tool schemas, and conversation history) is projected via the ridge mapper.
3. The frontier target model resumes execution instantly without waiting for a 2,000ms re-prefill phase.

Empirical Performance and Accuracy Retention

Evaluations across frontier open-weight model families demonstrate substantial reductions in latency with minimal degradation in downstream task accuracy.

Accuracy Retention Across Benchmark Tasks

According to empirical findings across matched-KV architectures (NVIDIA, 2026), closed-form linear ridge mapping retains between 73% and 98% of standalone target model performance:

• Qwen3 14B to 32B (MMLU): Standalone target accuracy 74.2%, projected transfer accuracy 72.8%, achieving 98.1% retention.
• Qwen3 14B to 32B (GSM8K): Standalone target accuracy 85.6%, projected transfer accuracy 83.5%, achieving 97.5% retention.
• Qwen3 14B to 32B (HellaSwag): Standalone target accuracy 86.1%, projected transfer accuracy 84.1%, achieving 97.6% retention.
• Llama 3.1 8B to 70B (MMLU): Standalone target accuracy 79.4%, projected transfer accuracy 72.3%, achieving 91.1% retention.
• Llama 3.1 8B to 70B (GSM8K): Standalone target accuracy 84.2%, projected transfer accuracy 74.8%, achieving 88.8% retention.
• Llama 3.1 8B to 70B (ARC-Challenge): Standalone target accuracy 88.5%, projected transfer accuracy 81.2%, achieving 91.8% retention.

On model pairs where pure linear ridge mapping experiences higher residual error (such as extreme parameter disparities), inserting a compact two-layer Multi-Layer Perceptron (MLP) adapter recovers downstream accuracy to above 90% across all evaluated benchmarks.

Latency and Throughput Speedups

By replacing dense $O(N^2)$ transformer attention compute with an $O(N)$ linear matrix projection, TTFT latency drops significantly across sequence lengths:

• 4K Tokens: Standalone 70B TTFT is 185 ms, while 8B prefill with ridge projection takes 36 ms (5.1x effective speedup).
• 16K Tokens: Standalone 70B TTFT is 720 ms, while 8B prefill with ridge projection takes 112 ms (6.4x effective speedup).
• 32K Tokens: Standalone 70B TTFT is 1,850 ms, while 8B prefill with ridge projection takes 262 ms (7.1x effective speedup).
• 64K Tokens: Standalone 70B TTFT is 4,200 ms, while 8B prefill with ridge projection takes 595 ms (7.1x effective speedup).

When evaluating target GPU utilization, skipping the target prefill entirely yields up to a 25x speedup for the target model instance, liberating target GPU tensor cores to process active decode batches.

Implementation Guidelines and Operational Guardrails

To deploy cross-model KV transfer safely in production environments, infrastructure teams should observe three core engineering constraints:

1. Calibration Data Distribution

Because the ridge regression matrix $W = (X^T X + \lambda I)^{-1} X^T Y$ relies on an unconstrained least-squares fit, the calibration dataset must contain diverse token distributions.

• Corpus Composition: Use 300 to 500 lines of multi-lingual text, code snippets, mathematical reasoning, and markdown formatting.
• Regularization Parameter: Set the ridge penalty $\lambda$ between $10^{-4}$ and $10^{-2}$ to prevent ill-conditioned matrix inversions without over-smoothing distinct head projections.

2. Numerical Precision and Accumulation

KV projections must maintain FP16 or BF16 precision. While weight matrices can be stored in quantized formats (such as FP8), performing the matrix multiplication $K_{\text{content}} W_K$ in lower precision (such as INT4 or FP4) introduces accumulation errors that degrade downstream attention logits across long sequences.

3. Network Bandwidth Budgeting

Transferring KV caches between physical servers introduces network I/O. For a 32,768-token sequence in FP16 precision across 8 KV heads with dimension 128:

$$\text{KV Cache Size} = 2 \times 32,768 \times 8 \times 128 \times 2 \text{ bytes} \approx 134.2 \text{ MB}$$

On a standard 100 Gbps RoCEv2 datacenter fabric, transmitting 134 MB takes approximately 10.7 ms, easily fitting within the 200ms+ compute savings realized by avoiding target prefill. On standard 10 Gbps public cloud networks, network transfer latency will bottleneck performance; deployments should restrict cross-node KV streaming to environments with minimum 50 Gbps cluster interconnects.

Sources

• Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse - arXiv
• Qwen Technical Report - arXiv
• The Llama 3 Herd of Models - arXiv
• Disaggregated Prefill and Decode in Production - arXiv