In multi-agent systems and tiered model cascades, routing requests across different model sizes is a standard architecture for balancing inference cost and generation quality. However, switching models mid-session has historically imposed a severe compute tax: because each neural architecture maintains its own distinct internal representations, the receiving model cannot read the key-value (KV) states generated by the previous model. The target model must recompute the entire conversation history from raw text during an initial prefill phase, creating latency spikes and redundant GPU spend on sequences exceeding tens of thousands of tokens.
To address this serving inefficiency, researchers at NVIDIA have introduced cross-model KV cache transfer, detailed in a research preprint titled "Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse". The technique demonstrates that key-value representations across different parameter scales within the same model family share strong linear structures. By applying a closed-form linear ridge regression mapping, serving engines can transform the existing KV cache of a source model directly into the format expected by a target model, skipping the prefill phase entirely.
The Prefill Bottleneck in Tiered LLM Architectures
In production serving, multi-tier routing architectures dynamically select models based on query complexity. A lightweight small language model (such as a 3B to 8B parameter model) often handles routine conversational turns, initial document filtering, or structured triage. When a query demands complex multi-step reasoning or mathematical verification, the orchestrator routes the session to a larger frontier model (such as a 32B or 70B parameter model). Once the reasoning step completes, execution can drop back to a smaller model to stream final answers.
While this pattern cuts decode costs, the transition between models invalidates the KV cache. While the decode phase generates tokens incrementally by attending to cached key-value states, the prefill phase scales quadratically or linearly with prompt length depending on attention implementations. At 32,768 tokens, computing the prompt prefill on a 32-billion parameter model can take several seconds of dedicated GPU time.
Previous attempts to share memory states across heterogeneous models required training deep neural adapters with backpropagation, enforcing identical layer counts, or restricting models to identical internal hidden dimensions.
Architecture of the Closed-Form Ridge Mapper
The NVIDIA research team focused on matched-KV model configurations, where source and target models share the same number of key-value heads and per-head dimensions, even though their total parameter counts, hidden layer counts, and MLP dimensions differ. This structure is common across model families such as Qwen, Llama, and Mistral.
The cross-model cache transfer pipeline relies on three core design components:
- Content-Space Representation Mapping: Modern transformers apply Rotary Position Embedding (RoPE) to key and query vectors to inject relative positional information. Because RoPE rotates vectors based on absolute token index, mapping raw rotated states creates position-dependent artifacts that do not generalize across arbitrary prompt lengths. The mapper strips the RoPE rotations before transformation, performs the mapping in invariant content space, and reapplies the target model's RoPE transformations afterward.
- Cross-Layer Source Selection: Because source and target models differ in layer depth (for example, transferring from a 48-layer source to a 64-layer target), the framework evaluates cross-layer correlation. Rather than a rigid one-to-one mapping, the algorithm selects the source layer combinations that maximize variance recovery for each target layer.
- Per-Head Ridge Regression: The actual feature projection is computed independently for each attention head using closed-form ridge regression. The projection matrices are solved analytically on a small calibration set of 500 sequences (1,024 tokens each) drawn from the FineWeb-Edu corpus, avoiding iterative gradient descent or complex parameter fine-tuning.
In experiments transferring states between Qwen3 14B and Qwen3 32B, a single linear regression layer recovered 56% of the variance in the target model's key states and 32% of the variance in its value states. When aggregating features from multiple source layers, variance recovery increased to 79% for keys and 65% for values.
Benchmark Results and Serving Speedups
The researchers evaluated the transfer pipeline across six model pairs across three open-weight architectures: Qwen3, Llama 3.1, and Ministral 3, spanning model sizes from 3B to 70B parameters. Evaluations tested downstream accuracy across ARC-Challenge, HellaSwag, WinoGrande, MMLU, and GSM8K, alongside WikiText-2 perplexity and CoQA multi-turn dialog stability.
Key experimental findings include:
- Inference Speedups: On an 8x NVIDIA H100 node connected via NVLink, transferring a 32,768-token KV cache from Qwen3 14B to Qwen3 32B took 277.6 milliseconds. In contrast, running a standard full prefill on the target 32B model took 6,975.3 milliseconds (nearly 7 seconds). The linear mapper delivered speedups between 2.7x and 25x across varying context lengths.
- Accuracy Retention: On four of the six tested model pairs, the closed-form linear mapper retained between 73% and 98% of the target model's standalone baseline accuracy. Even across an 8.8x parameter leap from Llama 3.1 8B to Llama 3.1 70B, the transferred cache retained 72.8% of standalone target accuracy.
- Multi-Turn Stability: During sequential multi-turn evaluations spanning 10 conversational turns on CoQA, representation drift remained bounded, preventing compound degradation during continuous agent loops.
- Non-Linear Recovery: On pairs where linear regression encountered representation misalignment (such as specific Ministral configurations), replacing the closed-form linear mapper with a lightweight two-layer multi-layer perceptron (MLP) with 1,024 hidden units restored benchmark retention to above 90%, including a 37 percentage point recovery on HellaSwag.
Implications for Inference Engines and Agent Runtimes
Cross-model KV cache transfer demonstrates that internal attention representations within model families retain geometric coherence across parameter scales. By replacing compute-intensive prefill passes with analytical matrix multiplications in content space, inference engines like vLLM, SGLang, and TensorRT-LLM gain a path toward zero-overhead model switching.
As agentic frameworks execute long-running loops across tool calls and verification routines, eliminating the prefill latency penalty enables dynamic model tiering without throughput degradation.
