Hierarchical KV Cache Offloading in Production LLM Serving: Host RAM, Local NVMe, Remote Storage, and LMCache Architecture

High-concurrency large language model (LLM) serving faces an acute memory capacity bottleneck. While modern GPUs deliver high floating-point compute throughput, High-Bandwidth Memory (HBM) capacity remains severely constrained. In workloads involving multi-turn conversations, agentic coding loops, and long-document retrieval-augmented generation (RAG), Key-Value (KV) cache tensors rapidly consume available GPU memory, forcing engines to either reject requests, preempt running sequences, or recompute extensive prompt prefixes.

Hierarchical KV cache offloading addresses this limit by treating GPU HBM as the top tier of a multi-layer storage hierarchy. By tiering KV caches across host CPU system memory, local NVMe solid-state storage, and distributed object stores, inference engines can retain large token contexts across requests at a fraction of the cost of raw GPU memory.

1. The GPU Memory Cliff in Production Serving

During the autoregressive decoding phase of transformer execution, attention mechanisms require continuous access to past key and value vectors. In models utilizing Grouped-Query Attention (GQA), such as Llama 3 70B (80 layers, 8 KV heads, head dimension of 128, FP16 precision), the KV cache memory footprint is calculated as:

$$\text{Memory per Token} = 2 \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times b_{\text{bytes}}$$

$$\text{Memory per Token} = 2 \times 80 \times 8 \times 128 \times 2 = 327,680 \text{ bytes} \approx 327.68 \text{ KB/token}$$

For an agentic workflow or enterprise document analysis processing a 128,000-token context, storing the uncompressed KV cache for a single request requires approximately 41.94 GB of memory. On an 80 GB NVIDIA H100 SXM5 GPU, after allocating memory for model weights across a tensor-parallel cluster, a single node can only support a handful of concurrent active sequences before running out of HBM.

When GPU memory is exhausted, standard serving engines face an operational dilemma:

• Prefix Recomputation: Evicting the cache and recomputing key-value tensors during subsequent turns wastes GPU compute FLOPs and introduces high Time to First Token (TTFT) latency.
• Request Preemption: Swapping active decodes to idle states degrades throughput and tail latency under heavy batching conditions.
• Capacity Limits: Rejecting incoming requests caps system scalability.

2. The Multi-Tier Memory Hierarchy

Hierarchical offloading organizes memory into four distinct tiers based on latency, bandwidth, capacity, and cost:

1. Tier 1: GPU High-Bandwidth Memory (HBM3/HBM3e)

• Bandwidth: 3.35 TB/s (H100) to 4.8 TB/s (H200).
• Latency: < 100 nanoseconds.
• Capacity: 80 GB to 141 GB per GPU.
• Role: Active decoding and immediate attention computation.

1. Tier 2: Host CPU DRAM (System Memory)

• Bandwidth: ~64 GB/s bidirectional over PCIe Gen5 x16; up to 128 GB/s in dual-socket configurations; extended further via CXL 2.0/3.0 interfaces.
• Latency: 1 to 5 microseconds.
• Capacity: 512 GB to 2 TB per server node.
• Role: High-speed intermediate cache for recently active multi-turn sessions and shared system prompts.

1. Tier 3: Local NVMe Solid-State Storage

• Bandwidth: 7 to 14 GB/s sequential read throughput per PCIe Gen4/Gen5 enterprise U.2/E1.S drive; aggregated to 30-50 GB/s over RAID arrays.
• Latency: 20 to 100 microseconds.
• Capacity: 4 TB to 30 TB per server node.
• Role: Warm secondary storage for long-context document databases and persistent agent session states.

1. Tier 4: Distributed Remote Storage & Network Memory Pools

• Bandwidth: 12.5 to 50 GB/s over 100 Gbps to 400 Gbps RDMA (RoCEv2 or InfiniBand); 1 to 5 GB/s over standard TCP/IP object storage (S3, MinIO, Ceph).
• Latency: 100 microseconds (RDMA) to 20 milliseconds (S3).
• Capacity: Terabytes to petabytes across the cluster.
• Role: Cross-node cache sharing, global prefix repositories, and cold session persistence across horizontally scaled inference replicas.

3. Systems Architecture: FlexGen, vLLM Swap, LMCache, and Mooncake

Several open-source and research systems implement different aspects of this hierarchy:

FlexGen: High-Throughput Batch Offloading

FlexGen pioneered high-throughput generation on memory-constrained hardware using a linear programming formulation. Rather than focusing on single-request interactive latency, FlexGen optimizes batch throughput by coordinating the movement of weights, activations, and KV caches across GPU, CPU, and NVMe disk tiers. Its zigzag execution schedule processes multiple batches across transformer layers in block-level waves, amortizing weight transfer overhead over large sequence counts.

vLLM Native Swapping and OffloadingConnector

vLLM manages GPU memory using PagedAttention, which allocates KV tensors in fixed-size blocks (typically 16 or 32 tokens) to eliminate internal and external memory fragmentation.

• Native Block Swapper: In standard vLLM, when GPU memory fills up, the engine evicts physical blocks of preempted sequences to a pre-allocated pinned CPU memory buffer using non-blocking CUDA memory copies (cudaMemcpyAsync). When scheduled for execution, blocks are swapped back into GPU HBM.
• OffloadingConnector (v0.11.0+): Modern vLLM versions introduce standardized connector interfaces to offload and reload prompt prefix blocks, allowing decoupled CPU-GPU memory management across independent requests.

LMCache: Decoupled Knowledge Delivery Network

LMCache abstracts KV cache management away from the monolithic serving engine into a specialized Knowledge Delivery Network (KDN). LMCache operates as an external layer that extracts, serializes, indexes, and moves KV caches across heterogeneous backends including CPU DRAM, local NVMe, Redis, and S3-compatible object storage.

• Decoupled Architecture: LMCache separates the storage lifecycle from the serving runtime, allowing multiple vLLM or SGLang worker processes to read and write from a shared distributed cache pool.
• Chunk-Level Indexing: KV tensors are segmented into fixed-token chunks (e.g., 256 tokens) and indexed by prefix hash, enabling sub-request cache hits when queries share common document fragments or system prompts.

Mooncake: KVCache-Centric Distributed Storage

Mooncake implements a distributed memory pool by aggregating underutilized DRAM and NVMe storage across all nodes in an inference cluster. Using an asynchronous RDMA transport engine (NIXL), Mooncake allows prefill-heavy worker nodes to stream computed KV caches directly to decode-centric worker nodes or centralized storage nodes without passing through intermediate CPU host bottlenecks.

4. Key Systems Engineering Optimizations

Moving tens of gigabytes of high-dimensional tensors across physical buses requires specialized systems-level optimizations to prevent transfer latency from dominating execution time.

Asynchronous Layer-Wise Transfer Pipelining

Blocking GPU computation while loading KV caches over PCIe introduces substantial latency penalties. Modern offloading frameworks implement layer-wise double buffering:

• While transformer layer $l$ executes self-attention on GPU HBM, the direct memory access (DMA) engine asynchronously transfers the KV cache tensors for layer $l+1$ from host CPU DRAM over PCIe.
• By matching the compute duration of matrix-multiplication kernels with PCIe transmission windows, transfer latency is hidden behind token computation.

Chunk-Based Prefix Hashing

Instead of indexing full sequence strings, systems divide the prompt token stream into uniform chunks of size $C$ (e.g., $C = 256$ tokens):

• Each chunk computes a cryptographic hash (such as SHA-256) over the concatenation of the preceding chunk hash and its local token IDs:

$$H_k = \text{Hash}(H_{k-1} \parallel T_{k, 1}, T_{k, 2}, \dots, T_{k, C})$$

• When an incoming query shares a prefix with existing cache entries, the engine matches chunk hashes against its local Radix tree and remote indices, fetching only the required missing segments while reusing previously offloaded blocks.

Tensor Compression and Quantization

The effective transfer throughput across PCIe and network interconnects can be multiplied by compressing KV tensors prior to offloading:

• FP8 / INT4 Quantization: Quantizing KV tensors from FP16 to FP8 (E4M3 or E5M2) or INT4 reduces tensor size by 50% to 75%, cutting PCIe transit duration proportionally.
• CacheGen: Frameworks like CacheGen employ customized context-aware compression and streaming encoders, reducing KV cache bandwidth requirements by up to 3.5x to 4.3x with negligible impact on generation accuracy.

GPUDirect Storage (GDS) and cuFile Integration

When offloading to NVMe storage tiers, standard POSIX read/write system calls route data through host CPU page caches and kernel memory buffers, causing CPU core saturation and memory bandwidth contention.

• By leveraging NVIDIA GPUDirect Storage (GDS) via cuFile APIs, inference engines establish direct DMA pathways between NVMe PCIe controllers and GPU HBM.
• Benchmarks show GDS reduces offload latency by 40% to 60% compared to POSIX-based buffered I/O while freeing host CPU cycles for concurrent orchestration tasks.

5. Break-Even Economics: Prefill Recomputation vs. Offload Reloading

The fundamental systems engineering decision is determining when loading a KV cache from a secondary tier is faster and more cost-effective than recomputing the prefill forward pass on the GPU.

Mathematical Formulation

Let:

• $P$ be the active parameter count of the model (e.g., 70 billion parameters).
• $L$ be the sequence prefix length in tokens (e.g., 32,768 tokens).
• $\text{TFLOPS}_{\text{eff}}$ be the effective prefill matrix-multiplication throughput of the GPU cluster (e.g., 400 TFLOPS in FP16 on H100).
• $\text{Size}_{\text{KV}}(L)$ be the KV cache size in bytes for sequence length $L$.
• $\text{BW}_{\text{tier}}$ be the effective bandwidth of the storage tier in GB/s.

The prefill recomputation time $T_{\text{prefill}}$ is governed by forward-pass FLOP requirements: $$T_{\text{prefill}} \approx \frac{2 \times P \times L}{\text{TFLOPS}_{\text{eff}}}$$

The reload transfer time $T_{\text{reload}}$ is governed by interconnect bandwidth: $$T_{\text{reload}} = \frac{\text{Size}_{\text{KV}}(L)}{\text{BW}_{\text{tier}}} = \frac{2 \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times b_{\text{bytes}} \times L}{\text{BW}_{\text{tier}}}$$

Break-Even Condition

Loading from the secondary tier is faster than recomputing ($T_{\text{reload}} < T_{\text{prefill}}$) when: $$\frac{2 \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times b_{\text{bytes}} \times L}{\text{BW}_{\text{tier}}} < \frac{2 \times P \times L}{\text{TFLOPS}_{\text{eff}}}$$

Canceling $L$ reveals that the break-even condition is independent of sequence length for linear attention implementations: $$\text{BW}_{\text{tier}} > \text{TFLOPS}_{\text{eff}} \times \left( \frac{N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times b_{\text{bytes}}}{P} \right)$$

Numerical Analysis on Llama 3 70B

• Model Parameters: $P = 70 \times 10^9$.
• KV Cache per Token: $327.68 \text{ KB/token} = 3.2768 \times 10^5 \text{ bytes/token}$.
• Prefill Compute Intensity: $2 \times 70 \times 10^9 = 1.4 \times 10^{11} \text{ FLOPs/token}$.
• At an effective prefill throughput of $400 \text{ TFLOPS}$ ($4 \times 10^{14} \text{ FLOPs/s}$), computing 32,768 tokens takes:

$$T_{\text{prefill}} = \frac{1.4 \times 10^{11} \times 32,768}{4 \times 10^{14}} \approx 11.47 \text{ seconds}$$

• The KV cache size for 32,768 tokens is:

$$\text{Size}_{\text{KV}} = 32,768 \times 327.68 \text{ KB} \approx 10.74 \text{ GB}$$

• Loading from Host DRAM via PCIe Gen5 (64 GB/s):

$$T_{\text{reload, CPU}} = \frac{10.74 \text{ GB}}{64 \text{ GB/s}} \approx 0.168 \text{ seconds}$$ Speedup over recomputation: ~68x faster TTFT.

• Loading from Local NVMe Array via GPUDirect Storage (14 GB/s):

$$T_{\text{reload, NVMe}} = \frac{10.74 \text{ GB}}{14 \text{ GB/s}} \approx 0.767 \text{ seconds}$$ Speedup over recomputation: ~15x faster TTFT.

• Loading from Distributed Object Storage / S3 (2 GB/s):

$$T_{\text{reload, S3}} = \frac{10.74 \text{ GB}}{2 \text{ GB/s}} \approx 5.37 \text{ seconds}$$ Speedup over recomputation: ~2.1x faster TTFT.

Even over moderate network connections, loading precomputed KV tensors yields significant latency gains over re-executing 70B parameter prefill passes, while freeing GPU compute engines to serve decoding batches.

6. Production Deployment Guidelines

To deploy hierarchical KV offloading reliably in high-throughput environments:

1. Size Host DRAM Pools Proportionally to Concurrency:

Provision at least 4x to 8x the aggregate GPU HBM capacity in host DDR5 memory per node. For an 8x H100 (640 GB HBM) node, configuring 2 TB of host DRAM provides sufficient headroom to retain thousands of concurrent multi-turn dialogue histories without NVMe thrashing.

1. Configure Multi-Tier Eviction Watermarks:

Implement high and low watermarks for memory eviction:

• GPU HBM watermarks: Trigger asynchronous eviction to host RAM at 85% utilization; drop low-priority scratch tensors at 95%.
• Host DRAM watermarks: Migrate least-recently-used (LRU) chunk blocks to local NVMe when DRAM reaches 80% capacity.
• NVMe watermarks: Flush cold long-context indices to S3-compatible object storage after 24 hours of inactivity.

1. Deploy High-Speed Network Interfaces for Distributed Serving:

When running disaggregated inference clusters (separating prefill nodes from decode workers), ensure nodes are interconnected via at least 200 Gbps RoCEv2 or InfiniBand interfaces. High network bandwidth ensures cross-node KV cache migration latency remains negligible compared to token generation intervals.

1. Combine Offloading with FP8 Cache Quantization:

Enable native FP8 KV cache storage formats where hardware supports it (NVIDIA Ada Lovelace, Hopper, and Blackwell architectures). Halving the on-wire tensor footprint doubles the effective transfer bandwidth of PCIe and network links without necessitating recomputation.

Sources

• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference (arXiv:2510.09665)
• FlexGen: High-Throughput Generation with Offloading for Large Language Models (arXiv:2303.06865)
• Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180)
• CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving (arXiv:2310.07240)
• Mooncake: Distributed KV Cache Storage Architecture (LMCache Documentation)
• Optimize LLM Inference with vLLM KV Cache Offloading on Amazon SageMaker AI (AWS Builder Center)
• NVIDIA GPUDirect Storage Overview Guide (NVIDIA Documentation)