Large language model serving systems have historically treated transformer execution as a homogeneous sequence of forward passes over a single unified GPU pool. Under continuous batching engines, incoming requests execute their prompt evaluation (prefill) and autoregressive token generation (decode) on the exact same accelerators, co-locating both phases within shared iteration batches.
While continuous batching improves GPU compute utilization compared to static batching, co-locating prefill and decode creates structural scheduling interference. Prefill and decode exhibit opposing hardware bottlenecks, distinct arithmetic intensities, and divergent latency objectives. Production deployments handling long-context prompts, conversational threads, and agentic tool-use loops suffer severe Service Level Objective (SLO) violations under co-located serving.
Disaggregated prefill and decode (PD disaggregation) resolves this interference by physically separating the serving infrastructure into dedicated prefill clusters (P-pools) and decode clusters (D-pools). Prefill instances process prompt sequences at maximum compute saturation, populate Key-Value (KV) cache tensors, and transmit these tensors across high-bandwidth interconnects (such as RDMA or NVLink) to dedicated decoding instances optimized for memory bandwidth and output latency.
+-----------------------------------------------------------------------------------+
| DISAGGREGATED SERVING ARCHITECTURE |
+-----------------------------------------------------------------------------------+
Incoming Requests
│
▼
┌───────────────────────────┐
│ Global Request Router & │
│ KV-Aware Scheduler │
└─────────────┬─────────────┘
│
┌────────────────────┴────────────────────┐
│ │
▼ (Prompt Request) ▼ (Cached Prefix)
┌───────────────────────────┐ ┌───────────────────────────┐
│ PREFILL WORKER POOL │ │ DECODE WORKER POOL │
│ (Compute-Bound / GEMM) │ │ (Memory-Bound / GEMV) │
│ │ │ │
│ GPU 0 GPU 1 │ │ GPU A GPU B │
│ ┌───────┐ ┌───────┐ │ │ ┌───────┐ ┌───────┐ │
│ │Tensor │ │Tensor │ │ │ │ KV │ │ KV │ │
│ │Cores │ │Cores │ │ │ │ Cache │ │ Cache │ │
│ └───────┘ └───────┘ │ │ └───────┘ └───────┘ │
└─────────────┬─────────────┘ └─────────────▲─────────────┘
│ │
│ High-Bandwidth KV Transfer │
│ (GPUDirect RDMA / NVLink / RoCEv2) │
└─────────────────────────────────────────┘The Fundamental Serving Dichotomy: Arithmetic Intensity and Interference
The motivation for PD disaggregation stems from the Roofline model of GPU execution. The computational characteristics of the prefill and decode phases reside on opposite ends of the operational spectrum:
- Prefill Phase (Prompt Ingestion): In the prefill phase, the model processes S prompt tokens simultaneously. The core operations consist of general matrix multiplications (GEMMs) across the model weights and token representations. Because the matrix dimension M = S is large (frequently 1,024 to 128,000 tokens), the arithmetic intensity (ratio of floating-point operations to memory access) is high, frequently exceeding 100 FLOPs per Byte. Prefill is compute-bound, saturating the FP16/FP8 Tensor Cores of modern GPUs near theoretical peak TFLOPS.
- Decode Phase (Autoregressive Generation): In the decode phase, the model generates one token per sequence per forward step (M = 1). The matrix multiplications collapse into matrix-vector multiplications (GEMVs). For each generated token, the GPU must read all parameter weights from High Bandwidth Memory (HBM) into SRAM, along with the historical KV cache of all previous tokens. Arithmetic intensity drops to roughly 1 to 2 FLOPs per Byte. Decode is strictly memory-bandwidth bound, with Tensor Cores remaining idle while waiting for memory busses to fetch weights and cache entries.
Co-Location Interference and SLO Mismatch
When both phases share the same GPU under continuous batching engines (such as standard vLLM or TensorRT-LLM without phase splitting), batch execution exhibits severe cross-phase interference:
- Head-of-Line Blocking: When a long prompt (such as 16,000 tokens) enters the batch, the prefill iteration can consume 200 to 800 ms of GPU time. Any active decoding sequences batched alongside or queued behind this prefill are stalled, causing severe spikes in Time Between Tokens (TBT) and Time Per Output Token (TPOT).
- Coupled Parallelism Constraints: Optimal parallelism configurations differ between phases. Prefill benefits from higher Tensor Parallelism (TP = 4 or TP = 8) to reduce Time to First Token (TTFT) by distributing compute across multiple accelerators. Decode performs best under smaller TP degrees or Pipeline Parallelism (PP) to avoid recurring all-reduce communication overheads on tiny batch vector operations.
- SLO Contradiction: LLM applications enforce independent Service Level Objectives: TTFT (e.g., P95 < 500 ms) and TPOT (e.g., P95 < 25 ms per token). In a co-located system, optimizing for TTFT by prioritizing prefill directly violates TPOT, while prioritizing decode starvation leads to catastrophic TTFT queuing delays.
Academic systems like DistServe (OSDI 2024) and Splitwise (ISCA 2024) demonstrated that separating these phases completely eliminates co-location interference, delivering up to 7.4x higher request goodput while meeting strict tail-latency constraints.
KV Cache Network Migration: Mechanics and Bandwidth Math
The primary engineering challenge introduced by PD disaggregation is the network migration of the KV cache. Once a prefill worker finishes computing representations for a prompt, it must transfer the resulting Key and Value tensors to a designated decode worker before generation can begin.
Quantifying KV Cache Footprint
For a transformer model utilizing Grouped-Query Attention (GQA), the KV cache size generated by a sequence of length S is given by:
KV Cache Size = 2 * Layers * KV_Heads * Head_Dimension * Element_Bytes * Sequence_LengthRepresentative production KV cache payloads in FP16 format:
- Llama 3 8B (32 Layers, 8 KV Heads, 128 Head Dim): 64 KB per token. Total KV size is 256 MB at 4,096 tokens, 2.0 GB at 32,768 tokens, and 8.0 GB at 128,000 tokens.
- Llama 3 70B (80 Layers, 8 KV Heads, 128 Head Dim): 160 KB per token. Total KV size is 640 MB at 4,096 tokens, 5.0 GB at 32,768 tokens, and 19.5 GB at 128,000 tokens.
- Qwen 2.5 72B (80 Layers, 8 KV Heads, 128 Head Dim): 160 KB per token. Total KV size is 640 MB at 4,096 tokens, 5.0 GB at 32,768 tokens, and 19.5 GB at 128,000 tokens.
- DeepSeek V3 (61 Layers, Multi-head Latent Attention): 70.2 KB per token. Total KV size is 281 MB at 4,096 tokens, 2.25 GB at 32,768 tokens, and 8.8 GB at 128,000 tokens. Multi-head Latent Attention compresses KV representations into a 576-dimensional latent vector per layer, reducing transfer payload by over 50% relative to standard GQA architectures.
Network Interconnect Transfer Times
The transfer latency over the network directly impacts the overall TTFT:
Total TTFT = Queue Time + Prefill Compute Time + KV Transfer Latency + First Token Decode TimeTo maintain a low TTFT overhead, the transmission bandwidth between the P-pool and D-pool must match or exceed the model execution throughput:
- PCIe Gen5 x16 (Local Node, 64 GB/s effective): Transfer takes 4.0 ms for 256 MB (8B, 4k context), 78.1 ms for 5.0 GB (70B, 32k context), and 304.7 ms for 19.5 GB (70B, 128k context).
- NVLink 4 (Inter-GPU, 450 GB/s effective): Transfer takes 0.57 ms for 256 MB, 11.1 ms for 5.0 GB, and 43.3 ms for 19.5 GB.
- InfiniBand NDR / RoCEv2 (400 Gbps, 50 GB/s effective): Transfer takes 5.12 ms for 256 MB, 100.0 ms for 5.0 GB, and 390.0 ms for 19.5 GB.
- RoCEv2 (200 Gbps, 25 GB/s effective): Transfer takes 10.24 ms for 256 MB, 200.0 ms for 5.0 GB, and 780.0 ms for 19.5 GB.
- Standard 100 Gbps Ethernet (10 GB/s practical TCP): Transfer takes 25.6 ms for 256 MB, 500.0 ms for 5.0 GB, and 1,950.0 ms for 19.5 GB.
On a 400 Gbps RDMA network, transferring a 5 GB KV cache adds exactly 100 ms of network transfer time. For long context sequences exceeding 32k tokens, unoptimized sequential transfer can become the primary bottleneck.
Mitigating Migration Latency: Layer-by-Layer Pipelined Streaming
Production implementations such as Mooncake (USENIX FAST 2025) and Perplexity AI Research avoid bulk post-computation transfer by implementing asynchronous layer-by-layer streaming.
As the prefill GPU completes the forward pass for layer L, the generated Key-Value tensors for layer L are immediately dispatched via GPUDirect RDMA to the target decode instance while the prefill GPU proceeds to compute layer L + 1. Because the GEMM compute time for deep models exceeds the per-layer network transmission time across 400 Gbps links, network transfer is almost completely hidden beneath compute execution.
Prefill Node: [ Layer 1 GEMM ] [ Layer 2 GEMM ] [ Layer 3 GEMM ] ... [ Layer 80 GEMM ]
│ │ │
▼ (RDMA Stream) ▼ (RDMA Stream) ▼ (RDMA Stream)
Decode Node: [ Recv Layer 1 ] [ Recv Layer 2 ] [ Recv Layer 3 ] ... [ First Token Step ]Chunked Prefill vs. Physical PD Disaggregation
Before adopting full physical disaggregation, many operators implement Chunked Prefill (introduced in SARATHI and integrated into vLLM). Understanding the structural trade-offs between chunked prefill and disaggregation is essential for infrastructure sizing.
+-----------------------------------------------------------------------------------------+
| CHUNKED PREFILL vs. DISAGGREGATED SERVING |
+-----------------------------------------------------------------------------------------+
1. CHUNKED PREFILL (Colocated GPU):
Iteration 1: [ Chunk 1 (512 tokens) + Decode 1 + Decode 2 + Decode 3 ] (Compute balanced)
Iteration 2: [ Chunk 2 (512 tokens) + Decode 1 + Decode 2 + Decode 3 ]
Iteration 3: [ Chunk 3 (512 tokens) + Decode 1 + Decode 2 + Decode 3 ]
- Problem: Prolongs TTFT by splitting prompt across multiple iterations;
GEMM arithmetic intensity drops for smaller chunks;
Memory fragmentation increases under high concurrency.
2. PHYSICAL DISAGGREGATION (Separate Pools):
Prefill Pool: [ Full Prompt 1 (16,384 tokens) ] -> Max Compute Saturation (Peak TFLOPS)
Decode Pool: [ Decode 1 + Decode 2 + ... + Decode 64 ] -> Pure Memory Bandwidth GEMV
- Transfer: Zero-copy RDMA streams KV pages concurrently with execution.Detailed Comparison
- Arithmetic Intensity and Kernel Efficiency:
- Chunked Prefill: Slicing a 16k prompt into 512-token chunks reduces the inner matrix dimension M, causing the GPU to operate below its peak Roofline compute ceiling. Slicing adds kernel launch overhead and intermediate tensor staging.
- PD Disaggregation: Prefill instances execute full-length sequence GEMMs at maximum batch efficiency, achieving peak hardware utilization (60% to 75% Model FLOPs Utilization).
- Latency Predictability (TBT and TTFT):
- Chunked Prefill: Mitigates extreme decode starvation, but still introduces noticeable TBT variance (2x to 4x jitter) whenever large prompt bursts enter the scheduling queue.
- PD Disaggregation: Decode instances run dedicated token generation loops without interruption. TBT variance is virtually zero (<5% jitter), guaranteeing hard TPOT SLOs.
- Memory Pressure and KV Retention:
- Chunked Prefill: Prefill activation memory, intermediate chunk states, and long-term decode KV caches compete for the same on-device HBM.
- PD Disaggregation: Prefill nodes require only minimal KV memory (just enough to hold active prefill batches before offloading), allowing virtually all memory to be allocated to compute workspace. Decode nodes dedicate 90% or more of HBM strictly to KV cache storage.
Asymmetric Hardware Economics
One of the greatest economic advantages of PD disaggregation is breaking the requirement for homogeneous GPU clusters. Because prefill and decode exhibit fundamentally different hardware bottlenecks, they do not require the same class of hardware.
+------------------------------------------------------------------------------------+
| ASYMMETRIC HARDWARE ALLOCATION |
+------------------------------------------------------------------------------------+
PREFILL CLUSTER (Compute-Optimized) DECODE CLUSTER (Bandwidth-Optimized)
- Primary Metric: Dense FP8/FP16 TFLOPS - Primary Metric: HBM3e Bandwidth (TB/s)
- Tensor Parallelism: TP=4 or TP=8 - Parallelism: TP=1 or TP=2 + Pipeline/Spec
- Memory Size: Moderate (64-80 GB sufficient) - Memory Size: Massive (141-192 GB+ for KV)
- Node Profile: High Compute Density - Node Profile: Maximum Memory Capacity
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ 4x NVIDIA B200 / H100 │ │ 8x AMD MI300X / H200 / L40S │
│ (Peak Tensor Core Compute) │ │ (High HBM Capacity/Bandwidth│
└──────────────┬──────────────┘ └──────────────▲──────────────┘
│ │
└─────────────── RDMA / 400GbE ───────────────┘Hardware Profile Matching
- Prefill Instances (Compute-Dense Profile):
- Ideal Hardware: NVIDIA H100 SXM5, NVIDIA B200, or TPU v5e.
- Characteristics: Maximum FP8/FP16 TFLOPS, high inter-card NVLink bandwidth to support large Tensor Parallelism (TP = 4 or 8), low requirement for massive per-GPU HBM since requests leave the node immediately after prefill.
- Decode Instances (Memory-Dense Profile):
- Ideal Hardware: NVIDIA H200 (141 GB HBM3e, 4.8 TB/s), AMD Instinct MI300X (192 GB HBM3, 5.3 TB/s), or high-density PCIe accelerators (L40S / RTX 6000 Ada with large host memory staging).
- Characteristics: Massive HBM capacity and bandwidth. A cluster of 8x AMD MI300X GPUs provides 1.5 TB of unified HBM and over 42 TB/s of aggregate memory bandwidth, allowing hundreds of concurrent sequences to decode without running out of KV cache capacity.
Total Cost of Ownership (TCO) Impact
In production environments with a 4:1 input-to-output token ratio (typical of retrieval-augmented generation and code review tasks), a co-located H100 cluster frequently operates at only 20% to 30% average MFU due to memory-bound decode stalls.
By disaggregating into a ratio of 1 compute-dense H100 prefill node to 3 memory-optimized decode nodes (or lower-cost high-memory accelerators), overall cluster serving capacity increases by 2.35x to 4.0x for the same capital expenditure, reducing the effective cost per million tokens by 35% to 50%.
Production Implementation: Routing Topologies and Scheduling Topologies
Implementing PD disaggregation requires a coordinated control plane and network protocol. Modern open-source serving runtimes (including SGLang Disaggregated and vLLM Disaggregated Router) utilize the following architectural building blocks:
+-----------------------------------------------------------------------------------+
| PRODUCTION DISPATCH AND MIGRATION |
+-----------------------------------------------------------------------------------+
Client Request
│
▼
┌───────────────────────────────────────────────────────────────────────────────────┐
│ Router & Dispatcher: │
│ 1. Hash prefix tokens -> Check prefix cache hit across P-Pool and D-Pool │
│ 2. If prompt cache misses: Select P-Node with lowest queue latency │
│ 3. Select D-Node with available KV block capacity and matching tensor partition │
│ 4. Reserve KV blocks on target D-Node (allocate physical page addresses) │
└────────────────────────────────────────┬──────────────────────────────────────────┘
│
┌─────────────────────┴─────────────────────┐
│ │
▼ (Step 1: Execute Prefill) ▼ (Step 2: Pre-allocate)
┌──────────────────────────────────────────────┐ ┌─────────────────────────────────┐
│ Prefill Node (P-Node): │ │ Decode Node (D-Node): │
│ - Run forward GEMM over prompt tokens │ │ - Mark reserved memory blocks │
│ - Extract KV tensors from attention blocks │ │ as non-evictable │
│ - Stream directly to D-Node physical memory │ │ - Listen on RDMA QP endpoint │
└──────────────────────┬───────────────────────┘ └─────────────────▲───────────────┘
│ │
└────────── [ GPUDirect RDMA Write ] ────────┘
│
▼ (Step 3: Transfer Complete)
┌───────────────────────────────────┐
│ D-Node begins step generation │
│ Output tokens stream back to user │
└───────────────────────────────────┘Production Engineering Pitfalls and Failure Modes
- RDMA Incast Congestion: When multiple prefill nodes finish long prompts simultaneously and target the same decode node, network switch buffers can saturate, triggering PFC (Priority Flow Control) pause frames or packet drops. Production systems mitigate this with rate-limiting transmission rings and multi-path ECMP routing.
- Decode Memory Deadlock: If prefill completes but the target decode instance has exhausted its unpinned KV cache pages due to sudden generation expansion, the transfer blocks. Schedulers must enforce strict two-phase memory reservations: a prefill request is only dispatched if the target decode node has guaranteed, un-evictable memory slots reserved upfront.
- Prefix Caching Fragmentation: When prefix caching (RadixAttention) is enabled, prompts sharing common system instructions should not recompute prefill. If the cached prefix resides in a decode node memory, transferring or recomputing requires a cache-aware global directory. Hierarchical KV stores (such as Mooncake Store) manage multi-tiered caching across GPU HBM, Host DRAM, and remote NVMe pools.
Architectural Summary
+---------------------------------------------------------------------------------------------------+
| COMPARATIVE SUMMARY |
+--------------------------------------+----------------------+------------------+------------------+
| Architectural Metric | Continuous Batching | Chunked Prefill | PD Disaggregation|
+--------------------------------------+----------------------+------------------+------------------+
| Throughput (Goodput Multiplier) | 1.0x (Baseline) | 1.3x - 1.6x | 2.5x - 7.4x |
| TTFT Stability | Poor (Stalled) | Moderate | Optimal |
| TPOT / TBT Tail Latency Jitter (P99) | High (>5x P50) | Moderate (~2x) | Minimal (~1.1x) |
| GPU Memory Specialization | Compromised | Fragmented | Specialized (90%)|
| Interconnect Requirement | Local Only | Local Only | RDMA / 400GbE |
| Hardware Heterogeneity | Homogeneous Only | Homogeneous Only | Fully Asymmetric |
+--------------------------------------+----------------------+------------------+------------------+Sources
- Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., & Zhang, H. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24), 193–210.
- Patel, P., Choukse, E., Zhang, C., Shah, A., & Goiri, I. (2024). Splitwise: Efficient Generative LLM Inference Using Phase Splitting. Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA '24).
- Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., & Xu, X. (2025). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. USENIX Conference on File and Storage Technologies (FAST '25).
- Agrawal, A., Kedia, N., Panwar, A., Jayaram, K. R., Mutlu, O., & Ramjee, B. (2024). SARATHI: Efficient LLM Inference by Chunking Prefills. ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '24).
- Perplexity AI Research. (2024). Disaggregated Prefill and Decode. Perplexity Engineering Blog.
- vLLM Team. (2024). vLLM Disaggregated Prefill and Decode Architecture. vLLM Documentation.
- SGLang Project. (2024). SGLang PD Disaggregation and Router Architecture. SGLang Repository.
