SGLang2 articles

SGLang

Articles

Prefix-Tree KV Cache Management in Production: RadixAttention, Tree-Structured LRU Eviction, and Token-Level Sharing in SGLang and vLLM
Prefix-Tree KV Cache Management in Production: RadixAttention, Tree-Structured LRU Eviction, and Token-Level Sharing in SGLang and vLLM Autoregressive large language model inference is heavily constrained by memory bandwidth and the computational overhead of the prefill phase. For workloads such as multi-turn conversations, autonomous agent tool loops, few-shot prompt evaluations, and tree-search decoding, consecutive requests often share substantial token prefixes. In a standard multi-turn ses
August 22, 20261 min
LLM Serving Frameworks in Production: SGLang vs. vLLM vs. TensorRT-LLM Architecture, Memory Management, and Throughput Trade-Offs
Serving large language models in production has moved beyond simple iteration-level scheduling. While basic continuous batching solved the fundamental problem of GPU starvation during autoregressive decoding, modern production demands have shifted the engineering bottleneck toward KV cache memory reuse, compilation overhead, and structured generation efficiency. Three dominant serving runtimes currently lead high-throughput deployments: vLLM, SGLang, and NVIDIA TensorRT-LLM. Each framework appr
August 18, 20261 min