The open-source LLM serving engine SGLang has released version 0.5.18, introducing an overlapped startup engine that significantly reduces cold-start latency for large language models, alongside communication kernel optimizations and expanded architecture support.
Comprising 710 pull requests from 212 contributors, the release addresses operational overheads in LLM infrastructure where autoscaling, rolling cluster deployments, and worker node recovery frequently pay steep restart penalties.
Overlapping Checkpoint Staging and CUDA Graph Capture
In traditional LLM serving runtimes, initialization executes sequentially: the engine first loads model weights from disk or network storage into host and device memory, runs kernel autotuning, and then captures CUDA graphs across multiple batch bucket sizes to minimize runtime kernel launch overhead.
SGLang v0.5.18 introduces an overlapped startup mode, enabled via the --startup-weight-load-mode overlap flag. The mechanism stages model checkpoint pages from storage concurrently with the initialization and capture of CUDA execution graphs.
According to benchmark data published in the release notes, initializing a Qwen3-32B checkpoint on an NVIDIA H100 system drops from 84.8 seconds under standard serial loading to 35.6 seconds with overlapped staging, representing a 2.38x speedup. The approach also outperforms standard serial prefetching pipelines by 8.6% to 11.7%.
All-to-All Communication for Tensor-Parallel LMHead
The release restructures the final language model head (LMHead) computation across tensor-parallel ranks. Previously, computing logits over large vocabularies across partitioned ranks required an allgather collective followed by a scatter operation.
Version 0.5.18 refactors this path into a single all-to-all communication pattern for pure data-parallel attention layouts. On NVIDIA B200 accelerators running DeepSeek-V4-Pro decode workloads, execution time in the LMHead drops from 320 microseconds to 169 microseconds, reducing time per output token (TPOT) from 36.97 ms to 35.67 ms.
Additionally, the runtime integrates FlashInfer's Multi-Node NVLink (MNNVL) workspace for non-fused allreduce sites, avoiding fallback to standard NCCL collectives. When serving DeepSeek-V4-Flash under tensor parallelism 4 (TP4) on Blackwell GPUs, decode throughput improves by up to 6.9% at low batch sizes.
Unified Kernel Caching and Architectural Additions
SGLang v0.5.18 unifies caching infrastructure across compilation layers. Previously, runtime artifacts from Triton, FlashInfer, PyTorch Inductor, DeepGEMM, and CUDA driver JIT compilers were distributed across separate system paths. All compiled artifacts are now centralized under SGLANG_CACHE_DIR.
The release expands architecture coverage across autoregressive and diffusion models, adding native serving engines for:
- Muse Glimmer: Multimodal autoregressive vision-language model.
- Intern-S2-Mobius: Autoregressive foundation architecture.
- Diffusion Video & Image Frameworks: SANA-Video, LingBot-Video-MoE, LTX-2.5, Cosmos3 Edge and Distilled, and LongCat-Image.
- Recipe Additions: Validated serving recipes for Qwen 3.8, Ling-3.0, Nemotron 3.5 Lightning, Dots3-Note, and DeepSeek-V4-Pro-0813.
Core framework dependencies have been updated to PyTorch 2.13.0 with Triton 3.7.1, FlashInfer 0.6.17, CuTeDSL 4.6.2, and sgl-kernel 0.4.6.post1.
