llms.blog
Systems6 articles

Systems

Articles

  • Zero-Downtime Model Updates in Production LLM Serving: In-Place Weight Transfer, CUDA IPC vs. NCCL Syncing, and Traffic Draining Architectures

    Zero-Downtime Model Updates in Production LLM Serving: In-Place Weight Transfer, CUDA IPC vs. NCCL Syncing, and Traffic Draining Architectures In high-throughput LLM serving infrastructure, updating model checkpoints presents a severe operational dilemma. Traditional microservice deployment patterns such as blue-green deployments or rolling pod restarts fail to scale economically when applied to multi-GPU LLM clusters. Tearing down a serving instance to load a newly fine-tuned checkpoint or pol

    1 min

  • GPU Memory Profiling in Production LLM Serving: CUDA Allocator Internals, PyTorch Snapshots, and VRAM Optimization

    In high-throughput large language model serving, memory is the primary constraint governing latency, batch concurrency, and context length. While model parameter footprints are static and easily calculated, runtime GPU memory (VRAM) dynamics are governed by low-level caching allocators, dynamic key-value (KV) cache allocation pools, transient activation spikes, and memory fragmentation. When an inference worker crashes with torch.cuda.OutOfMemoryError, default system diagnostics such as nvidia-

    1 min

  • Speculative Tool Execution and Parallel Action Graphs in Production AI Agents: DAG Scheduling, Optimistic Concurrency, and Side-Effect Rollback

    Autonomous AI agents operating in complex environments (such as software engineering, scientific discovery, and automated workflow orchestration) face an operational bottleneck: execution latency. While foundation models have become faster at token generation, the standard agent interaction loop remains strictly serial. A model reasons, outputs a single tool call, halts generation, waits for the external environment to return a result, appends the output to context, and resumes reasoning. In mu

    1 min

  • Custom LLM Kernel Optimization in Production: Triton vs. CUDA C++ vs. torch.compile vs. CUTLASS

    Serving large language models at scale requires extracting maximum performance from modern GPU architectures like NVIDIA Ampere, Hopper, and Blackwell. While early production deployments relied on standard PyTorch eager execution and standard cuBLAS calls, high-throughput serving systems such as vLLM, SGLang, and TensorRT-LLM depend on specialized fused GPU kernels to eliminate memory bandwidth bottlenecks and saturate Tensor Cores. Engineering teams face four primary paradigms for kernel devel

    1 min

  • Dynamic KV Cache Eviction in Production: Architecture, Sparsity Policies, and Serving Trade-Offs

    In long-context large language model serving, the key-value (KV) cache is the primary hardware bottleneck limiting concurrency and throughput. While model weights remain static during inference, KV cache memory scales linearly with sequence length, batch size, and layer count. For modern 70B parameter models utilizing Grouped-Query Attention (GQA), serving a 128,000-token context across a modest batch size of 4 requires over 80 GB of VRAM solely for KV states in 16-bit precision, exceeding the m

    1 min

  • Speculative Decoding in Production: Architecture, Economics, and Serving Trade-Offs

    Standard autoregressive large language model (LLM) generation produces one token per forward pass. Because modern inference architectures must read tens of gigabytes of model weights from high-bandwidth memory (HBM) into SRAM to process each solitary token at low batch sizes, generation is strictly memory-bandwidth bound rather than compute bound. Speculative decoding alters this hardware equation. By pairing a fast draft mechanism with parallel verification by the primary target model, inferen

    1 min