llms.blog
LLM Serving3 articles

LLM Serving

Articles

  • CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination

    CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination In high-throughput large language model serving, the autoregressive generation (decode) phase presents an operational bottleneck distinct from prefill processing. While prefill computation is dominated by compute-bound General Matrix Multiplications (GEMMs) operating across extended sequence lengths, autoregressive decoding processes a single token per sequence at each ite

    1 min

  • Replit Launches Free Mode Powered by OpenAI's GPT-5.6 Luna

    Software development platform Replit announced the rollout of Free Mode, a tier powered by OpenAI's GPT-5.6 Luna model designed to support zero-cost planning, exploration, and codebase assistance. The integration utilizes recent inference cost reductions and efficiency improvements within the GPT-5.6 model family to provide unmetered conversational assistance without drawing from paid compute budgets. Model Routing and Persistent Project Context Replit Free Mode integrates directly into the

    1 min

  • LLM Serving Frameworks in Production: SGLang vs. vLLM vs. TensorRT-LLM Architecture, Memory Management, and Throughput Trade-Offs

    Serving large language models in production has moved beyond simple iteration-level scheduling. While basic continuous batching solved the fundamental problem of GPU starvation during autoregressive decoding, modern production demands have shifted the engineering bottleneck toward KV cache memory reuse, compilation overhead, and structured generation efficiency. Three dominant serving runtimes currently lead high-throughput deployments: vLLM, SGLang, and NVIDIA TensorRT-LLM. Each framework appr

    1 min