CUDA2 articles

CUDA

Articles

CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination
CUDA Graph Capture in Production LLM Serving: Dynamic Batch Bucketing, Static Memory Pooling, and Kernel Launch Elimination In high-throughput large language model serving, the autoregressive generation (decode) phase presents an operational bottleneck distinct from prefill processing. While prefill computation is dominated by compute-bound General Matrix Multiplications (GEMMs) operating across extended sequence lengths, autoregressive decoding processes a single token per sequence at each ite
August 22, 20261 min
Custom LLM Kernel Optimization in Production: Triton vs. CUDA C++ vs. torch.compile vs. CUTLASS
Serving large language models at scale requires extracting maximum performance from modern GPU architectures like NVIDIA Ampere, Hopper, and Blackwell. While early production deployments relied on standard PyTorch eager execution and standard cuBLAS calls, high-throughput serving systems such as vLLM, SGLang, and TensorRT-LLM depend on specialized fused GPU kernels to eliminate memory bandwidth bottlenecks and saturate Tensor Cores. Engineering teams face four primary paradigms for kernel devel
August 20, 20261 min