llms.blog
Kubernetes3 articles

Kubernetes

Articles

  • GPU Cluster Scheduling in Production: Slurm vs. Kubernetes (Kueue/Volcano) vs. Ray

    GPU Cluster Scheduling in Production: Slurm vs. Kubernetes (Kueue/Volcano) vs. Ray Modern AI infrastructure represents a radical departure from traditional cloud computing. Standard cloud workloads (such as stateless microservices, web applications, and independent batch jobs) rely on fine-grained elasticity, independent container scheduling, and horizontal autoscaling. In contrast, distributed large language model (LLM) training and high-throughput inference pipelines violate virtually every a

    1 min

  • GPU Slicing in Production AI Systems: Comparing MIG, MPS, Time-Slicing, and Dynamic Partitioning

    GPU Slicing in Production AI Systems: Comparing MIG, MPS, Time-Slicing, and Dynamic Partitioning Modern production AI systems rarely deploy a single standalone large language model. Contemporary compound AI architectures rely on heterogeneous pipelines comprising embedding models (such as BGE or E5), cross-encoder rerankers, safety classifiers (such as Llama Guard), speculative decoding draft models, and vision-language encoders. While primary generation models typically require dedicated multi

    1 min

  • LLM Autoscaling and Cold Starts in Kubernetes: Architecture, KEDA Metrics, Model Weight Caching, and Ephemeral GPU Provisioning

    Autoscaling large language model workloads on Kubernetes presents a fundamentally different engineering problem than traditional stateless microservices. While web APIs scale on CPU utilization or request rate within seconds, LLM inference instances require specialized GPU accelerators, massive container images, multi-gigabyte weight tensors, and intensive runtime compilation before serving a single token. Without proactive architectural design, a cold-starting LLM pod on Kubernetes often requi

    1 min