LLM Inference on Enterprise CPUs in Production: Architecture, NUMA Topologies, Matrix Extensions, and Serving Economics

Deploying large language models has historically been treated as an exclusively GPU-centric problem. Accelerators like NVIDIA H100 and A100 GPUs provide multi-terabyte-per-second High Bandwidth Memory (HBM) and tensor cores essential for training and high-concurrency serving. However, the operational economics of enterprise inference often diverge from frontier training requirements. Many enterprise applications, including internal code assistance, document extraction, low-concurrency agents, and on-premises deployments, experience sporadic traffic profiles where dedicating a $30,000 GPU or a continuous $3-per-hour cloud instance results in single-digit hardware utilization.

Recent developments in CPU silicon architectures and inference software runtimes have transformed CPU-based LLM inference from a slow fallback into a viable production strategy. Hardware innovations like Intel Advanced Matrix Extensions (AMX), AMD AVX-512 VNNI, ARM SVE2, and multi-channel DDR5 memory subsystems, combined with runtimes like Intel IPEX-LLM, OpenVINO, llama.cpp, and the vLLM CPU backend, allow modern multi-socket server processors to deliver 15 to 40 tokens per second on quantized 7B to 14B parameter models.

Understanding how to extract production performance from CPU inference requires analyzing the memory-bandwidth bottleneck, configuring Non-Uniform Memory Access (NUMA) topologies, selecting the right software execution engines, and calculating the exact cost-performance trade-offs.

The Two Compute Regimes: Why Prefill and Decode Diverge on CPU

To understand CPU inference performance, serving must be separated into its two distinct operational phases:

1. The Prefill Phase (Compute-Bound GEMM): During prompt ingestion, the model processes all input tokens concurrently. This operation consists of large General Matrix Multiplications (GEMMs), where the arithmetic intensity (FLOPs performed per byte of memory moved) is high. On modern CPUs equipped with matrix engines like Intel AMX, prefill latency scales with available compute tiles and clock frequency, allowing prompt processing speeds to exceed several hundred tokens per second for moderate context lengths.
2. The Decode Phase (Memory-Bandwidth-Bound GEMV): During autoregressive generation, the model generates output tokens one at a time. For each new token, every weight in the network must be read from system RAM into processor caches to execute General Matrix-Vector (GEMV) multiplications. Because arithmetic intensity in single-batch decoding is extremely low (roughly 1 FLOP per byte loaded in standard 16-bit precision), generation speed is constrained by DRAM bandwidth rather than CPU clock speed.

The theoretical upper bound for single-sequence token generation speed on any architecture is governed by memory bandwidth:

$$\text{Maximum Decode Speed (Tokens/s)} \approx \frac{\text{Sustained Memory Bandwidth (GB/s)}}{\text{Model Parameter Memory Footprint (GB)}}$$

For example, on a dual-socket server with 16 channels of DDR5-4800 memory providing approximately 600 GB/s of aggregate bandwidth:

• A 7B parameter model quantized to 4 bits (INT4, requiring ~4.5 GB of weight storage) achieves a theoretical decoding limit of over 100 tokens/s across parallel threads, with real-world single-socket single-stream decodes routinely delivering 25 to 35 tokens/s.
• An unquantized 70B parameter model in FP16 (~140 GB) is limited to 3 to 4 tokens/s, illustrating why model quantization is mandatory for interactive CPU serving.

Hardware Accelerators on Modern CPUs

Modern enterprise server CPUs integrate specialized execution units that bridge the gap with dedicated AI silicon:

• Intel Advanced Matrix Extensions (AMX): Introduced in 4th Gen Intel Xeon Scalable processors (Sapphire Rapids) and expanded in 5th Gen (Emerald Rapids) and Xeon 6 (Granite Rapids), Intel AMX implements 2D tile registers (TMM0-TMM7) and a Matrix Multiplication Engine (TMUL). AMX operates on 8-bit integers (INT8) and 16-bit Bfloat16 (BF16), providing up to a 7x to 10x throughput boost in matrix-dense operations over standard vector instructions.
• AMD AVX-512 with VNNI: AMD EPYC 9004 and 9005 processors (Genoa and Turin) implement 512-bit Vector Neural Network Instructions (VNNI) that accelerate INT8 and BF16 dot products across 12-channel DDR5 memory controllers, providing up to 460 GB/s of memory bandwidth per socket.
• Intel Xeon Max Series (HBM2e): The Xeon Max 9400 series packages 64 GB of on-die High Bandwidth Memory (HBM2e) directly onto the CPU package, providing up to 1.6 TB/s of aggregate memory bandwidth without requiring external discrete GPUs.
• ARM Neoverse & AWS Graviton: Systems like AWS Graviton4 utilize ARM Neoverse V2 cores with dual 256-bit SVE2 (Scalable Vector Extension) pipelines and 12-channel DDR5 subsystems to optimize energy-efficient integer matrix inference.

NUMA Topologies, Core Pinning, and Memory Locality

The most common failure mode in CPU LLM deployment is ignoring system Non-Uniform Memory Access (NUMA) architecture. In multi-socket servers or processors with multiple compute chiplets (CCDs), accessing memory attached to a remote socket or remote memory controller introduces significant latency and saturates cross-socket interconnects (such as Intel Ultra Path Interconnect or AMD Infinity Fabric).

To achieve predictable, low-latency decoding, production deployments must adhere to four system configuration rules:

1. Sub-NUMA Clustering (SNC): In BIOS settings, enable Sub-NUMA Clustering (SNC-2 or SNC-4 on Intel; NUMA Nodes Per Socket / NPS=2 or NPS=4 on AMD). This partitions the cores, L3 cache, and local memory controllers into discrete NUMA domains, minimizing intra-socket ring bus contention.
2. Strict Process and Memory Binding: Run inference workers pinned strictly to their local NUMA node using numactl. Never allow a single LLM worker to span multiple NUMA nodes for single-stream generation. For example:

numactl --cpunodebind=0 --membind=0 python3 -m vllm.entrypoints.openai.api_server --model ...

1. Core Isolation and Thread Pinning: Disable hyperthreading (Simultaneous Multi-Threading / SMT) for inference worker threads or pin workers exclusively to physical cores (OMP_PLACES=cores, OMP_PROC_BIND=close). Hyperthreaded sibling cores compete for execution pipelines and L1/L2 caches, causing erratic token latency.
2. Deploy Independent Replicas Instead of Inter-Socket Parallelism: Rather than running one large instance across a dual-socket server with Tensor Parallelism (TP=2) over the inter-socket interconnect, deploy independent single-socket or single-NUMA-node instances behind a local load balancer. This eliminates cross-socket communication overhead and doubles aggregate concurrency.

Production Runtime Comparison: IPEX-LLM, OpenVINO, and llama.cpp

Selecting the right runtime determines how effectively hardware vector units and memory bandwidth are utilized:

• Intel IPEX-LLM: A specialized library providing optimized PyTorch kernels and integration with vLLM, Ollama, and Hugging Face. IPEX-LLM implements Weight-Only Quantization (WOQ) for INT4, INT8, and FP4 formats, dynamically decompressing weights into registers for direct execution on AMX and AVX-512 units. It delivers the lowest Time to First Token (TTFT) on Intel Xeon hardware.
• OpenVINO Runtime: Intel's model optimization toolkit converts models into OpenVINO Intermediate Representation (IR) with advanced graph fusions (such as fused Multi-Query Attention and RoPE kernels). OpenVINO incorporates the Neural Network Compression Framework (NNCF) for asymmetric 4-bit weight compression and offers an out-of-the-box C++ runtime suitable for embedded and microservice deployments.
• llama.cpp / ggml: The standard for standalone quantized CPU inference. Written in pure C/C++, llama.cpp uses custom quantized formats (such as Q4_K_M, Q5_K_M, and IQ4_XS) optimized for SIMD dot products. Its memory-mapped file loading (mmap) allows multiple processes on the same host to share read-only weight pages in physical RAM, drastically reducing memory overhead for multi-instance deployments.
• vLLM CPU Backend: Designed for high-concurrency API serving, the vLLM CPU backend brings PagedAttention and continuous batching to x86 and ARM servers. Operating with VLLM_CPU_KVCACHE_SPACE allocations mapped per NUMA node, vLLM allows CPU clusters to serve standard OpenAI-compatible endpoints with dynamic request preemption and streaming.

Economic Sizing and Workload Suitability

Evaluating whether CPU inference is appropriate requires analyzing latency thresholds against infrastructure costs.

CPU inference is economically and technically optimal in the following scenarios:

• Low-Concurrency Internal Tooling: Applications where user requests occur irregularly (such as internal support bots, HR query assistants, and document QA) do not justify continuous GPU rental. An instance running on standard general-purpose or memory-optimized cloud instances can sit idle at nominal cost and handle requests within acceptable human reading speeds (15-25 tokens/s).
• High-Memory Working Contexts (Large Context RAG): GPUs are constrained by physical VRAM capacity (e.g., 24 GB on an A10G or 80 GB on an A100). Expanding the KV cache for a 128K context window across multiple concurrent requests can quickly cause GPU Out-of-Memory (OOM) errors. Server CPUs support terabytes of commodity DDR5 RAM, allowing massive context windows and extensive KV cache allocations at a fraction of the cost per gigabyte.
• Air-Gapped and Edge Deployments: Defense, healthcare, and industrial edge environments often lack liquid cooling, high-voltage power delivery, or physical space for GPU accelerator chassis. Modern CPU servers fit into standard enterprise rack envelopes and power budgets while fulfilling inference requirements locally.
• Batch Document Processing Pipelines: Non-interactive batch jobs (such as entity extraction, offline summarization, and data categorization) prioritize aggregate throughput per dollar over real-time latency. Distributing batch jobs across hundreds of existing idle CPU cores in corporate compute clusters eliminates the need for dedicated GPU procurement.

Conversely, CPU inference remains unsuitable for high-concurrency public APIs, real-time voice agents requiring sub-100ms response times, and unquantized frontier models (70B+ parameters) where low memory bandwidth results in sluggish generation.

By matching model precision, NUMA topology binding, and modern execution frameworks to the appropriate workload profile, engineering teams can build robust, cost-effective LLM production architectures on standard enterprise CPU infrastructure.

Sources

• GitHub: Intel IPEX-LLM Acceleration Library
• Intel: Practical Guide to CPU-Optimized LLM Deployment on Xeon Processors
• Lenovo Press: Accelerating LLM Performance with Intel AMX and IPEX
• vLLM Documentation: CPU Backend Installation and Configuration
• OpenVINO Documentation: LLM Inference Pipeline Optimization
• Databricks: LLM Inference Performance Engineering Best Practices