Serving Mixture-of-Experts Models in Production: Architecture, Distributed Parallelism, All-to-All Bottlenecks, and Serving Economics

Mixture-of-Experts (MoE) architectures have become the standard structural paradigm for frontier open-weight and proprietary large language models. Architectures such as DeepSeek-V3, Mixtral 8x22B, and GLM-5 deliver frontier-grade reasoning and generation by activating only a fraction of their total parameter count on any given token. For example, DeepSeek-V3 routes each token to 8 active experts out of 256 routed experts (plus 1 shared expert), executing 37 billion active parameters out of a total 671 billion parameter footprint.

While sparse gating significantly reduces the floating-point operations (FLOPs) required per token during forward passes, deploying MoE models in high-concurrency production environments introduces acute distributed systems challenges. Inference engines must store the full parameter footprint in GPU high-bandwidth memory (HBM), balance dynamically shifting token-to-expert distributions, and execute high-frequency All-to-All collective communications across GPU clusters without stalling compute pipelines.

This analysis examines the architectural mechanics of production MoE serving, evaluating Tensor Parallelism versus Expert Parallelism, All-to-All communication overheads, dynamic load balancing algorithms, and the underlying economics governing MoE inference infrastructure.

1. The Sparse Memory-Compute Disconnect

The core operational challenge of MoE serving stems from the structural asymmetry between memory capacity requirements and compute workloads.

In dense transformer architectures, memory footprint and compute load scale together. A 70-billion-parameter dense model requires roughly 140 GB of VRAM in FP16 (or 70 GB in FP8) and performs approximately 140 billion FLOPs per generated token.

MoE architectures decouple these dimensions: active parameters per token remain small, while total parameters in VRAM remain massive.

For a model with total experts where top-K experts are routed per token alongside shared base layers:

1. Static Memory Footprint: The entire weight tensor matrix for all experts must reside permanently in fast memory (HBM) to avoid catastrophic PCIe or NVLink swapping penalties during autoregressive decoding.
2. Dynamic Compute Intensity: Each token touches only the attention layers and K Feed-Forward Network (FFN) expert blocks.

During low-concurrency or single-stream autoregressive decoding, MoE models become severely memory-bandwidth bound. Because every generated token requires streaming the weights of K distinct experts from HBM into SRAM, low batch sizes yield poor Arithmetic Intensity (FLOPs per byte transferred). MoE models only unlock high Hardware FLOPs Utilization (HFU) and Model Flops Utilization (MFU) when concurrent batch sizes are large enough that multiple tokens within the same batch activate overlapping experts, reusing loaded weight tensors across batched matrix multiplications.

2. Parallelism Topologies: TP vs. EP vs. Wide-EP

Distributing a multi-hundred-billion parameter MoE checkpoint across GPUs requires selecting the appropriate distributed execution topology.

+-----------------------------------------------------------------------------+
|               Data-Parallel Attention + Wide-EP FFN Architecture            |
+-----------------------------------------------------------------------------+
|                                                                             |
|  [GPU Rank 0]        [GPU Rank 1]        [GPU Rank 2]        [GPU Rank 3]   |
|  +--------------+    +--------------+    +--------------+    +--------------+
|  | DP Attention |    | DP Attention |    | DP Attention |    | DP Attention |
|  | (Tokens 0-31)|    | (Tokens 32-63|    | (Tokens 64-95|    | (Tokens 96..)|
|  +------+-------+    +------+-------+    +------+-------+    +------+-------+
|         |                   |                   |                   |       |
|         +-------------------+---------+---------+-------------------+       |
|                                       |                                     |
|                       [All-to-All Token Dispatch Matrix]                    |
|                               (DeepEP / NCCL)                               |
|                                       |                                     |
|         +-------------------+---------+---------+-------------------+       |
|         |                   |                   |                   |       |
|  +------+-------+    +------+-------+    +------+-------+    +------+-------+
|  | EP Expert 0-7|    | EP Exp 8-15  |    | EP Exp 16-23 |    | EP Exp 24-31 |
|  | FFN Compute  |    | FFN Compute  |    | FFN Compute  |    | FFN Compute  |
|  +------+-------+    +------+-------+    +------+-------+    +------+-------+
|         |                   |                   |                   |       |
|                       [All-to-All Token Combine Matrix]                     |
|                                       |                                     |
|  +------+-------+    +------+-------+    +------+-------+    +------+-------+
|  | Residual Sum |    | Residual Sum |    | Residual Sum |    | Residual Sum |
|  +--------------+    +--------------+    +--------------+    +--------------+
+-----------------------------------------------------------------------------+

Pure Tensor Parallelism (TP)

In pure Tensor Parallelism, every expert linear layer is sharded across all GPUs in the tensor parallel group.

• Execution Flow: Each GPU retains a slice of every expert. For every token routed to an expert, each GPU computes a partial GEMM output on its local weight shard, followed by an All-Reduce collective operation to sum the results across the TP rank.
• Trade-Offs: Pure TP works well when total expert count is small (such as 8 experts in Mixtral 8x7B) and all GPUs reside within a single NVLink domain (8-GPU node). However, as expert counts scale to 64, 128, or 256, slicing already small expert matrices across 8 or 16 GPUs creates memory-bandwidth inefficiencies where matrix dimensions fall below the threshold for efficient GPU Tensor Core tile occupancy.

Pure Expert Parallelism (EP)

In Expert Parallelism, complete, unsharded expert FFN blocks are assigned to distinct GPUs across the cluster. If a model has 64 experts and is deployed across 8 GPUs, each GPU hosts 8 full experts.

• Execution Flow: When token representations exit the shared self-attention layer, the router assigns top-K expert IDs. Tokens are dispatched to the specific GPUs hosting the selected experts using an All-to-All collective operation (AllToAllv). The destination GPUs execute full-width GEMM kernels on their local experts and send the computed activations back to the originating rank via a second All-to-All step.
• Trade-Offs: Individual expert GEMMs run at full matrix dimensions with optimal Tensor Core occupancy. However, execution latency depends entirely on cross-GPU interconnect bandwidth (InfiniBand or RoCEv2).

Modern Standard: Data-Parallel Attention with Wide-EP

Modern production frameworks, including vLLM Wide-EP implementation and SGLang distributed runtime, combine Data Parallelism (DP) for attention with Expert Parallelism (EP) for MoE layers.

In this setup:

1. Self-attention layers run in Data Parallel mode, where each GPU processes an independent slice of batch tokens without inter-GPU communication during attention.
2. MoE layers switch dynamically to Expert Parallel mode via All-to-All token dispatch.
3. This eliminates Tensor Parallel All-Reduce operations during the attention block and preserves maximal KV cache capacity on each rank.

3. The All-to-All Collective Communication Bottleneck

The primary latency barrier in distributed MoE serving is the two-step All-to-All communication pattern required at every MoE layer:

1. Dispatch: Routing token vectors from source GPUs to target expert GPUs.
2. Combine: Returning processed token vectors back to original token-holding ranks to be added to the residual stream.

      Token Dispatch (Scatter by Expert Assignment)
      GPU 0: [Token A -> Exp 15]  ===\     /===> GPU 0 (Hosts Exp 0-7)
      GPU 1: [Token B -> Exp 3]   ====\   /====> GPU 1 (Hosts Exp 8-15)
      GPU 2: [Token C -> Exp 28]  ==== \ / =====> GPU 2 (Hosts Exp 16-23)
      GPU 3: [Token D -> Exp 6]   ====  X  =====> GPU 3 (Hosts Exp 24-31)
                                       / \
      Token Combine (Gather to Source for Residual Accumulation)

In an R-rank EP deployment with hidden dimension D (such as D = 7168 for DeepSeek-V3), routing B tokens per rank with top-K routing requires transmitting 2 * B * K * D * BytesPerElement data per layer.

Across 60 or more MoE layers, if cross-node interconnect bandwidth is constrained (such as dual 400 Gbps NICs versus multi-terabit NVLink), network serialization latency dominates kernel execution time.

Optimized Communication Kernels: DeepEP and Asymmetric Scheduling

To prevent communication stalls, serving architectures rely on low-latency MoE communication libraries like DeepEP. DeepEP optimizes NVLink and RDMA primitives by:

• Intra-node SM-bypass: Utilizing direct peer-to-peer NVLink reads and writes without saturating GPU Streaming Multiprocessors.
• Inter-node low-latency IB dispatch: Implementing packet-level streaming where expert GEMM execution begins immediately as early token chunks arrive, rather than waiting for the entire All-to-All transfer to complete.
• Dual-Batch Overlap (DBO): Pipelining two independent micro-batches such that while micro-batch N computes expert GEMMs on local hardware, micro-batch N+1 executes its All-to-All communication over the network fabric.

4. Load Imbalance and Dynamic Routing Sizing

While MoE pre-training incorporates auxiliary load-balancing losses to distribute tokens evenly across experts, real-world inference workloads exhibit severe routing skew. Specific user domains (such as Python programming, legal analysis, or math proofs) consistently activate a narrow subset of domain-specialized experts.

This skew creates the straggler problem: if 70% of tokens in a batch route to Expert 12 located on GPU Rank 1, GPU 1 remains compute-saturated while GPUs 0, 2, and 3 sit idle, waiting for the All-to-All combine barrier.

Sizing and Routing Strategies Compared

• Capacity Factor Dropping: Enforces a strict token cap per expert. Tokens exceeding the cap bypass the expert FFN via residual connection. This provides deterministic, bounded latency, but introduces severe output degradation and reasoning breakdown when tokens are dropped.
• Dynamic Padding (No Dropping): Sizes GPU GEMM buffers to accommodate the maximum observed expert token count in the current batch, padding other buffers with zeros. This preserves 100% output quality, but results in variable tail latency because the straggler GPU dictates the step execution time.
• Expert Parallelism Load Balancer (EPLB): Continuously monitors expert activation statistics over a sliding time window (such as 1,000 steps) and replicates the top hottest experts across multiple EP ranks. This reduces peak expert token count by 30% to 50%, evening out step latency with zero quality loss while requiring only 2 GB to 5 GB of VRAM per rank for redundant weights.
• Metro Algorithm (Activated Expert Balancing): For memory-bound decode phases, routes tokens to minimize the total number of distinct active experts per GPU rather than pure token counts, maximizing weight reuse in SRAM. This cuts decode step latency by 11% to 22% on memory-bound workloads without quality degradation.

Production serving stacks, including vLLM EPLB module and recent routing research, strongly prefer EPLB dynamic replication over token dropping. In production deployments of DeepSeek-V3, configuring 2 to 4 redundant expert slots per EP rank consumes less than 5% additional VRAM while recovering up to 35% throughput lost to routing skew.

5. Serving Economics: Sizing Infrastructure for MoE Deployments

Evaluating the infrastructure cost of MoE models requires balancing VRAM hardware minimums against concurrent throughput capacity.

The Minimum Node Ceiling

A 671B model in FP8 precision requires roughly 625 GB of raw weight storage. Adding 100 GB to 200 GB for multi-tenant KV caches, communication buffers, and CUDA runtime states establishes a strict baseline: a minimum of 800 GB of aggregate VRAM is required simply to load the model into memory.

This mandates deploying at least:

• 8x H100 or H200 (80GB or 141GB) GPUs on a single node, or
• 4x B200 (192GB) GPUs.

At standard cloud rates of approximately $2.00 to $3.50 per H100 GPU-hour, a single 8-GPU serving node costs roughly $16.00 to $28.00 per hour ($11,500 to $20,000 per month), regardless of whether traffic is incoming.

Throughput vs. Cost Curves

Cost per 1M Tokens Generated ($)
 ^
 |  [Low QPS Regime]
 |  |\
 |  | \  MoE Idle Overhead
 |  |  \
 |  |   \
 |  |    \--------------------+
 |  |                         |  [High Concurrency Regime]
 |  |       Dense 70B Model   |  MoE Outperforms Dense
 |  +-------------------------+---------------------------->
 0                             Concurrent Batch Size (Tokens)

1. At Low Request Concurrency (fewer than 16 concurrent streams): Dense 70B models or quantized small language models achieve significantly lower cost per token. A dense 70B model can run on a single 80GB GPU with INT4 or FP8 quantization, keeping baseline infrastructure spend low. An idle or low-traffic MoE node incurs the full 8-GPU infrastructure bill while processing few tokens.
2. At High Request Concurrency (greater than 128 concurrent streams): The economic advantage reverses dramatically. Because an MoE model executes only 37B active FLOPs per token while utilizing the compute and memory bandwidth of an entire 8-GPU node, it sustains aggregate generation speeds of 2,000 to 4,000 tokens per second per node. Under full saturation, MoE models cut the effective cost per million generated tokens by 2.5x to 4x compared to running equivalent-capacity dense models.

6. Production Implementation Guidelines

Engineering teams deploying MoE models in production should enforce the following operational checklist:

1. Adopt Data-Parallel Attention with Wide-EP: Avoid pure Tensor Parallelism for large expert counts. Configure DP attention across ranks to maximize per-GPU KV cache allocation and eliminate attention All-Reduce collective operations.
2. Enable Dynamic Load Balancing (EPLB): Set sliding window metrics (window size ~1,000 requests) and dedicate 2 to 4 redundant expert slots per EP rank to eliminate straggler stalls caused by domain-specific prompt routing.
3. Deploy Overlapped Communication (DeepEP and DBO): Ensure serving engines utilize dedicated SM-bypass kernels for NVLink and asynchronous RDMA pipelines for inter-node InfiniBand fabrics.
4. Quantize Weights to FP8 or NVFP4: Utilize block-quantized FP8 (such as 1x128 block scaling) for expert weights to fit 600B+ checkpoints within a single 8-GPU node while maintaining reasoning fidelity.
5. Route Concurrency via Front-End Gateways: Front-end model gateways must buffer or batch incoming requests to keep MoE instances operating in the compute-bound high-concurrency regime, preventing idle memory-bandwidth waste.

Sources

• DeepSeek-V3 Technical Report (arXiv:2412.19437)
• vLLM Expert Parallel Deployment Documentation
• vLLM Blog: Large Scale Serving DeepSeek with Wide-EP
• SGLang Distributed Expert Parallelism Architecture
• Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens (arXiv:2512.09277)
• Mixtral of Experts (arXiv:2401.04088)
• DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Light-Speed (arXiv:2201.05596)