Mixture of Experts2 articles

Mixture of Experts

Articles

Auxiliary-Loss-Free Load Balancing in Mixture-of-Experts: How Dynamic Bias Adjustments Eliminate Gradient Conflict and Routing Collapse
Sparse Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute cost by activating only a small subset of feed-forward network (FFN) parameters for any given token. While dense transformers evaluate every parameter across all sequence positions, MoE models route tokens dynamically to specialized sub-networks, enabling parameter scaling to hundreds of billions or trillions of parameters at the inference and training cost of much smaller dense models. However, condit
August 21, 20261 min
Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures
Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures Scaling dense Large Language Models (LLMs) requires activating every parameter in the network for every token in a sequence. While techniques like Tensor Parallelism, Pipeline Parallelism, and Fully Sharded Data Parallelism distribute billions of dense parameters across clusters, computational cost scales linearly with parameter count. Mixture-of-Experts (Mo
August 19, 20261 min