Mixture-of-Experts: Efficient Scaling to Trillion-Parameter Models

Abstract

Training and deploying language models with trillions of parameters presents severe computational and memory challenges that limit practical deployment. Dense transformer architectures require activating all parameters for every input token, creating linear scaling of computation with model size. Mixture-of-Experts (MoE) architectures address these limitations through conditional computation: routing each token to a subset of expert networks while keeping most parameters dormant. We present comprehensive analysis of MoE designs spanning sparse gating mechanisms, expert specialization patterns, and training dynamics across models from 1B to 1.6T parameters. Our Switch Transformer architecture achieves 7x speedup compared to dense baselines at equivalent quality by activating only 1/64th of parameters per token. Through systematic investigation of routing algorithms, load balancing strategies, and expert capacity allocation, we identify design principles enabling stable training and effective specialization in trillion-parameter sparse models. We demonstrate that MoE models develop interpretable expert specialization, with different experts capturing distinct linguistic phenomena, semantic domains, and computational primitives. These findings enable practical trillion-parameter models deployable on current hardware, with significant implications for democratizing access to powerful language models.