Sparse Mixture of Experts and How It Improves Computational Efficiency in AI

This structure enables models with billions of parameters to reduce pretraining costs and increase inference speed. By limiting active computational power, MoE bridges the gap between the high capacity of massive models and the efficiency of smaller models.

The Evolution of Conditional Computation

While most modern MoE implementations have emerged only recently, the concept dates back to 1991, when research into adaptive mixtures of local experts began. This early work proposed training separate networks for different subsets of training examples, managed by a gating network that assigned tasks to the appropriate expert. These experimental models achieved their accuracy targets in half the time it took conventional dense models.

For years, there were still obstacles to implementation. It wasn’t until 2017 that researchers succeeded in overcoming the algorithmic and performance-related challenges necessary to make “sparse gating” viable on a large scale. This progress coincided with the emergence of generative AI, where models became too computationally intensive for traditional dense architectures.

Dense versus Sparse Architectures

Standard deep learning models are “dense.” They use feed-forward neural networks, in which every single neuron and every connection is utilized to process every single data point. As developers add more parameters to enhance a model’s ability to learn complex patterns, computational costs increase linearly.

MoE replaces these dense layers with sparse blocks. This introduces conditional computation. A router—or gating network—learns a mapping function to determine which experts are most effective for a given token.

In the Mixtral-8x7B model, each layer contains eight feed-forward blocks. For each individual token, the router selects two of these experts to process the data. The results are then combined and passed on to the next layer. This means that the model can have an enormous total number of parameters, while the actual computational cost per token remains relatively low.

Total Parameters versus Active Parameters

There is a difference between the number of sparse parameters and the number of active parameters. The number of sparse parameters represents the model’s total capacity, while active parameters represent the actual computational load during inference.

For example, Mixtral 8x7B has a total of about 47 billion parameters. However, to process a given input, it uses only about 12.9 billion active parameters. Thanks to this efficiency, it outperforms larger, dense models such as LLaMA 2 70B in certain benchmarks while running at higher speeds.

Despite this computational efficiency, memory requirements remain high. All parameters must be loaded into RAM or VRAM, regardless of whether they are active for a specific token. Consequently, the hardware requirements for memory do not decrease simply because the computation is sparse.

Challenges in Routing and Load Balancing

The effectiveness of a MoE model depends on the gating function. If the router is poorly designed, an “expert collapse” can occur. This happens when, at the beginning of training, certain experts are selected more frequently, making them more accurate and causing the router to favor them even more. Other experts remain unused and become dead weight. To prevent this, researchers employ various techniques:

• 1. Noisy Top-k Gating: Adding Gaussian noise to the probability values ensures a more even distribution of work among the experts.
• 2. Regularization: Systems can penalize excessive reliance on a single expert or reward the diverse use of all available experts.
• 3. Random Routing: In some setups, such as GShard, the first expert is selected using a softmax function, while the second is selected partially at random.
• 4. Expert Capacity: Setting a fixed upper limit on how many tokens a single expert can process prevents bottlenecks. If an expert is full, the token is forwarded to the next layer.

Stability in Fine-Tuning and Command Optimization

Fine-tuning MoE models is more complex than fine-tuning dense models. They generally tend to overlearn. Research shows that updating only the MoE parameters can actually degrade performance, as these layers account for only a small portion of the total network depth. Updating non-MoE feed-forward networks often leads to better results.

Instruction optimization has proven particularly beneficial for sparse architectures. Experiments comparing dense T5 models with their MoE counterparts showed that while dense models may perform better after standard fine-tuning, instruction-optimized MoE models significantly outperform their dense counterparts. This suggests that the potential of MoE models can be more effectively harnessed through high-quality instruction sets than through conventional fine-tuning methods.