Mixture of Experts (MoE) Explained: How AI Models Use Expert Routing to Scale

Mixture-of-Experts (MoE) is a transformer design that swaps one big feed-forward network for many smaller "expert" networks plus a router that picks a handful of them for each token — so the model's total parameter count can grow into the hundreds of billions while the compute spent on any single token stays close to that of a much smaller dense model. 🧠

Why does this matter outside a research paper? Because it's the design choice behind most of the largest deployed language models today. Google's Switch Transformer, Mistral's Mixtral, Databricks' DBRX, and DeepSeek's V3/R1 line all lean on MoE to add capacity without a proportional jump in the GPU-hours (and dollars) needed to serve every request. Get the routing, load-balancing, or serving strategy wrong, and a team can end up with a model that trains unevenly, ships an expert that barely gets used, or blows its latency budget in production — so understanding the mechanics isn't optional for anyone building or buying one of these systems. ⚙️

A dense FFN block (left) fires every parameter for every token. A sparse MoE block (right) routes each token to only a small subset of experts.

📑 In This Post

• What a Mixture-of-Experts Model Actually Is
• The Router: How a Token Finds Its Experts
• Sparse Activation: Total Parameters vs. Active Parameters
• Load Balancing: Keeping Every Expert Busy
• Fine-Grained and Shared Experts
• Rolling Out MoE Models at Enterprise Scale
• Common Mistakes
• FAQ
• References & Further Reading
• Summary

🔀 Quick Comparison

Dimension	Dense Transformer	Sparse MoE Transformer
Parameters used per token	100% of the FFN weights	Only the top-k experts (often 2–9 of 8–256+)
Total capacity	Bounded by what one dense forward pass can afford	Can scale into hundreds of billions of stored parameters
New failure mode	Compute-bound scaling limits	Routing collapse, expert starvation, load imbalance
Memory footprint	Matches active parameter count	Must hold ALL experts in memory, even unused ones
Serving complexity	Straightforward tensor/pipeline parallelism	Needs expert parallelism and traffic-aware routing

1. What a Mixture-of-Experts Model Actually Is

🧒 Kid analogy: imagine a school where every homework question — math, spelling, history — got sent to the exact same one teacher who had to know everything. That teacher would be slow and stretched thin. Now imagine a school with a front-desk clerk who reads each question and sends it to the two best specialist teachers for that subject. The school as a whole "knows" far more, but no single question ever needs the whole staff — just the couple of teachers suited to it. That front desk is the router; the specialist teachers are the experts.

Technically, an MoE layer replaces the single feed-forward network (FFN) found in a standard transformer block with a bank of parallel FFNs — the "experts" — plus a small trainable routing network (the "gate"). For every token at that layer, the router scores each expert and selects a small subset (commonly the top-1 or top-2, though newer designs select more) to actually process the token. Attention layers are typically left dense and shared; it is almost always the FFN sub-layer that gets "MoE-ified," because the FFN is where most of a transformer's parameters live.

✅ Worked example — Mixtral 8x7B (Mistral AI). Mixtral keeps the same overall architecture as the dense Mistral 7B model, but at every layer the single FFN block becomes 8 FFN "experts." A router chooses the top 2 for each token. Mixtral has the same architecture as Mistral 7B, except each layer is composed of 8 feedforward blocks, and for every token a router network selects two experts to process the current state and combine their outputs. Each token can technically reach 47B parameters spread across the experts, but only about 13B of those are actually used on any given forward pass, which is why Mixtral runs about as fast as a ~13B dense model while drawing on far more stored knowledge.

🎯 Use this when: you need to explain, in one paragraph, why "parameter count" alone is a misleading way to compare model sizes once MoE is involved.

2. The Router: How a Token Finds Its Experts

🧒 Kid analogy: picture a raffle where every ticket (token) gets a score against every prize booth (expert), and only the booths with the top scores actually get to hand out a prize for that ticket. The scoring isn't random — the raffle organizer (the router) has learned, from experience, which booths tend to be good at which kinds of tickets.

Mechanically, the router is a small learned linear layer followed by a softmax over the number of experts, producing a probability-like score for each one. A gating function then keeps only the top-k scores (this is the "top-k routing" you'll see in papers) and zeroes out the rest. The token's representation is passed through each of the selected experts, and their outputs are combined — usually a weighted sum using the router's own scores as weights, so an expert the router was more confident about contributes more to the final output.

The router scores every expert, keeps the top-k, and blends their outputs by that score.

✅ Worked example — Google's Switch Transformer. Earlier MoE work generally routed each token to more than one expert to keep the routing decision differentiable enough to train well. Google's Switch Transformer simplified this to top-1 routing — a single expert per token — and showed the model still trained stably at enormous scale. The researchers simplified the routing algorithm to combine data, model, and expert-parallelism, enabling a model with an outrageous number of parameters while achieving a four-times pretraining speedup over a strongly tuned T5-XXL baseline. The largest version, the Switch-C transformer, used 15 switch blocks, each with 32 attention heads and 2,048 experts, scaling the model into the trillion-parameter range while the compute per token stayed close to a much smaller dense model.

💡 Contrast — more experts per token isn't automatically better. Top-1 routing (Switch Transformer) is cheap and stable but gives the router less room to "hedge" between two plausible specialists; top-2 routing (Mixtral, most Mixtral-style designs) costs a bit more compute per token but tends to smooth out routing decisions the model is less confident about. Neither is universally correct — it's a tradeoff between routing simplicity/training stability and per-token expressiveness that a team has to make deliberately, not by default.

🎯 Use this when: someone asks "why doesn't the model just always pick the single best expert?" — the answer is stability and expressiveness, not just cost.

3. Sparse Activation: Total Parameters vs. Active Parameters

🧒 Kid analogy: a public library can own two million books (its total "capacity"), but you only carry the three or four you actually need off the shelf for tonight's homework (your "active" load). The library's size and your backpack's weight are two completely different numbers — and MoE models make that same split explicit.

This is the core MoE tradeoff: total parameters (everything stored on disk/in GPU memory, including every expert whether or not it's used for a given token) versus active parameters (what actually participates in the matrix multiplications for that specific token). A dense model has no such split — total and active are the same number. An MoE model can have a total parameter count many times larger than its active count, which is what lets it "know more" without a proportionally larger compute bill per token — though, importantly, it still needs enough memory (or fast enough offloading/networking) to hold every expert, since routing decisions change token to token.

✅ Worked example — DeepSeek-V3. DeepSeek-V3 is a Mixture-of-Experts model with 671B total parameters, of which 37B are activated for each token, built on Multi-head Latent Attention and the DeepSeekMoE architecture. That's roughly a 5% "active slice" of the full model per token — the rest of the 671B stays resident in memory but idle for that particular token, ready for the next one that needs a different combination of experts.

🎯 Use this when: a stakeholder asks "how big is this model?" — the honest answer needs both numbers, not just one.

4. Load Balancing: Keeping Every Expert Busy

🧒 Kid analogy: if the front-desk clerk from our earlier example always sends questions to the same two favorite teachers because they happened to look good on the first few questions, those two get overwhelmed and everyone else's expertise goes to waste — and worse, the favorites never get a fair shot at every subject, so their teaching stays shallow. A good clerk deliberately spreads questions around so every teacher gets enough practice to actually become good at something.

Left unchecked, a trained router tends to collapse onto a small favorite subset of experts early in training, because those experts get more gradient updates, which makes them better, which makes the router pick them even more — a rich-get-richer spiral called routing collapse. Two main fixes have emerged. The classic approach adds an auxiliary load-balancing loss term during training that penalizes uneven expert usage across a batch, nudging the router toward a more even distribution. A newer approach removes that extra loss term entirely and instead adjusts a per-expert bias directly based on recent load, avoiding the tug-of-war between the main training objective and the balancing objective.

1. Set a capacity per expert. Each expert is only allowed to process a fixed number of tokens per batch (its "capacity"); tokens beyond that either get dropped or overflow, which is itself a modeling and infrastructure decision.
2. Score routing evenness during training. A load-balancing signal — either an auxiliary loss or a bias adjustment — measures how far actual expert usage is from the ideal even split.
3. Push the router toward balance. The training update nudges the gate's weights (or the per-expert bias term) so underused experts become more attractive next time similar tokens show up.
4. Monitor in production, not just training. Real traffic distributions shift after launch, so expert-utilization dashboards matter after deployment too, not only during pretraining.

✅ Worked example — DeepSeek-V3's auxiliary-loss-free balancing. DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing alongside a multi-token prediction training objective, an approach the team carried forward from ideas validated in DeepSeek-V2. Instead of fighting the main language-modeling loss with a separate balancing loss, the router's per-expert bias is adjusted directly based on how overloaded or underloaded each expert has recently been — reported by the team to reduce the quality tax that traditional auxiliary losses tend to impose.

💡 Warning — capacity limits create silent drops. If an expert hits its per-batch capacity, the tokens that "lose" the tie-break for that expert's remaining slots are typically dropped or routed to a fallback, meaning that token effectively skipped that layer's specialization. This is invisible in most quality metrics unless someone is specifically tracking a token-drop rate, which is exactly the kind of first-class monitoring signal a team can forget to build.

🎯 Use this when: you're diagnosing why an MoE model trains unevenly or why a few experts seem to dominate every response.

5. Fine-Grained and Shared Experts

🧒 Kid analogy: would you rather have 8 generalist tutors, or 64 narrower specialists plus one "homeroom teacher" who covers the basics everyone needs (grammar, arithmetic) so the specialists don't each have to re-learn them? Splitting experts into smaller, more numerous pieces — and keeping one or two "always-on" shared experts for common knowledge — is a deliberate design lever, not an accident.

"Fine-grained" MoE means slicing each expert into several smaller experts (while activating proportionally more of them), which increases the number of possible expert combinations the router can choose from without changing the active-parameter budget. A related idea is the shared expert: one or more experts that process every token regardless of what the router decides, intended to absorb general-purpose knowledge so the routed experts can specialize more cleanly instead of each re-learning the basics.

✅ Worked example — Databricks DBRX vs. Mixtral. DBRX uses a fine-grained MoE architecture with 132B total parameters, of which 36B are active on any input, built from 16 experts of which 4 are chosen per token — compared to Mixtral and Grok-1, which use 8 experts and choose 2. Databricks reports this gives roughly 65 times more possible expert combinations than the coarser 8-expert designs, which the team found improved model quality at a similar active-parameter budget.

✅ Worked example — DeepSeek-V3's shared expert. DeepSeek-V3 carries this a step further: it routes among 256 specialized experts (selecting a subset per token) while also running 1 shared expert on every token, so general-purpose patterns don't have to be independently re-discovered inside every routed specialist.

🎯 Use this when: you're deciding between "fewer, bigger experts" and "more, smaller experts" for a new architecture — the granularity itself is a tunable design choice, not a fixed property of MoE.

6. Rolling Out MoE Models at Enterprise Scale

MoE's efficiency story changes shape the moment a model leaves the research cluster and has to serve real, bursty production traffic. A few operational realities matter here that don't show up in a benchmark table:

• Memory footprint, not just FLOPs. Serving infrastructure has to hold every expert in fast memory even though only a few fire per token — a 671B-parameter model needs 671B parameters' worth of memory (or careful offloading/quantization), regardless of the 37B active figure.
• Expert parallelism. Because different tokens route to different experts, production serving stacks typically shard experts across devices ("expert parallelism") alongside standard tensor and pipeline parallelism, and route tokens across the network to whichever device holds the chosen expert — adding a genuinely new class of inter-device communication cost that dense models don't have.
• Batch composition and latency variance. If a batch's tokens route unevenly, some devices sit idle while others are saturated, which can make MoE inference latency less predictable than a dense model's unless the serving system actively load-balances traffic across expert shards.
• Governance over routing changes. Any change to router weights, expert count, or capacity factor is effectively an architecture change, not a config tweak — it deserves the same regression testing and staged rollout discipline as a full model swap, including before/after checks on expert utilization and per-category quality.
• Cost accounting. Because "37B active" undersells the actual GPU memory and networking bill, procurement and capacity-planning conversations need both the active and total parameter numbers, not just the marketing-friendly active figure.

💡 Warning — DBRX's own release notes underline the operational cost. Databricks' team described building a repeatable MoE training and serving pipeline as one of the harder engineering problems in the project, not just a modeling one — reinforcing that the infrastructure discipline around routing, parallelism, and monitoring is a first-class part of adopting MoE, not an afterthought bolted on post-launch.

🎯 Use this when: a leadership team is comparing "cost per token" across a dense and an MoE candidate model and only looking at the active-parameter number.

7. Common Mistakes

• Quoting only the active-parameter count as "the model's size." Total parameters drive memory, storage, and often licensing/cost conversations; active parameters drive per-token compute. Reporting only one number — usually the smaller, more flattering active count — misleads anyone doing capacity planning.
• Assuming more experts always means better quality. Granularity (DBRX's 16-of-16 vs. Mixtral's 8-of-8) is a real design lever, but it interacts with training stability, routing collapse risk, and communication overhead — cranking expert count up without also revisiting load balancing and capacity factors can hurt more than help.
• Treating load balancing as a training-time-only concern. A router that was balanced on the pretraining data distribution can drift once real user traffic (which looks nothing like the training corpus) starts flowing through it — expert utilization needs to be a live production metric, not just a pretraining checkpoint chart.
• Ignoring token drops from capacity limits. When an expert's per-batch capacity is exceeded, overflow tokens are dropped or rerouted — a silent quality tax that won't show up unless someone explicitly tracks a drop rate.
• Underestimating memory requirements because "only X billion parameters are active." Every expert still has to live somewhere in fast memory since routing changes token by token — sizing hardware off the active-parameter figure alone leads to under-provisioned deployments.
• Shipping router or capacity-factor changes without regression testing. Because routing decisions are learned and data-dependent, even a small change to the gating network or expert count can shift which experts specialize in what — this deserves the same before/after evaluation rigor as swapping the base model entirely.

❓ FAQ

Is Mixture-of-Experts the same as ensembling several separate models?

No. An ensemble typically runs several complete models and combines their outputs after the fact. MoE experts sit inside a single model's layers, share the same attention mechanism and training run, and only a subset activates per token — it's one integrated model with conditional computation, not several independent models voting.

Does an MoE model run faster than a dense model with the same total parameter count?

Yes, for the compute (FLOPs) side, because only the active experts do matrix multiplications for a given token. But raw wall-clock latency also depends on memory bandwidth, expert-parallel communication, and how evenly a batch's tokens route — so an MoE model isn't automatically faster in every serving setup, especially under memory or network constraints.

Why don't all MoE models just route every token to every expert?

That would defeat the purpose — it would be a dense model with extra steps, since every parameter would fire for every token again. The entire efficiency gain of MoE comes specifically from sparse (partial) activation via the router.

What is "routing collapse" and why is it dangerous?

It's when the router repeatedly favors the same small set of experts because they received more gradient updates early in training, making them disproportionately better and even more likely to be picked — a self-reinforcing spiral that leaves many experts undertrained and wastes the extra capacity MoE was supposed to add.

Do shared experts defeat the purpose of sparsity?

Not really — a shared expert is usually just one (or a few) small, always-on experts among many routed ones, so it adds only a modest, fixed compute cost while letting the routed experts specialize more cleanly instead of re-learning general-purpose patterns independently.

🔗 References & Further Reading

• Fedus, Zoph & Shazeer, "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity," JMLR — jmlr.org/papers/volume23/21-0998/21-0998.pdf
• Jiang et al., "Mixtral of Experts," arXiv — arxiv.org/abs/2401.04088
• DeepSeek-AI, "DeepSeek-V3 Technical Report," arXiv — arxiv.org/abs/2412.19437
• DeepSeek-AI, official DeepSeek-V3 model repository — github.com/deepseek-ai/DeepSeek-V3
• Databricks, "Introducing DBRX: A New State-of-the-Art Open LLM" (official Databricks blog) — databricks.com/blog/introducing-dbrx-new-state-art-open-llm
• Hugging Face, Switch Transformers model documentation — huggingface.co/docs/transformers/en/model_doc/switch_transformers

All product and model names (Switch Transformer, Mixtral, DBRX, DeepSeek-V3, and others) are trademarks of their respective owners and are referenced here for identification and educational purposes only. 

📝 Summary

• MoE replaces a transformer's dense FFN with many smaller expert FFNs plus a learned router.
• The router scores every expert and activates only the top-k for each token (Switch Transformer: top-1; Mixtral: top-2).
• Sparse activation splits "total parameters" from "active parameters" — MoE models can be far larger than what they actually compute per token (DeepSeek-V3: 671B total, 37B active).
• Load balancing — via an auxiliary loss or an auxiliary-loss-free bias adjustment — keeps the router from collapsing onto a favorite handful of experts.
• Fine-grained routing and shared experts (DBRX, DeepSeek-V3) let teams tune specialization independently of the active-parameter budget.
• Enterprise rollout brings new operational realities — memory footprint, expert parallelism, latency variance, and routing-change governance — that don't show up in a parameter count alone.
• The most common mistakes all trace back to treating "active parameters" as the whole story instead of one half of it.

That's the architecture, end to end — from one token walking into a router, to a trillion-parameter model humming along in production. Happy building!