![\"One](\"/54fe54d20975860a27a656abd38b92b8/moe-routing.svg\")
The first time I looked at deploying a Mixture-of-Experts (MoE) model, the spec sheet confused me. Mixtral 8x7B is described as an 8×7B model that “only activates about 13B parameters per token.” I read that as “runs like a 13B model” and sized a GPU for a 13B model. The weights would not fit.
\nThat gap, between what a MoE model computes and what it costs to hold in memory, is the whole story. It is also the part most summaries skip.
\nThis post is for engineers who are choosing a model to serve, or trying to understand why a MoE model is fast to run but expensive to host. I cover what an expert actually is, how the router picks experts, why compute and memory scale off completely different numbers, and the failure modes that show up once real traffic hits the model.
\nStart with an ordinary transformer block. Attention mixes information across tokens. Then a feed-forward network (FFN) processes each token position on its own. In a dense model, that FFN is a single pair of large matrices, and every token goes through all of it.
\nA Mixture-of-Experts layer replaces that one FFN with several FFNs, called experts, plus a small router that decides which experts handle each token. Each expert has the same shape as the FFN it replaced; there are just more of them. The idea goes back to the sparsely-gated MoE layer from Shazeer et al. (2017). Google’s Switch Transformer scaled it up and cut routing down to a single expert per token to keep things cheap.
\nThe word “expert” oversells it. Nobody assigns Expert 3 to Python and Expert 6 to French. Experts start random and specialize during training, in ways that mostly do not map to anything a human would name. What matters operationally is the count: more experts means more total parameters, and the router means only a few of them run for any given token.
\nFor each token, the router:
\nk highest-scoring experts.Mixtral 8x7B uses eight experts per layer and k=2. Two experts run, the router’s weights combine their outputs, and the other six do nothing for that token.
Here is the routing math with nothing else around it. scores.topk picks each token’s k experts, the softmax turns their scores into weights, and the loop adds up each chosen expert’s output scaled by its weight:
import torch\nimport torch.nn.functional as F\n\ndef moe_layer(x, router, experts, k=2):\n # x: (num_tokens, d_model)\n scores = router(x) # (num_tokens, num_experts)\n topk_w, topk_idx = scores.topk(k, dim=-1)\n topk_w = F.softmax(topk_w, dim=-1) # normalize over the chosen experts\n\n out = torch.zeros_like(x)\n for slot in range(k):\n idx = topk_idx[:, slot] # which expert each token picked\n w = topk_w[:, slot].unsqueeze(-1) # its weight\n for e, expert in enumerate(experts):\n mask = idx == e\n if mask.any():\n out[mask] += w[mask] * expert(x[mask])\n return outThe nested loop is written for clarity, not speed. A real kernel groups tokens by their chosen expert and runs each expert once over its batch, and that is where the engineering effort actually goes. But the shape is right: the softmax picks, the top-k prunes, and the outputs get summed with the router’s weights.
\n
This is the part I got wrong. The two costs scale off different numbers.
\nCompute per token scales with the active parameters, because a token only touches k experts. Mixtral activates roughly 12.9B parameters per token even though the model holds 46.7B in total. A forward pass therefore costs about what a 13B dense model costs, not a 47B one. That is the selling point: you get the quality that comes with a large parameter count while paying a small model’s FLOPs per token.
Memory scales with the total parameters, because the router can send the next token to any expert. You do not know in advance which two of the eight a token will pick, and across a batch every expert gets used. So all of them have to be resident in GPU VRAM. There is no version of this where you keep only the active experts loaded.
\nThat means Mixtral 8x7B needs memory for its full ~47B parameters. At bf16 (two bytes per parameter) that is roughly 94GB of weights before you add anything else, which does not fit on a single 80GB card. You are in multi-GPU territory for a model whose per-token compute looks like a 13B.
\nOn top of the weights you still pay for the KV cache, and MoE does not shrink it at all. Attention is shared, so the KV cache math is the same as for a dense model of the same width and depth.
\nIn short, MoE trades memory for compute: you spend VRAM to buy cheaper tokens. Whether that is a good trade depends entirely on whether you have the VRAM to spend.
\nLeft alone, routers collapse. Early in training a couple of experts get slightly better, so the router sends them more tokens. They get more gradient, so they get better still. After a while most tokens funnel into a small set of experts while the rest sit dead. You paid for eight experts and trained three.
\nThe standard fix is an auxiliary load-balancing loss: an extra training term that penalizes uneven routing and pushes the router to spread tokens across experts. The Switch Transformer paper describes the version most implementations follow.
\nThis mostly matters at training time, but it leaks into serving in one way worth knowing. Even a well-balanced router only balances on average. Any given batch can send an uneven share of tokens to one expert, and that expert becomes the batch’s bottleneck.
\nProduction MoE serving handles this with a capacity factor, a cap on how many tokens one expert will accept per batch. Tokens over the cap get dropped from that expert. (They still pass through via the residual connection, just without expert processing.) Setting the cap is a genuine tradeoff, with no default that is right for every workload:
\nThree things follow once you actually put a MoE model behind traffic.
\nSharding works differently. A dense 47B model splits across GPUs with tensor parallelism (each GPU holds a slice of every weight matrix). A MoE model can also use expert parallelism: put different experts on different GPUs, and route each token to whichever card holds its expert. That cuts per-GPU memory, but it adds a network hop in the middle of the layer and makes throughput sensitive to how evenly tokens land across cards. A skewed batch means one GPU works while the others wait.
\nBatching still helps, but the arithmetic shifts. With continuous batching on a dense model, more concurrent requests keep the matrix multiplies full. On a MoE model those tokens scatter across experts, so a small batch can leave individual experts underfed even while the GPU as a whole looks busy. You often need more concurrency before a MoE model reaches the efficiency a dense model hits sooner.
\nThe sizing rule flips from what the marketing implies. Budget memory for the total parameter count and compute for the active count. If you size the GPU off the active count, the weights will not load, which is exactly the mistake I opened with.
\nMoE is a good deal when you are throughput-bound and have the VRAM: you serve big-model quality at small-model token cost. It is a bad deal when you are memory-bound, because you pay to hold parameters that any single token rarely runs through. For a lot of self-hosted setups on one or two GPUs, a dense model in the same memory budget is simpler, and its latency is easier to reason about.
\nWhen I do run one, the metric I care about is routing balance under real traffic, not the average the training loss optimized. If one expert is consistently hot, latency gets spiky in a way that is hard to trace back. The model is “working”; a subset of it is just overloaded. Per-expert token counts, exported the same way I’d wire up any other service metric into OpenSearch, turn that mystery into a dashboard.
\nMost of the LLM plumbing I built for Archi, the RAG copilot for CMS operations, sits in front of whatever model is behind the endpoint. So for me the MoE-versus-dense choice is a hosting-cost and latency question rather than an application one. But it is the kind of question that decides your GPU bill, and “only activates 13B parameters” is the phrasing that gets people to size it wrong. Read it as: cheap to run, expensive to hold.
\n