Mixture of Experts: Sparse Compute, Dense VRAM
Mixture of Experts routes each token to a couple of expert layers, so the model runs cheaper per token yet still needs every expert sitting in GPU memory.
Mixture of Experts routes each token to a couple of expert layers, so the model runs cheaper per token yet still needs every expert sitting in GPU memory.