Mixture of experts

A mixture-of-experts model replaces one large feed-forward layer with several smaller "expert" layers and a router that sends each token to a few of them. The active parameter count per token is lower than the total, so compute is cheaper than a dense model of the same total size. The total parameter count is what you download and what has to be resident in memory.

That distinction causes two common mistakes. The first is dividing the total by the active number and planning hardware as if the model were that small; all the experts still occupy memory, even the ones not used. The second is comparing an active parameter count to a dense model's total and concluding the mixture is smaller. It is not; it is cheaper to run per token.

For sizing, use the total parameter count for weights and the active count for a rough compute estimate. For quality, read the model card's own comparison, not the arithmetic. See VRAM and size a local model.