Jan 21, 2026 Research
Composing Weight and Data Sparsity in MoE
Sparse weights meet sparse data
This work shows weight sparsity and data sparsity compose: mixture-of-experts models trained on skill-filtered data keep accuracy while cutting active compute on both sides of the equation.
The result is a practical recipe for cheaper frontier-class training.
Two sparsities, one model
Pruned experts plus curated batches multiply: each token touches fewer parameters, and each parameter sees more informative tokens.
No accuracy cliff
Up to the sparsity levels we tested, composed sparsity matches dense baselines on reasoning and grounding suites.
Reproducible recipe
Hyperparameters, pruning schedules, and data mixes are documented end to end for independent replication.