Deleting 62.5% of gpt-oss-120b's Experts: Full-Model Quality on a Single 32 GB GPU

Bryan Vine · August 2026 · code: github.com/bryanvine/gpt-oss-subset · weights: gpt-oss-120b-reap-48

My company, reach.jobs, scores resumes against job descriptions with openai/gpt-oss-120b behind vLLM: strict-JSON output, 6-10k token prompts, real hiring decisions downstream. At ~61 GiB of weights the full model needs shared server hardware. I wanted the exact same scorer on a single consumer-class card, so I removed 80 of the 128 experts in every layer and published, as far as I can tell, the first expert-pruned gpt-oss-120b checkpoint: 36 layers, 48 experts per layer, ~45B total parameters, 26.4 GB on disk, serving a 128k context window in stock vLLM on one 32 GB GPU, on both CUDA and Intel XPU.

On a frozen 311-prompt replay of the production scoring workload it matches the model it was cut from. Cohen's kappa against stored production decisions: 0.508 (Intel Arc B70, XPU) and 0.538 (RTX 5090, CUDA), bracketing the full model's own run-to-run range of 0.485-0.527, at a 100% strict-JSON parse rate on both backends. The result took two attempts, and the failed one is the finding I most want other practitioners to take away.

Why pruning, not offloading

Three ways to fit a 61 GiB MoE on a 32 GB card. CPU offload (llama.cpp --n-cpu-moe) works but my best measured config costs ~46 seconds of prefill on an 8k prompt: a fallback, not a service. Dynamic expert paging dies on arithmetic: decode touches 4 experts x 36 layers = 144 experts per token, ~1.83 GB of weights with essentially no cache locality, which caps out around 14 tok/s on PCIe 4.0 before any compute happens. Deleting experts, though, produces just a smaller checkpoint: no custom kernels, no runtime, no offload policy, and it serves in stock vLLM on any vendor's backend. gpt-oss makes this unusually clean, because the router softmaxes over only the selected top-4 logits, so survivors renormalize among themselves with no router surgery and no retraining.

The cost is proving the smaller model still does the job. Prior art (Cerebras' REAP, EASY-EP) de-risks removing 25-50% of experts. I wanted 62.5%, which nobody had validated, on a model nobody had pruned.

The instructive failure: prompt-only calibration deletes the experts that write

The obvious pipeline is: run llama-imatrix over a calibration corpus of real prompts, rank experts by routing frequency, keep the top 48 per layer. It looks great on paper: my corpus showed top-48 covering 89.1% of routing mass. It failed in three escalating ways. Uniform-48 scored far below gpt-oss-20b at twice the memory (0.641 agreement with the full model against a 0.890 self-agreement ceiling, 87% parse). Uniform-32 broke the harmony output format outright. And the variable-budget variant that cut late layers hardest produced fluent, knowledgeable generations with a 0% parse rate: it hadn't lost knowledge, it had lost a format.

That symptom is the diagnosis. A prompt-only corpus records which experts fire while the model reads, and nothing about which fire while it writes: harmony channel markers, analysis-channel reasoning, JSON emission. The experts serving output format were invisible to the ranking and got deleted, worst in late layers, exactly where prompt-token routing looked most concentrated. The fix is cheap: replay your own model over the calibration prompts, keep the generation traces, and calibrate on prompt-plus-generation. I also switched the ranking criterion from routing frequency to REAP saliency (router gate weight times the L2 norm of the expert's output). The two changes together produce keep-sets that differ from the frequency-based ones by 60% of their membership, with the disagreement concentrated in exactly the late layers where the format failures appeared.

If you take one thing from this: routing-mass coverage arguments ("the top-N experts carry X% of the mass, so pruning is safe") use the wrong distribution. Under REAP saliency the same top-48 carry only 56% of the mass. And a model pruned on prompt-only statistics fails by format collapse, not gradual quality decay, so it can look fine right up until every response is unparseable.

Results

model	parse	agree vs full-120b	kappa vs prod
gpt-oss-120b, self-agreement ceiling	100%	0.890	0.485
gpt-oss-120b, reference run	99.7%	(reference)	0.527
reap-48 (Arc B70, XPU)	100%	0.835	0.508
reap-48 (RTX 5090, CUDA)	100%	0.823	0.538
reap-v2, variable budgets (B70)	100%	0.826	0.488
gpt-oss-20b (B70)	100%	0.781	0.456
subset-48, frequency-ranked (B70)	87%	0.641	0.149
subset-32 / subset-v90, frequency-ranked	format collapse		n/a

The honest bar in that table is the top row: run the full model twice through the identical harness and it agrees with itself on only 89.0% of decisions (kappa 0.776), because temperature-0 kernels are not deterministic and borderline cases flip. Any pruning result that reports agreement-with-the-original without agreement-of-the-original-with-itself is reporting an unknown mixture of quality loss and noise. Against that ceiling, reap-48's 0.835 is near-parity, and its kappa against real stored production decisions lands inside the full model's own self-consistency range on both backends. Two more replication checks: the same checkpoint serves on stock CUDA vLLM and stock Intel XPU vLLM with matching results, and the saliency computation re-run on a different host and GPU generation produces 99.2% identical keep-sets (rank correlation 0.9973).

What's in the repo

Everything is Apache 2.0 at github.com/bryanvine/gpt-oss-subset, with a full write-up in docs/PAPER.md:

Artifact	Notes
gpt-oss-120b-reap-48	The recommended checkpoint. Uniform 48/128 experts, MXFP4 preserved, serves in stock vLLM (CUDA and XPU).
`scripts/reap_saliency.py`	Layerwise REAP saliency runner that fits the whole procedure on a 16 GB card: one dequantized decoder layer in VRAM at a time, true sequential residual stream.
`scripts/prune_gptoss.py`	Surgical pruner. Slices MXFP4 blocks and scales as raw bytes, no dequantize/requantize round trip; emits a manifest with the exact kept indices.
`patches/gpt_oss.py`	vLLM patch adding per-layer expert budgets (`num_local_experts_per_layer`): the first way to serve a non-uniformly pruned gpt-oss checkpoint. Variable budgets tied uniform-48 at this compression level, but deeper cuts will need this.
`scripts/eval_scoring.py`	The frozen-replay eval harness (kappa, agreement, parse rate as a first-class metric).

One deployment bonus: EAGLE3 speculative decoding works against the pruned target using the stock, unpruned RedHatAI/gpt-oss-120b-speculator.eagle3 draft, at ~40% acceptance (mean acceptance length ~2.1-2.3, measured on both CUDA and the deployed XPU service). Drafts trained on the full 128-expert model transfer to the 48-expert one without retraining.

Caveats, stated plainly

Every number here is from one task: reach.jobs resume/job scoring with strict-JSON output, calibrated on 60 prompts from that distribution. I make no claim about MMLU, code, math, or multilingual behavior, and the honest expectation is that general capability is reduced; prune with your own traffic using the tooling, not my keep-sets. The fix is also not a clean ablation (I changed corpus and ranking criterion together). And the 16 GB tier remains unsolved: my only deep cuts broke the output format, and REAP's published validation stops at 50% removal. With n=311 the defensible claim is "reap-48 is inside the full model's self-consistency range and clearly ahead of gpt-oss-20b," not that one backend beats the other.

Hardware: the Intel Arc B70 (32 GB) and RTX 4080 Super (16 GB) are mine; the RTX 5090 (32 GB) used for the CUDA replication and saliency cross-check was borrowed for this research, with thanks to its owner. Production now runs reap-48 on the B70 as the reach.jobs quality tier, full 128k context on a single card.