Last week I pruned 62.5% of gpt-oss-120b's experts to fit reach.jobs's production resume/job scorer on a single 32 GB GPU at full-model quality, and ended with an honest admission: the 16 GB tier remains unsolved. Every deep cut I tried broke the output format. This is the sequel, and the answer turned out to be a different tool entirely. Instead of cutting the teacher down, I distilled it: a QLoRA fine-tune of Qwen3-8B on 15.8k of the 120b's own stored production outputs, trained in 14.3 hours on one RTX 4080 Super. On the same frozen 311-prompt replay it beats the pruned 120b on every metric, at 5x the speed, and it has now replaced it as the live shadow model in production.
The usual distillation workflow starts with an expensive step: run the teacher over a
corpus and collect outputs. But a production scoring system has been doing exactly that for
months, for free. Every FAST-tier score in the database is a stored
(prompt inputs, strict-JSON output) pair from gpt-oss-120b: the
scores themselves plus a rationale object with the reasoning, key matches, gaps, risk, and
location fields. The dataset builder reconstructs the exact production prompt for each row,
scrubs resume PII, rebuilds the JSON object the system prompt demands from the stored
columns, and excludes the 311 frozen eval prompts. Selection: all ~7k apply/review
decisions (positives are scarce), 12k sampled skips, and every row a human later voted on.
15,810 training pairs, 322 validation, p99 prompt length 6.4k tokens.
Two properties of this data matter. It is exactly on-distribution: same system prompt, same JSON schema, same score semantics the serving path expects. And the targets are direct-JSON with no chain-of-thought, so the student learns the fast serving mode natively: no reasoning tokens to generate, none to pay for.
The recipe is deliberately boring: QLoRA rank 32 on the 4-bit base via unsloth, 8192 sequence length, loss masked to the assistant response, 2 epochs, cosine LR 2e-4, bf16, 8-bit Adam. It fits in 16 GB with room to spare and finishes overnight.
prediction_step materializes
full-vocabulary fp32 logits during in-training eval even with
prediction_loss_only=True. At 8k sequence length that is an instant OOM on a
16 GB card, and because the eval at step N runs before the checkpoint save at step
N, it took two hours of training down with it. I disabled in-training eval entirely
(eval_strategy="no"); the metric that matters is the offline replay anyway.
If you keep eval, save first.On the frozen replay the distilled 8B scores Cohen's kappa 0.642 against stored production decisions. The pruned 120b scored 0.508-0.538. A fresh run of the full, unpruned 120b scores 0.485-0.527. The student appears to beat the model that generated its own training data, which should set off alarm bells, so here is the honest mechanism.
The stored production outputs were generated with research and exemplar context that a replay cannot reconstruct: whatever retrieval and history the live system attached at scoring time. A fresh 120b call sees only the bare prompt, so it agrees with its own stored decisions at kappa ~0.5. The student was trained on the stored outputs themselves, so it absorbed the average effect of all that invisible context into its weights. It is not smarter than the 120b; it is a better predictor of what production actually decided, which, for a shadow model whose job is to reproduce production behavior on cheaper hardware, is precisely the objective.
| model | kappa vs prod | agree vs 120b | sem MAE vs 120b | s/prompt |
|---|---|---|---|---|
| gpt-oss-120b, fresh-run ceiling | 0.485-0.527 | 0.890 (self) | 0.059 | |
| reap-48 pruned 120b (B70 / 5090) | 0.508 / 0.538 | 0.835 / 0.823 | 0.105 / 0.101 | 7.2 (B70) |
| gpt-oss-20b stock | 0.456 | 0.781 | 0.125 | |
| qwen3-8b distilled (4080S, 4-bit) | 0.642 | 0.842 | 0.081 | 1.4 |
Parse rate is 100% strict JSON across all 311 prompts, with no format guardrails beyond the fine-tune itself. The agreement-vs-120b column is the cleaner comparison (same bare replay conditions for everyone), and there the student still leads the pruned model while staying under the full model's 0.890 self-agreement ceiling, which is the sanity check that nothing pathological is going on.
Offline replays are a controlled lab. The real test is production shadow traffic: every FAST score fires an async duplicate call to the candidate on the identical prompt and logs both outputs. The pruned 120b ran as the shadow for ~46 hours; the distilled 8B took over the same stream immediately after. First ~13 hours, 1,906 pairs:
| vs primary 120b | reap-48 (n=11,653) | qwen3-8b distilled (n=1,906) |
|---|---|---|
| semantic delta, mean / median | 0.108 / 0.060 | 0.094 / 0.050 |
| within 0.1 / within 0.2 | 65.4% / 83.5% | 73.0% / 85.6% |
| confidence MAE | 0.180 | 0.079 |
| risk MAE | 0.229 | 0.161 |
| location_ok agreement | 93.4% | 93.6% |
The pattern is not random improvement. Confidence and risk were exactly the two fields where the pruned model drifted most from the full teacher (pruning preserved the primary semantic judgment better than the calibration-flavored side channels), and they are exactly where distillation on real stored outputs helps most: confidence error is cut by more than half, risk error by 30%. The student also serves dense bf16 on the Intel Arc B70 with a third of the card left over, where the pruned 120b needed all 32 GB at 4-bit.
The repo (Apache 2.0) has the dataset builder (selection and target-reconstruction logic), the unsloth trainer with the OOM workaround, merge and serve recipes for both CUDA bnb-4bit and Intel XPU bf16, the frozen-replay eval harness, the shadow-traffic aggregator, and the eval results. Two things are absent on purpose. The data: real user scoring records, scrubbed or not, stay private. And unlike last week, no weights: reap-48 could be published because pruning a public model touches no user data, but this model was trained on user resumes and stored decisions, and an SFT'd 8B can memorize its training set. That asymmetry (pruned checkpoint publishable, distilled checkpoint not) is worth internalizing if you are choosing between the two techniques for your own stack.
Everything here is one task and one distribution: reach.jobs scoring with strict-JSON output. The 8B is a specialist; the honest expectation is that its general capability is whatever Qwen3-8B had, minus whatever the fine-tune displaced, and I have not measured it. The kappa-vs-production comparison favors the student by construction (it trained on production's outputs; the replayed teachers did not have their original context), which is why I lean on the agreement-vs-reference column and the live shadow numbers. The shadow sample is 13 hours against reap-48's 46; time-of-day and job-mix effects are not fully averaged out, though every metric moved the same direction. And n=311 on the frozen replay means the defensible claim is "clearly at least as good, dramatically cheaper," not a precise ranking.
The larger point survives all the caveats: if you run an expensive model in production and store its outputs, you are already sitting on a distillation dataset that is perfectly on-distribution for your workload. A weekend, one consumer GPU, and a boring QLoRA recipe turned mine into a model that tracks production more faithfully than a pruned version of the teacher itself, at 5x the speed, on hardware a fifth the size. The teacher never left production while it happened.