← bryanvine.github.io

The Model That Only Says "!": Root-Causing Silent GPU Corruption in Production

Bryan Vine · August 2026 · issue: vllm-project/vllm#53480 · related: distilling the production scorer

At 11:00 UTC on a Friday, the 27B writing model behind reach.jobs, a W4A16 distill serving on an Intel Arc Pro B70, quietly stopped writing resumes and started answering every request with exclamation marks. Thousands of them. Not an error: HTTP 200, finish_reason: length, a healthy container, a green /v1/models, nothing in dmesg. Every piece of infrastructure that exists to tell you something is wrong said everything was fine, for fourteen hours, while every document the model produced was 4,096 tokens of "!".

This is the story of root-causing it, and it is mostly a story about ruling things out. The suspect list was the classic one: hardware, power, temperature, memory pressure, the quantized GEMM kernel, a build regression. Each got a targeted experiment. What survives all of them is narrower and stranger than any single suspect, and the hunt changed how I monitor GPU inference in general.

Why "!" specifically

The signature is the first clue. In the Qwen vocabulary, "!" is token id 0. A model that emits token 0 forever is not confused; its logits have collapsed to NaN or zero, and argmax over garbage degenerates to index 0. Something numeric is poisoned. And because the corruption survives across requests, but a process restart (which reloads weights) fully cures it, the poisoned thing lives in VRAM for the lifetime of the engine: the weights themselves, or some long-lived workspace. Per-request state is exonerated from the start.

Suspect one: the new card

The box has two identical B70s: one four months old running two smaller distilled scoring models without incident, and one brand-new card running the writer. All three degeneration events had hit the writer's card. Brand-new card, all failures on it: the prior says RMA. But the same card also had an unrelated idle-power quirk that had traced to slot wiring rather than silicon, so instead of shipping the card back I swapped the workloads: writer to the proven card, scoring models to the suspect one.

Then I ran a concurrency sweep on the proven card, partly for throughput numbers and partly as a stress test. At concurrency 12, the "!" corruption reproduced within minutes, on the four-month-old card, with the identical signature: every response garbage, corruption persisting into single-stream probes after the load stopped, zero driver events. Both physical cards produce the same failure. Hardware exonerated, RMA canceled. Power was never a serious suspect after that sweep either: the clean level at c=8 drew 195W and the corrupting level at c=12 drew 199W. No power-delivery story explains a 2% cliff, and no thermal story explains corruption from a cold start in minutes at the same wattage as a clean run.

Suspect two: the quantized GEMM kernel

The model's compute funnels through one custom op: torch.ops._xpu_C.int4_gemm_w4a16, selected by vLLM's XPUwNa16LinearKernel for compressed-tensors W4A16. A batch-dependent out-of-bounds write in that kernel would explain everything: it fires at every layer, a stray write could hit resident weights, and weight corruption is exactly what a restart-cures-it failure looks like.

So I put the kernel on a bench by itself: a standalone harness calling the op directly with the model's real layer shapes (5120x8192, 6144x5120, 5120x34816, 17408x5120, group size 128), sweeping the batch dimension M from 1 through 2048. No server, no scheduler, just the kernel. For every shape and every M: no NaN or Inf, bit-exact determinism across repeated calls, input buffers (packed weights and scales) checksum-identical before and after, and a baseline M=1 output that stayed bit-identical after every larger-M call. The kernel, in isolation, is clean. Whatever this is needs more of the stack.

Suspect three: the build

Here the investigation produced its best-looking wrong answer. The corrupting engine ran one intel/vllm image; the immune scoring models ran another, a month older. Version strings one dev-commit apart. Cross-testing was cheap, so I ran the full 2x2: the immune 14B model on the "bad" image stayed clean at c=12, and the corrupting 27B on the "good" image stayed clean at c=12. For an hour the story was a tidy build regression, with a faster older image as a free bonus.

Then I diffed the two images' vLLM trees, expecting a month of drift, and found they are byte-identical except for the version file and one API-middleware helper that the inference engine never imports. Same torch, same triton, same kernels package, same oneAPI, to the byte. There is no code difference that can explain corrupt-here, clean-there. The build-regression theory, which the cross-test seemed to confirm, is dead on arrival at the file level.

What replication attempts actually showed

That contradiction forced the honest experiment: replicate the corrupting cell exactly. Same image, same overrides, same flags, same gpu-memory-utilization 0.97, same concurrency ramp, same card. Result: clean, five levels, fifteen minutes. Seven isolation cells in total, spanning both images, utilizations 0.5 through 0.97, cold starts and ramps, even a synthetic mix of 7,000-token prompts (multi-chunk prefills) over twelve decode streams. All clean:

The one cell that ever corrupted was the production engine while real traffic (6.7k to 7.8k-token prompts, confirmed in the usage logs, landing mid-sweep) mixed with twelve synthetic decode streams. Production had also corrupted three times at concurrency four or less, but only after hours to days of sustained real load. The picture that fits every observation: an intermittent race in the XPU serving stack whose probability scales steeply with decode batch depth and with the batch compositions real traffic produces, and which on this box only the head_dim-256 27B ever triggers. The head_dim-128 siblings have hammered the same silicon for weeks, including c=12 sweeps, without one bad token. Notably, the kernels package in both images predates the upstream rework of mixed prefill/decode batch attention.

The fix that is also a speedup

You do not need the minimal repro to act. Every clean cell agreed on a posture, so production moved to it: the older image at gpu-memory-utilization 0.90. The older image turned out to be measurably faster on this model at every concurrency, which is the kind of fact you only learn by benchmarking both sides of a bug hunt:

Throughput scales near-linearly to c=12 while end-to-end latency stays flat at about 21 seconds: batching is close to free on this workload. So the production engine, which had been capped at 4 concurrent requests out of caution, now runs 12: a 2.7x capacity raise and a 20% single-stream speedup, out of a corruption investigation. The ten-minute content canary stays, because the honest status of an intermittent race is mitigated, not proven dead.

Caveats, stated plainly

I do not have a deterministic minimal reproducer, and I will not pretend the mitigation is a proof. The old posture corrupted four times in thirty hours of real load; the new posture's evidence is that every controlled cell on it was clean and that it maximizes distance from the one cell that failed. The validation is pre-registered and running: the canary log. A clean week of production traffic at triple the old concurrency is strong Bayesian evidence the operative factors are covered; a single recurrence goes straight into the upstream issue, which has the full matrix, the harness methodology, and a standing offer to run instrumented builds on this hardware.

The transferable lessons cost us fourteen silent hours and are cheap for you: token id 0 on repeat means NaN logits, and it means something persistent in VRAM is poisoned, not a bad request. Health endpoints, HTTP codes, and driver logs will all lie to you through this class of failure; only the model's words tell the truth, so probe the words. And when an intermittent bug hands you a clean A/B (two images, two cards, two models), spend the extra hour on the byte-level diff before you believe the tidy story. Mine died at two files, and what replaced it was the truth.