vLLM 0.21.1.dev's FlashAttention + fp8 KV vs TurboQuant on Arc Pro B70: characterized, then migrated

TL;DR

• Ran the head-to-head, then didn’t wait for a follow-up post to act on it. The plan going in was “characterize, defer the migrate verdict” — benchmark upstream intel/vllm:latest (compressed-tensors W4A16 + fp8 paged KV + the XPU FLASH_ATTN backend) against the production 0.19 + patched-GPTQ + TurboQuant stack on the Arc Pro B70, ship the numbers, decide later. The numbers came in clear enough that “later” happened the same session: production is now running the new stack. The old stack (TQ 0.19) is preserved as a one-command rollback.
• This is a stack-vs-stack comparison, not an isolated FlashAttention microbenchmark. Attention backend, weight quant (GPTQ → W4A16), and KV scheme (TurboQuant → fp8) all changed at once. None of the numbers below isolate cleanly to “FlashAttention did this” — saying so up front, not burying it in the limits section.
• The upstream image is vLLM 0.21.1.dev17+g0a4756bb5, not 0.23. intel/vllm:latest tracks a rolling dev branch; there’s no numbered 0.23 release inside it, despite that being this whole spike’s working title going in.
• Prefill: ~20x faster at 8K context, ~4.7x at 32K (1,929 → 39,430 tok/s @8K; 2,547 → 12,035 tok/s @32K, C=1, no draft either side). Decode: only ~1.5x (8.6–8.8 → 12–13 tok/s) — this MoE’s decode is bandwidth-bound on expert-weight loads, not attention, so a faster attention kernel barely touches it.
• fp8 KV holds ~40% less aggregate cache than TurboQuant (270,592 vs 453,184 tokens, both measured) — that gap, not the context ceiling, is the real bill for the speedup. The single-request ceiling as deployed that day was a conservative 131K; it went back up to 262K within 24 hours once tested (see the Update near the end of this post).
• Neither stack is stock. The old stack needs its patched gptq.py to boot on XPU at all; the “upstream” new stack needed its own bind-mounted marlin_utils.py carve-out, because it routes W4A16 MoE to a CUDA-only Marlin kernel that isn’t compiled for XPU. Both ship on a patch.
• Follow-up — can the upgraded stack run the newer hybrid models (Qwen3.5 / 3.6)? The native XPU Gated-DeltaNet kernel that needed a hand-built image back in April now ships in the box. But both 35B-A3B hybrids get forced onto --enforce-eager (torch.compile hangs on the GDN layers), which roughly halves throughput: Qwen3.5-GPTQ works but is a fragile, verbose thinking model at ~7.3 tok/s C=1 (~½ the 30B), and the one runnable Qwen3.6 checkpoint generates garbage. Kept the 30B. See the Qwen3.5/3.6 follow-up near the end of this post.
• Follow-up — the speculative-decoding call above was wrong; EAGLE-3 now ships. The Update’s “EAGLE3 is a dead end” verdict was a config bug, not a model limit: the draft’s config.json capped max_position_embeddings at 2048, zeroing acceptance above that. One-field fix — then a captured day of real traffic settled it. The workload is ~99% novel structured-JSON output (not the repetitive-input case ngram wins), so EAGLE-3 beats ngram 1.6x @C=1 / 1.4x @C=4 and replaced it in production. See the 2026-07-06 follow-up near the end.
• Fourth post in the Arc B70 series. Previous posts: the first SYCL PoC + 2x Triton fix, three SYCL attempts and the gap to Triton, TurboQuant + EAGLE3, six integration fixes.

Why now: the frontier moved

DeepSeek V4 shipped April 24, 2026, and cut its KV cache to roughly 10% of V3.2’s footprint by training in Compressed Sparse Attention, Heavily Compressed Attention, and DSA — architecture-level attention tricks that make long-context serving dramatically cheaper, if you can run them. This box can’t. V4 is 284B+ parameters, and the fast paths (FlashMLA, FlashInfer) are CUDA-only; none of it touches a 32 GiB Arc Pro B70. So V4 isn’t this post’s subject — it’s the reason to go looking for what frontier move a B70 actually can ride. The answer was sitting in the changelogs: vllm-xpu-kernels shipped a real FlashAttention implementation for Battlemage — XMX/DPAS-targeted kernels for both prefill and paged decode — plus fp8 paged KV, and it landed in upstream intel/vllm. That’s not V4’s 90% KV cut. It’s the actual, runnable-today upgrade available to this card, and it’s what the rest of this post measures.

The two stacks

Three things changed between the stack that was serving production yesterday and the one serving it now:

	Old stack (production until today)	New stack (production now)
Image	vllm-xpu:0.19.0-tr5, self-built, patched gptq.py	intel/vllm:latest = vLLM 0.21.1.dev17+g0a4756bb5
Weight quant	Qwen3-30B-A3B GPTQ-Int4	Qwen3-30B-A3B compressed-tensors W4A16
KV cache	TurboQuant k3v4_nc — 453,184 tokens measured @ 262,144 max-len	fp8 paged — 270,592 tokens measured @ 131,072 max-len
Attention	eager tiled online-softmax, O(N²) prefill wall	upstream XPU FLASH_ATTN backend (FlashAttention v2, “sycl-tla FMHA” kernels)
Speculative decoding	suffix (production-tuned)	none — not integrated
dtype	float16	float16
Load-bearing patch	gptq.py XPU fix + TurboQuant’s monkey-patch/mount layer	marlin_utils.py XPU carve-out

Two details in the right-hand column above went stale within a day: the max-len value in the KV-cache row (131,072 → 262,144; the pool size itself, 270,592, didn’t move) and the speculative-decoding row both changed. See the Update near the end of this post.

Every row moved together — this is a stack-vs-stack comparison, not a controlled FlashAttention-only experiment. The ~20x prefill number a few sections down reflects the attention-backend switch and the quant-format switch and the KV-scheme switch, all landing in the same container image at once. There’s no ablation in this dataset that isolates FlashAttention’s individual contribution, and I’m not going to write around that.

The pre-spike plan also assumed the new stack would land close to stock upstream, with the old stack carrying all the patch debt. It didn’t work out that clean. Getting W4A16 MoE to actually run on XPU needed a source patch of its own (below) — so the honest framing is patch-vs-patch, not patch-vs-stock.

What it took to run upstream on the B70

intel/vllm:latest is not a drop-in swap. Four fixes were needed before it would serve a single token on this box — three Compose/Docker plumbing issues and one source patch.

1. The entrypoint eats a bare command list

intel/vllm:latest’s ENTRYPOINT is ["/bin/bash", "-c", "source /opt/intel/oneapi/setvars.sh --force && exec \"$@\"", "--"] — a bash -c script -- "$@" wrapper. The trailing -- is consumed as $0; whatever command: says becomes "$@" and gets exec‘d verbatim. There is no implicit vllm serve anywhere. A command: of bare flags (--model foo ..., no program name) fails with exec: --: invalid option — confirmed empirically (docker run intel/vllm:latest vllm serve --help reaches vLLM’s own argparse; the bare-flags form doesn’t). Fix: command: starts with vllm, serve.

2. env_file: doesn’t feed Compose’s own ${VAR} interpolation

The compose file’s command: needed "${VLLM023_MODEL}". env_file: only injects variables into the container’s runtime environment; Compose’s own ${VAR} YAML interpolation reads shell env or a .env file in the compose project directory — it never sees env_file: contents. Caught via docker compose config (dry run) before ever booting for real: it rendered --model "". Fix: added deploy/.env alongside the pre-existing deploy/vllm023.env (kept, still used for the container’s own runtime env).

3. group_add: [render, video] fails — the image has no render group

Error response from daemon: Unable to find group render: no matching entries in group file

Docker resolves group_add names against the image’s own /etc/group, and this image’s /etc/group only defines video (gid 44) — no render entry. What actually matters for /dev/dri access is the host’s numeric gid that owns the bind-mounted device nodes (render=992, video=44 on this box). Fix: numeric gids instead of names — sidesteps the missing entry, and is arguably more portable in general.

4. Upstream routes W4A16 MoE to a CUDA-only Marlin kernel on XPU

After fixes 1–3 the container booted and loaded weights, then EngineCore crashed:

AttributeError: '_OpNamespace' '_C' object has no attribute 'gptq_marlin_repack'

Root cause, from the source: marlin_utils.py::check_moe_marlin_supports_layer() excludes only ROCm —

if current_platform.is_rocm():
    return False

— so on XPU it still returns True, and compressed_tensors_moe.py routes W4A16 MoE to CompressedTensorsWNA16MarlinMoEMethod, which calls a Marlin repack op that was never compiled into the XPU _C extension. No env-var escape hatch exists (checked VLLM_MARLIN_USE_ATOMIC_ADD, VLLM_MARLIN_INPUT_DTYPE, VLLM_MXFP4_USE_MARLIN, VLLM_TEST_FORCE_FP8_MARLIN — none apply; the platform check is the only gate). Fix: bind-mounted a corrected marlin_utils.py adding or current_platform.is_xpu(), mirroring the existing ROCm carve-out — the same mount-a-corrected-file pattern the old stack already uses for its gptq.py patch. After the fix: Using CompressedTensorsWNA16MoEMethod (the portable, non-Marlin path) and a clean boot through to serving.

Once the container could start at all, none of this touched attention-backend selection or KV-cache dispatch — those worked as hoped, unforced. Boot took ~3m51s cold (torch.compile 82.14s + profiling/warmup 24.09s included); a 1-token smoke test returned “Paris,” and a 40-token haiku coherence check came back well-formed and on-topic, ruling out silent numeric corruption in the W4A16/fp8/FlashAttention combination. Dispatch confirmed from the container logs:

[cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint...
[xpu.py:96] Using Flash Attention backend.
[flash_attn.py:641] Using FlashAttention version 2
[compressed_tensors_moe.py:113] Using CompressedTensorsWNA16MoEMethod
[kv_cache_utils.py:1710] GPU KV cache size: 270,592 tokens
[kv_cache_utils.py:1711] Maximum concurrency for 131,072 tokens per request: 2.06x

FLASH_ATTN dispatched on its own (--attention-backend was intentionally left unset), and fp8 KV did not force a fallback to a different backend — that specific combination was a known regression risk carried over from the 0.19 experience, and it didn’t recur here.

One honest caveat on the “XMX” half of “XMX FlashAttention”: the literal strings xmx and dpas do not appear anywhere in this image’s Python source (grep -r over the full vllm package tree — zero hits for both). platforms/xpu.py describes the kernel as “FMHA sycl-tla kernels,” and the actual attention math runs inside the compiled vllm_xpu_kernels package — SYCL, not Python — which, per this project’s own prior BMG-G31 findings (joint_matrix blocked on stock oneAPI 2025.3, ESIMD xmx::dpas as the working substitute), is the plausible mechanism underneath “sycl-tla.” But I did not disassemble the compiled kernel to confirm dpas instructions at the ISA level. What’s confirmed: the real FLASH_ATTN backend dispatched, not an eager/SDPA/CPU fallback. What’s not independently confirmed: ISA-level proof that it’s issuing XMX/DPAS underneath. Calling it XMX/DPAS by the kernel library’s design, not by verification.

Bench matrix

C=1 unless noted, 5-prompt amortization, prefix caching on, --dtype float16 on both stacks. Same 16-prompt mixed set (code / math / translation / prose / QA) as the April posts, truncated to 5. Harness: scripts/bench_vllm023.py — new, and unlike bench_c1_context.py, it separates prefill and decode instead of conflating them into one tok/s number.

metric	ctx	old (tok/s)	new (tok/s)	new/old
prefill	8,192	1,929.28	39,429.77	20.4x
prefill	32,768	2,546.79	12,034.88	4.7x
prefill	122,880	not measured	8,028.75	—
prefill	131,072	CITED, not re-run — ~20 min cold, O(N²) eager	not tested at this exact ctx	—
decode (C=1)	8,192	8.83	13.06	1.5x
decode (C=1)	32,768	8.57	12.90	1.5x
decode (C=1)	122,880	not measured	12.03	—
decode (C=1)	131,072	CITED, not re-run — ~flat vs context, bandwidth-bound MoE	not tested at this exact ctx	—
aggregate (C=12, ctx 8,192)	—	15.21 (production suffix draft)	110.92 (no draft)	not apples-to-apples — see Surprises

The two CITED old-stack rows at 131,072 are exactly that: cited from this project’s known historical risk profile, not re-run this session. Reaching 128K-class context on the old stack requires the ~20-minute cold eager prefill that’s the documented wedge risk for this box; the decode figure is cited as “roughly flat vs. context” because this MoE’s C=1 decode is bandwidth-bound on expert-weight loads rather than attention — a pattern the measured new-stack decode row confirms independently (13.06 → 12.90 → 12.03 tok/s across 8K→32K→123K, an ~8% drop over a 15x increase in context).

The prefill gap almost certainly widens past what’s measured here, not narrows. The old stack’s own trend from 8K to 32K actually ticks up slightly (1,929 → 2,547 tok/s — likely fixed per-request overhead amortizing over more tokens), but the cited ~20-minute cold prefill at 131,072 is only consistent with O(N²) eager attention cost overtaking that trend hard somewhere past 32K. We didn’t re-run that cell to get an exact number — that’s the point of citing it instead of guessing — but there’s no reading of “20 minutes for one prompt” that keeps the old stack within shouting distance of the new stack’s measured 8,028.75 tok/s at 122,880.

Surprises

• fp8’s measured KV budget (270,592 tokens) beat the pre-spike estimate (~207K) by about 30%. The plan’s back-of-envelope number came from a June changelog scan; the real number, read straight off kv_cache_utils.py’s boot-time log line, is meaningfully better.
• FlashAttention + fp8 KV coexisted without a forced fallback. This was the specific thing worth checking, because the 0.19 stack has a known history of KV-dtype changes silently knocking attention onto a slower backend. --attention-backend was left unset on purpose; XPU picked FLASH_ATTN on its own and fp8 didn’t move it. Confirmed, not assumed — this was worth being nervous about going in.
• The old stack’s aggregate C=12 number (15.21 tok/s, production suffix draft) is barely above its own C=1 no-draft decode number (8.83 tok/s) — only ~1.7x for 12x the concurrency. The new stack’s C=12 no-draft number (110.92 tok/s) is ~8.5x over its own C=1 no-draft decode (13.06). Different draft configs, so this isn’t a clean comparison (flagged in the table above), but the shape is real: switch-model.sh’s own production comments note suffix decoding is tuned for low-concurrency interactive latency and “deadlocks on XPU at C≥16” — consistent with per-request tree-verification overhead not scaling once the GPU is already compute-saturated by concurrent streams.
• That also means “old stack decode is 8.8 tok/s” undersells old-stack production on the right workload. The 8.83 figure is the no-draft, isolate-the-variable number used for the head-to-head table above. Actual old-stack production runs suffix decoding, and on repetitive content the April 16 post measured suffix+FP16 hitting 32.44 tok/s at 32K — the suffix tree loves repeated system-prompt padding. So the honest decode comparison depends heavily on the prompt: on fresh, non-repetitive content the new stack’s flat ~13 tok/s likely wins outright; on a stable system prompt or a cached document, the old stack’s suffix decoding could still be competitive or ahead, at 40% less KV headroom. Neither side had a speculative-decoding story on the new stack at the time; ngram closes part of that gap as of the next day (see the Update below), though it’s prompt-lookup rather than a trained draft model, and this specific repetitive-content/C=12 comparison hasn’t been rerun with it.
• The integration cost dropped, but didn’t hit zero. TurboQuant needed six historical fixes on the old stack (a Python 3.12 find_spec deprecation, dataclass Field.type patching, worker re-import, a cwd-shadowed import, a missing backend-registry enum, and a --max-num-seqs KV-sizing footgun). The new stack needed four, one of them a source patch. Less debt, not no debt.

The tradeoff, and the call I made

dimension	old stack (TQ 0.19)	new stack (upstream)
Speed	Prefill 1,929–2,547 tok/s @8–32K; ~20 min cold at 128K (cited). Decode 8.57–8.83 tok/s C=1; 15.21 tok/s C=12 with production suffix draft.	Prefill 8,029–39,430 tok/s @8–123K (20x/4.7x over old at 8K/32K). Decode 12.03–13.06 tok/s C=1 (~1.5x); 110.92 tok/s C=12 with no draft (not apples-to-apples vs. old’s suffix number — see Surprises).
KV budget	453,184 tokens measured @ 262,144 max-len (1.73x concurrency). Single-request ceiling 256K.	270,592 tokens measured @ 131,072 max-len (2.06x concurrency) — ~40% less aggregate. Single-request ceiling 131K as deployed that day; raised to 262K, the model’s native max, the next day — see Update.
Speculative decoding	suffix, production-tuned, integrated.	None as of this spike. ngram shipped the next day — see Update.
Stack maintenance	Custom fork of 0.19.0-tr5: hand-maintained gptq.py patch plus TurboQuant’s six historical integration fixes (monkey-patches, core-file mount overlays, worker-subprocess re-import hooks). Frozen at 0.19 indefinitely — no upgrade path without redoing the TQ integration.	Tracks intel/vllm:latest’s rolling dev branch directly. One bind-mounted file (marlin_utils.py) as the only carve-out. Re-pulling the image is the upgrade path, not a re-port.
Stability (this session)	No wedge, no Level-Zero/UR errors across two boots.	No wedge, no BMG crash signature (no SIGABRT #27408, no GP-fault #41663). Boots in ~3m51s cold; tool-calling (hermes parser) verified clean post-migration.

Given this, I migrated production the same day I finished collecting the numbers. Three things tipped it:

1. The prefill win changes what’s usable interactively, not just what’s fast on a benchmark. A cold 8K-token prompt going from ~14.5s to ~0.7s is the difference between “wait” and “instant” for the RAG and coding-assistant sessions that are most of what this box actually serves.
2. The stack-maintenance column isn’t close. Six historical TurboQuant integration fixes plus a hand-maintained GPTQ patch is an ongoing cost against every future vLLM upgrade. One bind-mounted marlin_utils.py against a rolling upstream branch is a cost I can plausibly stay ahead of.
3. Nothing broke that matters for the actual production workload. Tool-calling (the hermes parser, function-calling for the coding-assistant setup) came back clean after cutover — verified against a live get_weather-style request, not assumed. Both the old served-model-name (qwen3-30b-tq-256k) and a new one (qwen3-30b-a3b-w4a16) are served, so existing client configs didn’t need to change.

What I gave up: TurboQuant’s KV headroom (~40% less aggregate cache, single-request ceiling 256K → 131K) and suffix speculative decoding, which has no equivalent on the new stack yet. If a workload leans on suffix decoding’s edge on repetitive content — the April 16 post’s 32.44 tok/s at 32K on padded system prompts — this migration is a step back for that specific case until spec decode lands upstream. For this box’s actual traffic — interactive chat and coding-assistant sessions, not long repetitive-context batch workloads — the prefill win dominates and the decode/KV costs are acceptable. Both of these aged fast: the ceiling went back to 262K and ngram speculative decoding shipped within 24 hours (see the Update below). Ngram isn’t suffix, though — it closes part of the spec-decode gap, not all of it, and the repetitive-content case above is still unmeasured on the new stack.

Rollback isn’t theoretical, but it’s worth separating two different commands here. node_dark.sh restore brings production back after a bench-cycle drain (not a version rollback) — that’s what actually ran, repeatedly, restoring production between benchmark runs this session. The rollback to the TurboQuant stack is a different procedure: restore docker-compose.yml.tq-rollback + .env.tq-rollback, then switch-model.sh qwen3-30b-tq-256k, then docker compose --profile gpu up -d --force-recreate vllm — shown in full in the Repro section, and proven working this same session (450s to healthy). It just wasn’t needed for this call.

Honest limits

• The image is vLLM 0.21.1.dev17+g0a4756bb5, not 0.23. intel/vllm:latest tracks a rolling dev branch. This entire spike started under a “0.23” working title before the first boot corrected it — noting it here in case that number is floating around anywhere else in this project’s history.
• XMX/DPAS dispatch is not ISA-verified. The strings "xmx" and "dpas" don’t appear anywhere in this image’s Python source. Confirmed: the upstream XPU FLASH_ATTN backend dispatches — not an eager/SDPA/CPU fallback — unforced by fp8 KV, and platforms/xpu.py calls the kernel “FMHA sycl-tla kernels.” Not confirmed: ISA-level proof that the compiled vllm_xpu_kernels binary is issuing dpas instructions. Calling it XMX/DPAS by the kernel library’s design, not by verification.
• Both stacks run on a patch. The old stack needs gptq.py to boot on XPU at all. The new stack needed a bind-mounted marlin_utils.py carve-out because check_moe_marlin_supports_layer() excludes ROCm but not XPU, routing W4A16 MoE to a CUDA-only Marlin kernel that isn’t compiled for XPU. Neither is pure-stock.
• fp8’s aggregate KV budget is ~40% smaller than TurboQuant’s, and this deploy’s single-request ceiling dropped from 256K (262,144) to 131K (131,072) that day. That 131,072 cap was this spike’s deliberate conservative setting, carried into production — not a proven hard limit, and it didn’t stay a limit for long: pushed to the model’s full native 262,144 the very next day, KV pool unchanged (see the Update below). The aggregate budget itself (270,592 tokens) is unchanged either way — that’s the part of this tradeoff that’s actually load-bearing, not the per-request ceiling.
• No speculative decoding on the new stack, as of this writing. Every new-stack number above is draft-none. The C=12 aggregate comparison (110.92 vs. 15.21 tok/s) pits a no-draft config against old-stack production suffix decoding and is explicitly not apples-to-apples. Resolved the next day — ngram speculative decoding shipped to production; see the Update below.
• Stack-vs-stack, not an isolated attention-kernel benchmark. Attention backend, weight quant, and KV scheme all changed together; the measured deltas can’t be cleanly attributed to FlashAttention alone.
• C=1 is the primary story; exactly one C=12 aggregate cell per stack. No broader concurrency sweep.
• 5-prompt samples, not 16. Same noise caveat as the April 16 post — expect roughly ±10-15% at this sample size.
• Old stack’s 131,072-context prefill and decode rows are CITED, not re-run this session — the known ~20-minute O(N²) cold-eager-prefill wedge risk. The new stack’s largest measured context is 122,880, not literally 131,072 — close, not identical.
• Idle-node protocol, stated plainly: drained production via docker compose stop vllm open-webui (health-drain — the cluster router’s health check fails and reroutes), never iptables -j REJECT (a prior session found that triggers a reconnect storm that loads the host CPU and skews prefill timing). Verified idle before benching: connections on host :8000 dropped to only loopback TIME-WAIT entries, the legacy 192.168.0.62 feeder showed zero established connections, and host load average was quiescent. Old and new stacks were benched sequentially on dark port 8001, never concurrently, and production was restored on 8000 — then migrated — as an explicit exit step, not left drained.
• XPU-specific. Ratios here won’t transfer to other hardware. Battlemage-specific fragility this project has documented before (SIGABRT on B-series model inspection, TP=2 GP-fault) wasn’t triggered by this workload — that isn’t the same as saying it’s fixed.

Repro

Drain / verify / restore (scripts/node_dark.sh):

cd /apps/b70-vllm/turboquant-xpu/.worktrees/vllm-023-spike
scripts/node_dark.sh drain     # docker compose stop vllm open-webui on :8000
scripts/node_dark.sh verify    # confirms ~0 conns on :8000, legacy feeder quiet, load avg low
# ... bring the stack-under-test up on dark port 8001, bench, tear down ...
scripts/node_dark.sh restore   # brings current prod stack back up (compose up), polls /health up to 12 min — not a version rollback

Old stack on 8001:

cd /apps/b70-vllm
./switch-model.sh qwen3-30b-tq-256k
VLLM_PORT=8001 VLLM_SPEC_CONFIG='' docker compose --profile gpu up -d --force-recreate vllm

New stack on 8001 (deploy/docker-compose.vllm023.yml + deploy/.env + deploy/vllm023.env):

cd /apps/b70-vllm/turboquant-xpu/.worktrees/vllm-023-spike/deploy
docker compose -f docker-compose.vllm023.yml up -d

Bench harness (scripts/bench_vllm023.py — new; splits prefill and decode instead of conflating them like bench_c1_context.py):

.venv-sycl/bin/python scripts/bench_vllm023.py --stack old --metric prefill \
  --endpoint http://localhost:8001 --model-alias qwen3-30b-tq-256k \
  --ctx 8192 --n-prompts 5 --output docs/tuning/c1_vllm023_old_2026-07-04.txt

.venv-sycl/bin/python scripts/bench_vllm023.py --stack new --metric aggregate \
  --endpoint http://localhost:8001 --model-alias qwen3-30b-w4a16 \
  --ctx 8192 --concurrency 12 --max-tokens 200 --note nodraft,C12 \
  --output docs/tuning/c1_vllm023_new_2026-07-04.txt

Chart: scripts/plot_vllm023_compare.py --inputs docs/tuning/c1_vllm023_old_2026-07-04.txt docs/tuning/c1_vllm023_new_2026-07-04.txt --out site/assets/vllm023_compare_2026-07-04.png.

Raw data (committed): c1_vllm023_old_2026-07-04.txt / .kvlog, c1_vllm023_new_2026-07-04.txt / .kvlog.

Rollback, proven working during this same session (450s to healthy):

cp /apps/b70-vllm/docker-compose.yml.tq-rollback /apps/b70-vllm/docker-compose.yml
cp /apps/b70-vllm/.env.tq-rollback /apps/b70-vllm/.env
cd /apps/b70-vllm && ./switch-model.sh qwen3-30b-tq-256k
docker compose --profile gpu up -d --force-recreate vllm

Full deployment source: github.com/bryanvine/turboquant-xpu. The upstream Marlin-MoE carve-out patch lives at patches/vllm023_upstream_fixes/marlin_utils.py in the worktree (mirrored to production at /apps/b70-vllm/vllm-patches/marlin_utils.py, host-specific, not in-repo).

Update (2026-07-05): pushing to 262K and settling speculative decoding

Two of the items flagged above in “Honest limits” and “What’s next” didn’t survive 24 hours as open questions. The conservative 131K ceiling got tested properly, and the “no speculative decoding” gap got closed — though not cleanly on the first attempt at closing it.

The ceiling was self-imposed, not measured

131,072 was flagged above as “this spike’s deliberate conservative setting… not a proven hard limit,” with the fp8 pool already sized at more than double that. It held up under an actual test: --max-model-len went to 262,144 — Qwen3-30B-A3B’s native max context (~270K) — and the stack booted and served cleanly, no code changes required.

[cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint...
[xpu.py:96] Using Flash Attention backend.
[flash_attn.py:641] Using FlashAttention version 2
[kv_cache_utils.py:1710] GPU KV cache size: 270,592 tokens
[kv_cache_utils.py:1711] Maximum concurrency for 262,144 tokens per request: 1.03x

The number that matters here is the one that didn’t move: the KV pool is still 270,592 tokens — identical to the figure measured at the old 131,072 max-len earlier in this post. --max-model-len sets the per-request ceiling, not the pool size; the pool is fixed by --gpu-memory-utilization and the model/KV-dtype footprint, independent of it. So the “single-request ceiling 256K → 131K” framing above is now stale: the new stack’s ceiling is 262K, matching the old TurboQuant stack’s, at 1.03x concurrency for full-length requests. The real, still-standing gap between the two stacks is what it always was — the ~40%-smaller aggregate pool (270,592 vs. 453,184 tokens) — not the per-request cap.

Speculative decoding: three tried, one ships

• suffix — doesn’t boot. SpeculativeConfig.__post_init__ hard-fails at config validation (config/speculative.py:769), before the engine starts: ImportError: Arctic Inference is required for suffix decoding. Install via 'pip install arctic-inference==0.1.1'. The package isn’t in intel/vllm:latest. Not an XPU or FlashAttention incompatibility — a packaging gap.
• EAGLE3 — a dead end at both context settings tried. At 262,144 it doesn’t boot at all: loading the draft model brings total model memory to 16.06 GiB and leaves only 11.74 GiB free for KV cache (down from 12.39 GiB with no draft), and vLLM’s own pre-flight check refuses to start a config that can’t fit one full-length request — it needs 12.25 GiB and estimates a max feasible len of 251,200. Retried at 131,072, where it does boot — and decodes slower than no-SD (12.04 vs. 12.79 tok/s): draft acceptance runs 40% on short prompts but collapses to 0.0% at 8K context, almost certainly because the draft’s own config.json caps max_position_embeddings at 2048 and it’s being fed positions far outside that range. Output stays correct regardless (a rejected draft just falls back to the target model), but a 0%-acceptance draft is pure overhead.
• ngram — works, and costs nothing: it’s prompt-lookup, no draft weights, no draft KV, so the pool stays at the full 270,592 tokens regardless of whether it’s enabled. Whether it’s worth shipping turned out to depend entirely on concurrency.

The concurrency curve, and nearly shipping the wrong answer

Aggregate tok/s with --max-model-len set to 262,144, actual prompt context 8,192 tokens, warm/steady-state. (The literal first request after a cold boot pays a one-time ~20s JIT/compile tax that has nothing to do with speculative decoding — measured, confirmed via a warm re-run, and controlled for symmetrically on both sides below.)

concurrency	ngram	no-SD	ngram Δ
C=1	13.96	11.22	+24%
C=2	28.14	24.74	+14%
C=3	41.34	37.01	+12%
C=4	51.18	50.34	+2%
C=12	39.99	72.56	−45%

ngram forces two things at dispatch time, unconditionally: Async scheduling not supported with ngram-based speculative decoding and will be disabled, and max_num_scheduled_tokens is set to 2048. Neither costs anything at C≤4 — a decode step at that concurrency needs on the order of tens of scheduled token-slots, nowhere near the cap. But it’s the same mechanism either way, and once concurrency is high enough for async-scheduling-off and the 2048 cap to actually bind, the win inverts hard: by C=12 the scheduling tax outweighs ngram’s per-token benefit by a wide margin. The shape is clean and monotonic — +24% → +14% → +12% → +2% → −45% — exactly what you’d expect from a roughly fixed per-token speculative win competing against a scheduling cost that scales with concurrency.

That shape is also why this nearly shipped the wrong config. The first pass at this decision measured exactly two points — C=1 and C=12 — saw the C=12 number, and came within a hair of shipping no speculative decoding at all, on the strength of a single high-concurrency cell. The gap: this node’s actual traffic is C=1 through C=4, usually C=1, and effectively never C=12. That’s precisely the regime where ngram wins at every measured point — and wins biggest exactly where most of the traffic lives (C=1, +24%). The lesson isn’t “ngram is good” or “ngram is bad.” It’s measure at your actual concurrency, not a convenient round number. C=1 and C=12 were convenient to bench; they aren’t what this box serves.

Final config

Production now runs 262K context + ngram speculative decoding — --speculative-config '{"method":"ngram","num_speculative_tokens":5,"prompt_lookup_max":4,"prompt_lookup_min":2}' — on top of the same W4A16 + fp8 KV + FlashAttention stack described above. Post-change verification: tool-calling still comes back clean (a get_weather-style call returns a well-formed finish_reason:"tool_calls"), and the KV pool is unchanged at 270,592 tokens, 1.03x concurrency at the full 262,144 ceiling — ngram genuinely costs zero pool capacity, confirmed rather than assumed. Everything else in the production command is byte-identical to the config described earlier in this post; ngram’s config block is the only addition. A rollback snapshot (docker-compose.yml.pre-ngram) exists on the same pattern as the TQ rollback in Repro, and wasn’t needed.

Follow-up (2026-07-05): the newer hybrid models — Qwen3.5 & 3.6

The whole premise of this migration was riding whatever frontier move a B70 can actually ride. So the obvious next question: now that we’re on upstream, can the upgraded stack run the newer open models — Qwen3.5-35B-A3B and Qwen3.6-35B-A3B? Both are hybrid Gated-DeltaNet + sparse-MoE architectures (linear-attention layers interleaved 3:1 with full attention), and getting Qwen3.5 to run at all was a 17-task custom-image project back in April. Short version: one of them works, neither is worth swapping to, and the reason is instructive.

The good news: native GDN on XPU is real now

In April, the Gated-DeltaNet layers had no XPU dispatcher — they fell back to a Triton/FLA path that produced gibberish, and getting Qwen3.5 to ~7 tok/s took a hand-built 0.19.1rc0 image with an unmerged kernel PR. That work is now upstream: intel/vllm:latest ships torch.ops.vllm.gdn_attention_core_xpu, a native Battlemage GDN kernel wired into the model’s forward_xpu. (The boot log’s Using Triton/FLA GDN prefill kernel line is a red herring — the XPU platform hook reassigns the forward method to the native op right after it prints.) Confirmed working by Qwen3.5 producing coherent output — the thing April hand-built now comes in the box.

Qwen3.5-35B-A3B (official GPTQ-Int4): works, but slow and fragile

The official Qwen GPTQ checkpoint is a genuine multimodal build — text weights, fp16 vision tower, MTP draft layers — so it loads on the stock, fully-wired Qwen3_5MoeForConditionalGeneration class with zero code patches. It runs, and it’s coherent (it’s a thinking model — emits a visible reasoning trace before answering). But it took a specific, constrained config to get there:

• --enforce-eager is mandatory. With torch.compile on (the production default), boot hangs — one core pegged for 20+ minutes compiling the GDN splitting_ops, never reaching serving. Eager mode boots, but forfeits the compile speedup that the 30B stack enjoys.
• No prefix caching. Enabling it forces an experimental “align” mamba cache mode whose state-copy path crashes on this build (get_mamba_state_copy_func() returns None → TypeError: object of type 'NoneType' has no len()).
• GPTQ is fp16-only (it rejects bfloat16 outright) and boots with a "4-bit gptq_gemm kernel is buggy, switch to gptq_marlin" warning — and Marlin is CUDA-only on this card. It runs anyway; that’s just not a reassuring banner.
• ~5.4 min cold boot, and only 6.46 GiB left for KV after the 23 GB of weights + the vision tower.

Throughput at the concurrency this box actually serves (C=1–4):

metric	Qwen3.5-35B GPTQ	Qwen3-30B-A3B (current)
decode, C=1	7.3 tok/s	~13 tok/s
aggregate, C=2	14.6 tok/s	28.1 (ngram)
aggregate, C=4	29.0 tok/s	51.2 (ngram)
prefill @2K / @8K	3,429 / 5,033 tok/s	—

3.5’s aggregate cells are measured at 2K context vs. the 30B curve’s 8K, so those two columns aren’t a clean overlay — but the single-stream C=1 decode, the cleanest comparison, is unambiguous: ~7.3 vs ~13 tok/s, roughly half. That’s the --enforce-eager tax, mostly — the 30B gets torch.compile; the hybrid can’t.

Qwen3.6-35B-A3B (community CT-INT4): boots, generates garbage

3.6 is the newer model and the one I actually wanted. The only checkpoint on disk that could run on XPU is a community compressed-tensors INT4 quant, and it turned out to be a text-only-weights build wearing a multimodal config. Upstream’s text-only Qwen3.5/3.6 classes are only half-wired, so getting it to boot took a stack of patches: register the text-only class in the model registry, tag it IsHybrid (so mamba_block_size gets set at all), add the two get_mamba_state_*_from_config classmethods the hybrid-KV setup calls, and implement get_mrope_input_positions (Qwen3.5/3.6 use M-RoPE even for text). After all that it boots, profiles, and generates fast — 80 tok/s — but the output is word-salad ("om garò prezi domesticeder ò…"). fp16 vs. bf16 changed the flavor of garbage (!!!! → word-salad) but not the fact of it; keeping vs. stripping M-RoPE made no difference. Weights load with no missing-tensor warnings, so it’s either a bad community quant or a subtle fault in the text-only load path — and I couldn’t isolate which, because the only other 3.6 checkpoint available is a GGUF, which doesn’t run on XPU at all. Unresolved, and not deployable with the weights on hand.

The call: kept the 30B

Both hybrids share one disqualifying trait for this box: the torch.compile hang forces --enforce-eager, which roughly halves decode throughput versus the compiled 30B. Stack that on top of 3.5’s thinking-model verbosity (more tokens per answer, so felt latency is worse than the raw tok/s gap already shows), its GPTQ fragility and tiny KV budget, and 3.6 being outright broken — and there’s no version of “swap to a 35B” that comes out ahead for a C=1-dominant interactive box that cares about latency. The intelligence edge of a newer 35B-A3B is real, but it doesn’t buy back half the speed plus the operational tax.

Same lesson as the speculative-decoding curve above, from the other direction: the 30B wins where the traffic actually lives (C=1–4). The migration’s verdict stands — the 30B stays.

Two things would flip it, and I’ll re-check both on the next image bump: a bf16/W4A16 (non-GPTQ) Qwen3.5/3.6 checkpoint that loads via the stock multimodal class (no text-only patch surface, no fp16-only constraint), or torch.compile no longer hanging on the GDN layers — which would hand the hybrids their missing ~2x and likely put a 35B-A3B ahead of the 30B outright.

Follow-up (2026-07-06): the speculative-decoding call was wrong — EAGLE-3 ships

The Update above wrote EAGLE-3 off as “a dead end at this model’s context lengths” and shipped ngram — with one hedge buried in the diagnosis: acceptance “collapses to 0.0% at 8K context, almost certainly because the draft’s own config.json caps max_position_embeddings at 2048.” That hunch was right. It was also the entire story, and chasing it down flipped the production decision.

“~0% acceptance at long context” was a config bug, not a model limit

Put EAGLE-3 back on an isolated port, swept draft acceptance against prompt length. It isn’t a slope, it’s a cliff:

prompt context	mean accepted tokens/step
~1.2–1.6K	1.3–1.6 (healthy)
~1.9K	0.44 (sliding)
≥2.6K	0.00 (every position, zero)

Acceptance drops to exactly zero the moment the prompt crosses 2048 tokens, and stays pinned there. A weak draft still lands the occasional common token; zero across ~1,000 proposals is a switch flipping off. The draft’s config.json says max_position_embeddings: 2048, vLLM honours it, so past position 2048 the draft’s RoPE is garbage and the target rejects 100% of it. (The head’s README even ships a SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 escape hatch — a known sharp edge, just not one vLLM lifts for you.) And it is not chunked prefill: raising --max-num-batched-tokens to 32,768 so a 25K prompt prefills in one chunk changed nothing. The cliff tracks absolute context, not batching.

The fix: one field

Copy the head, set max_position_embeddings to the target’s 262K ceiling, point --speculative-config at the copy. No training, no long-context draft, no kernel work. Acceptance returns immediately — flat ~1.0–1.25 out to 20K, then a graceful decay at 50–74K (the genuine “this head was trained at short context” effect, but irrelevant here — see the traffic below). And it’s nearly free: the draft trims the KV pool from 270,592 to 265,664 tokens, ~1.8%. The old “EAGLE eats your pool” worry was real only for the 262K boot that couldn’t fit draft-weights-plus-a-full-length-request; sized normally, the 366 MB head barely registers.

What actually favors ngram vs EAGLE-3 (I had this backwards)

Before replaying real traffic I predicted ngram would win, because the traffic looked repetitive — the same few automated system prompts all day. Wrong axis. The rule isn’t “repetitive input → ngram.” It’s:

Does the output echo the input?

ngram is prompt-lookup — it can only propose tokens it finds in the context. On code continuation, where the output copies identifiers and boilerplate already on screen, it drafts 3–5 tokens at a time for free and beats EAGLE handily. On novel generation — a fresh answer, a sentiment verdict, a JSON object that isn’t sitting in the prompt — it has nothing to copy and stalls near one token, while EAGLE-3 (predicting from the model’s own hidden states) keeps drafting. Starkest single case in a content sweep: “write an original fable” → EAGLE 0.76 vs ngram 0.062. The shape of the input isn’t the question; whether the output is in there to be copied is.

Measuring it: a day of real traffic

Rather than argue from synthetic prompts, I put a fail-open capture proxy in front of the server — tees every /v1 request body to disk, forwards to vLLM, and if anything in the capture path throws it forwards anyway (never break prod for a measurement). A full business day: ~1,900 real requests, and the shape was its own surprise — ~99% automated structured-JSON classification (financial-news sentiment, sports-betting research; all response_format: json_object or guided_json), every one under 2K context, and exactly two human chat messages all day. Then I replayed a sample through both configs verbatim — same messages, same json_object/guided_json, same temperatures — so it’s real work, not normalized toy prompts. Faithfulness check: ngram’s replayed acceptance (1.00) matches ngram measured live on the full day (1.045).

The result, and the switch

	mean accept	C=1 tok/s	C=4 tok/s
EAGLE-3 (patched)	1.81	20.9	67.2
ngram	1.00	12.9	46.9

EAGLE-3 wins 1.6x at C=1 and 1.4x at C=4 — the margin narrows under load (both batch better as concurrency climbs) but never flips, and acceptance holds flat at ~1.8 across the range. The “repetitive input → ngram” instinct was exactly backwards here: the inputs repeat, but the outputs — the JSON verdicts — are novel every time, which is EAGLE’s court. Production now runs EAGLE-3: the patched draft on a stable path, --enforce-eager (same torch.compile caveat as the hybrids, but here it costs far less than the ~2x it buys), everything else byte-identical to the ngram config it replaced. Rollback to ngram is one docker compose up against the saved snapshot.

Two things I got wrong

Both were decisions, not just facts, so worth stating plainly:

1. I wrote EAGLE-3 off as “a dead end at this model’s context lengths.” It wasn’t the model or the context — it was a four-digit number in a JSON file. The 0%-acceptance symptom was load-bearing and I’d stopped one diagnostic step short of the cause.
2. I predicted ngram would win the real traffic, and it lost — because I reasoned from the shape of the input instead of the output. The replay is the only reason the right config shipped; the prediction would have kept the wrong one.

The ngram concurrency curve’s lesson was “measure at your real concurrency.” This one’s is adjacent: measure on your real traffic. The synthetic sweep got the mechanism right (novel output favors EAGLE); only the captured day got the weighting right — this box’s traffic is almost entirely novel output.

What’s next

1. Speculative decoding on the new stack — resolved, then re-resolved. ngram shipped first (see the 07-05 Update), but the 2026-07-06 follow-up above reversed it: the “EAGLE3 dead end” was the draft’s max_position_embeddings: 2048 cap, a one-field fix, and on a captured day of real traffic EAGLE-3 beat ngram 1.4–1.6x and now ships in production. Still open: suffix decoding, which needs a derived image with arctic-inference baked in (this image doesn’t carry it) — untested against EAGLE-3, and only likely to matter if the traffic mix ever shifts toward output-copies-input (code editing, RAG-quoting), where prompt-lookup regains its edge.
2. Whether to reclaim context if the 270,592-token aggregate budget turns out to bite. The single-request ceiling is now the model’s own native max (262K — see the Update above), so there’s no higher per-request context left to chase on this model; the only remaining lever is aggregate concurrency headroom (1.03x at full-length requests, as deployed). XQuant — the KV-compression idea parked from the never-built April 26 “nextgen stack” series — is the fallback if a real workload needs more concurrent full-length requests than that. fp8 already halves KV cost against fp16; XQuant would be stacking a second compression layer on top of it, the way TurboQuant did for the old stack.
3. Push --max-model-len closer to the measured fp8 ceiling — done. It goes all the way to 262,144, the model’s native max, with the KV pool unchanged (see the Update above). No further headroom to chase on this specific model; a higher single-request ceiling would need a different model or a context-extension trick, not just a flag change.
4. Re-check Qwen3.5 / 3.6 on the next image bump. The native GDN kernel works now (see the follow-up above); the blockers are the torch.compile hang that forces --enforce-eager (~halving throughput) and, for 3.6, a checkpoint that generates garbage. A bf16/W4A16 non-GPTQ checkpoint that loads via the stock multimodal class, or a compile path that doesn’t hang on GDN, would make a 35B-A3B genuinely competitive with — probably faster than — the 30B.

---

Fourth post in the Arc B70 series. Repo: github.com/bryanvine/turboquant-xpu. Prior posts: April 14, April 15, April 16.