• Ran the head-to-head, then didn’t wait for a follow-up post to act on it. The plan going in was “characterize, defer the migrate verdict” — benchmark upstream intel/vllm:latest (compressed-tensors W4A16 + fp8 paged KV + the XPU FLASH_ATTN backend) against the production 0.19 + patched-GPTQ + TurboQuant stack on the Arc Pro B70, ship the numbers, decide later. The numbers came in clear enough that “later” happened the same session: production is now running the new stack. The old stack (TQ 0.19) is preserved as a one-command rollback.
• This is a stack-vs-stack comparison, not an isolated FlashAttention microbenchmark. Attention backend, weight quant (GPTQ → W4A16), and KV scheme (TurboQuant → fp8) all changed at once. None of the numbers below isolate cleanly to “FlashAttention did this” — saying so up front, not burying it in the limits section.
• The upstream image is vLLM 0.21.1.dev17+g0a4756bb5, not 0.23. intel/vllm:latest tracks a rolling dev branch; there’s no numbered 0.23 release inside it, despite that being this whole spike’s working title going in.
• Prefill: ~20x faster at 8K context, ~4.7x at 32K (1,929 → 39,430 tok/s @8K; 2,547 → 12,035 tok/s @32K, C=1, no draft either side). Decode: only ~1.5x (8.6–8.8 → 12–13 tok/s) — this MoE’s decode is bandwidth-bound on expert-weight loads, not attention, so a faster attention kernel barely touches it.
• fp8 KV holds ~40% less aggregate cache than TurboQuant (270,592 vs 453,184 tokens, both measured) — that gap, not the context ceiling, is the real bill for the speedup. The single-request ceiling as deployed that day was a conservative 131K; it went back up to 262K within 24 hours once tested (see the Update near the end of this post).
• Neither stack is stock. The old stack needs its patched gptq.py to boot on XPU at all; the “upstream” new stack needed its own bind-mounted marlin_utils.py carve-out, because it routes W4A16 MoE to a CUDA-only Marlin kernel that isn’t compiled for XPU. Both ship on a patch.
• Follow-up — can the upgraded stack run the newer hybrid models (Qwen3.5 / 3.6)? The native XPU Gated-DeltaNet kernel that needed a hand-built image back in April now ships in the box. But both 35B-A3B hybrids get forced onto --enforce-eager (torch.compile hangs on the GDN layers), which roughly halves throughput: Qwen3.5-GPTQ works but is a fragile, verbose thinking model at ~7.3 tok/s C=1 (~½ the 30B), and the one runnable Qwen3.6 checkpoint generates garbage. Kept the 30B. See the Qwen3.5/3.6 follow-up near the end of this post.
• Follow-up — the speculative-decoding call above was wrong; EAGLE-3 now ships. The Update’s “EAGLE3 is a dead end” verdict was a config bug, not a model limit: the draft’s config.json capped max_position_embeddings at 2048, zeroing acceptance above that. One-field fix — then a captured day of real traffic settled it. The workload is ~99% novel structured-JSON output (not the repetitive-input case ngram wins), so EAGLE-3 beats ngram 1.6x @C=1 / 1.4x @C=4 and replaced it in production. See the 2026-07-06 follow-up near the end.
• Fourth post in the Arc B70 series. Previous posts: the first SYCL PoC + 2x Triton fix, three SYCL attempts and the gap to Triton, TurboQuant + EAGLE3, six integration fixes.

Why now: the frontier moved

DeepSeek V4 shipped April 24, 2026, and cut its KV cache to roughly 10% of V3.2’s footprint by training in Compressed Sparse Attention, Heavily Compressed Attention, and DSA — architecture-level attention tricks that make long-context serving dramatically cheaper, if you can run them. This box can’t. V4 is 284B+ parameters, and the fast paths (FlashMLA, FlashInfer) are CUDA-only; none of it touches a 32 GiB Arc Pro B70. So V4 isn’t this post’s subject — it’s the reason to go looking for what frontier move a B70 actually can ride. The answer was sitting in the changelogs: vllm-xpu-kernels shipped a real FlashAttention implementation for Battlemage — XMX/DPAS-targeted kernels for both prefill and paged decode — plus fp8 paged KV, and it landed in upstream intel/vllm. That’s not V4’s 90% KV cut. It’s the actual, runnable-today upgrade available to this card, and it’s what the rest of this post measures.

The two stacks

Three things changed between the stack that was serving production yesterday and the one serving it now:

Two details in the right-hand column above went stale within a day: the max-len value in the KV-cache row (131,072 → 262,144; the pool size itself, 270,592, didn’t move) and the speculative-decoding row both changed. See the Update near the end of this post.

Every row moved together — this is a stack-vs-stack comparison, not a controlled FlashAttention-only experiment. The ~20x prefill number a few sections down reflects the attention-backend switch and the quant-format switch and the KV-scheme switch, all landing in the same container image at once. There’s no ablation in this dataset that isolates FlashAttention’s individual contribution, and I’m not going to write around that.

The pre-spike plan also assumed the new stack would land close to stock upstream, with the old stack carrying all the patch debt. It didn’t work out that clean. Getting W4A16 MoE to actually run on XPU needed a source patch of its own (below) — so the honest framing is patch-vs-patch, not patch-vs-stock.

What it took to run upstream on the B70

intel/vllm:latest is not a drop-in swap. Four fixes were needed before it would serve a single token on this box — three Compose/Docker plumbing issues and one source patch.

1. The entrypoint eats a bare command list

intel/vllm:latest’s ENTRYPOINT is ["/bin/bash", "-c", "source /opt/intel/oneapi/setvars.sh --force && exec \"$@\"", "--"] — a bash -c script -- "$@" wrapper. The trailing -- is consumed as $0; whatever command: says becomes "$@" and gets exec‘d verbatim. There is no implicit vllm serve anywhere. A command: of bare flags (--model foo ..., no program name) fails with exec: --: invalid option — confirmed empirically (docker run intel/vllm:latest vllm serve --help reaches vLLM’s own argparse; the bare-flags form doesn’t). Fix: command: starts with vllm, serve.

2. env_file: doesn’t feed Compose’s own ${VAR} interpolation

The compose file’s command: needed "${VLLM023_MODEL}". env_file: only injects variables into the container’s runtime environment; Compose’s own ${VAR} YAML interpolation reads shell env or a .env file in the compose project directory — it never sees env_file: contents. Caught via docker compose config (dry run) before ever booting for real: it rendered --model "". Fix: added deploy/.env alongside the pre-existing deploy/vllm023.env (kept, still used for the container’s own runtime env).

3. group_add: [render, video] fails — the image has no render group

Error response from daemon: Unable to find group render: no matching entries in group file

Docker resolves group_add names against the image’s own /etc/group, and this image’s /etc/group only defines video (gid 44) — no render entry. What actually matters for /dev/dri access is the host’s numeric gid that owns the bind-mounted device nodes (render=992, video=44 on this box). Fix: numeric gids instead of names — sidesteps the missing entry, and is arguably more portable in general.

4. Upstream routes W4A16 MoE to a CUDA-only Marlin kernel on XPU

After fixes 1–3 the container booted and loaded weights, then EngineCore crashed:

AttributeError: '_OpNamespace' '_C' object has no attribute 'gptq_marlin_repack'

Root cause, from the source: marlin_utils.py::check_moe_marlin_supports_layer() excludes only ROCm —

if current_platform.is_rocm():
    return False

— so on XPU it still returns True, and compressed_tensors_moe.py routes W4A16 MoE to CompressedTensorsWNA16MarlinMoEMethod, which calls a Marlin repack op that was never compiled into the XPU _C extension. No env-var escape hatch exists (checked VLLM_MARLIN_USE_ATOMIC_ADD, VLLM_MARLIN_INPUT_DTYPE, VLLM_MXFP4_USE_MARLIN, VLLM_TEST_FORCE_FP8_MARLIN — none apply; the platform check is the only gate). Fix: bind-mounted a corrected marlin_utils.py adding or current_platform.is_xpu(), mirroring the existing ROCm carve-out — the same mount-a-corrected-file pattern the old stack already uses for its gptq.py patch. After the fix: Using CompressedTensorsWNA16MoEMethod (the portable, non-Marlin path) and a clean boot through to serving.

Once the container could start at all, none of this touched attention-backend selection or KV-cache dispatch — those worked as hoped, unforced. Boot took ~3m51s cold (torch.compile 82.14s + profiling/warmup 24.09s included); a 1-token smoke test returned “Paris,” and a 40-token haiku coherence check came back well-formed and on-topic, ruling out silent numeric corruption in the W4A16/fp8/FlashAttention combination. Dispatch confirmed from the container logs:

[cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint...
[xpu.py:96] Using Flash Attention backend.
[flash_attn.py:641] Using FlashAttention version 2
[compressed_tensors_moe.py:113] Using CompressedTensorsWNA16MoEMethod
[kv_cache_utils.py:1710] GPU KV cache size: 270,592 tokens
[kv_cache_utils.py:1711] Maximum concurrency for 131,072 tokens per request: 2.06x

FLASH_ATTN dispatched on its own (--attention-backend was intentionally left unset), and fp8 KV did not force a fallback to a different backend — that specific combination was a known regression risk carried over from the 0.19 experience, and it didn’t recur here.

One honest caveat on the “XMX” half of “XMX FlashAttention”: the literal strings xmx and dpas do not appear anywhere in this image’s Python source (grep -r over the full vllm package tree — zero hits for both). platforms/xpu.py describes the kernel as “FMHA sycl-tla kernels,” and the actual attention math runs inside the compiled vllm_xpu_kernels package — SYCL, not Python — which, per this project’s own prior BMG-G31 findings (joint_matrix blocked on stock oneAPI 2025.3, ESIMD xmx::dpas as the working substitute), is the plausible mechanism underneath “sycl-tla.” But I did not disassemble the compiled kernel to confirm dpas instructions at the ISA level. What’s confirmed: the real FLASH_ATTN backend dispatched, not an eager/SDPA/CPU fallback. What’s not independently confirmed: ISA-level proof that it’s issuing XMX/DPAS underneath. Calling it XMX/DPAS by the kernel library’s design, not by verification.

Bench matrix

C=1 unless noted, 5-prompt amortization, prefix caching on, --dtype float16 on both stacks. Same 16-prompt mixed set (code / math / translation / prose / QA) as the April posts, truncated to 5. Harness: scripts/bench_vllm023.py — new, and unlike bench_c1_context.py, it separates prefill and decode instead of conflating them into one tok/s number.

The two CITED old-stack rows at 131,072 are exactly that: cited from this project’s known historical risk profile, not re-run this session. Reaching 128K-class context on the old stack requires the ~20-minute cold eager prefill that’s the documented wedge risk for this box; the decode figure is cited as “roughly flat vs. context” because this MoE’s C=1 decode is bandwidth-bound on expert-weight loads rather than attention — a pattern the measured new-stack decode row confirms independently (13.06 → 12.90 → 12.03 tok/s across 8K→32K→123K, an ~8% drop over a 15x increase in context).

The prefill gap almost certainly widens past what’s measured here, not narrows. The old stack’s own trend from 8K to 32K actually ticks up slightly (1,929 → 2,547 tok/s — likely fixed per-request overhead amortizing over more tokens), but the cited ~20-minute cold prefill at 131,072 is only consistent with O(N²) eager attention cost overtaking that trend hard somewhere past 32K. We didn’t re-run that cell to get an exact number — that’s the point of citing it instead of guessing — but there’s no reading of “20 minutes for one prompt” that keeps the old stack within shouting distance of the new stack’s measured 8,028.75 tok/s at 122,880.

Surprises

• fp8’s measured KV budget (270,592 tokens) beat the pre-spike estimate (~207K) by about 30%. The plan’s back-of-envelope number came from a June changelog scan; the real number, read straight off kv_cache_utils.py’s boot-time log line, is meaningfully better.
• FlashAttention + fp8 KV coexisted without a forced fallback. This was the specific thing worth checking, because the 0.19 stack has a known history of KV-dtype changes silently knocking attention onto a slower backend. --attention-backend was left unset on purpose; XPU picked FLASH_ATTN on its own and fp8 didn’t move it. Confirmed, not assumed — this was worth being nervous about going in.
• The old stack’s aggregate C=12 number (15.21 tok/s, production suffix draft) is barely above its own C=1 no-draft decode number (8.83 tok/s) — only ~1.7x for 12x the concurrency. The new stack’s C=12 no-draft number (110.92 tok/s) is ~8.5x over its own C=1 no-draft decode (13.06). Different draft configs, so this isn’t a clean comparison (flagged in the table above), but the shape is real: switch-model.sh’s own production comments note suffix decoding is tuned for low-concurrency interactive latency and “deadlocks on XPU at C≥16” — consistent with per-request tree-verification overhead not scaling once the GPU is already compute-saturated by concurrent streams.
• That also means “old stack decode is 8.8 tok/s” undersells old-stack production on the right workload. The 8.83 figure is the no-draft, isolate-the-variable number used for the head-to-head table above. Actual old-stack production runs suffix decoding, and on repetitive content the April 16 post measured suffix+FP16 hitting 32.44 tok/s at 32K — the suffix tree loves repeated system-prompt padding. So the honest decode comparison depends heavily on the prompt: on fresh, non-repetitive content the new stack’s flat ~13 tok/s likely wins outright; on a stable system prompt or a cached document, the old stack’s suffix decoding could still be competitive or ahead, at 40% less KV headroom. Neither side had a speculative-decoding story on the new stack at the time; ngram closes part of that gap as of the next day (see the Update below), though it’s prompt-lookup rather than a trained draft model, and this specific repetitive-content/C=12 comparison hasn’t been rerun with it.
• The integration cost dropped, but didn’t hit zero. TurboQuant needed six historical fixes on the old stack (a Python 3.12 find_spec deprecation, dataclass Field.type patching, worker re-import, a cwd-shadowed import, a missing backend-registry enum, and a --max-num-seqs KV-sizing footgun). The new stack needed four, one of them a source patch. Less debt, not no debt.

The tradeoff, and the call I made

Given this, I migrated production the same day I finished collecting the numbers. Three things tipped it:

1. The prefill win changes what’s usable interactively, not just what’s fast on a benchmark. A cold 8K-token prompt going from ~14.5s to ~0.7s is the difference between “wait” and “instant” for the RAG and coding-assistant sessions that are most of what this box actually serves.
2. The stack-maintenance column isn’t close. Six historical TurboQuant integration fixes plus a hand-maintained GPTQ patch is an ongoing cost against every future vLLM upgrade. One bind-mounted marlin_utils.py against a rolling upstream branch is a cost I can plausibly stay ahead of.
3. Nothing broke that matters for the actual production workload. Tool-calling (the hermes parser, function-calling for the coding-assistant setup) came back clean after cutover — verified against a live get_weather-style request, not assumed. Both the old served-model-name (qwen3-30b-tq-256k) and a new one (qwen3-30b-a3b-w4a16) are served, so existing client configs didn’t need to change.

What I gave up: TurboQuant’s KV headroom (~40% less aggregate cache, single-request ceiling 256K → 131K) and suffix speculative decoding, which has no equivalent on the new stack yet. If a workload leans on suffix decoding’s edge on repetitive content — the April 16 post’s 32.44 tok/s at 32K on padded system prompts — this migration is a step back for that specific case until spec decode lands upstream. For this box’s actual traffic — interactive chat and coding-assistant sessions, not long repetitive-context batch workloads — the prefill win dominates and the decode/KV costs are acceptable. Both of these aged fast: the ceiling went back to 262K and ngram speculative decoding shipped within 24 hours (see the Update below). Ngram isn’t suffix, though — it closes part of the spec-decode gap, not all of it, and the repetitive-content case above is still unmeasured on the new stack.

Rollback isn’t theoretical, but it’s worth separating two different commands here. node_dark.sh restore brings production back after a bench-cycle drain (not a version rollback) — that’s what actually ran, repeatedly, restoring production between benchmark runs this session. The rollback to the TurboQuant stack is a different procedure: restore docker-compose.yml.tq-rollback + .env.tq-rollback, then switch-model.sh qwen3-30b-tq-256k, then docker compose --profile gpu up -d --force-recreate vllm — shown in full in the Repro section, and proven working this same session (450s to healthy). It just wasn’t needed for this call.

Honest limits

• The image is vLLM 0.21.1.dev17+g0a4756bb5, not 0.23. intel/vllm:latest tracks a rolling dev branch. This entire spike started under a “0.23” working title before the first boot corrected it — noting it here in case that number is floating around anywhere else in this project’s history.
• XMX/DPAS dispatch is not ISA-verified. The strings "xmx" and "dpas" don’t appear anywhere in this image’s Python source. Confirmed: the upstream XPU FLASH_ATTN backend dispatches — not an eager/SDPA/CPU fallback — unforced by fp8 KV, and platforms/xpu.py calls the kernel “FMHA sycl-tla kernels.” Not confirmed: ISA-level proof that the compiled vllm_xpu_kernels binary is issuing dpas instructions. Calling it XMX/DPAS by the kernel library’s design, not by verification.
• Both stacks run on a patch. The old stack needs gptq.py to boot on XPU at all. The new stack needed a bind-mounted marlin_utils.py carve-out because check_moe_marlin_supports_layer() excludes ROCm but not XPU, routing W4A16 MoE to a CUDA-only Marlin kernel that isn’t compiled for XPU. Neither is pure-stock.
• fp8’s aggregate KV budget is ~40% smaller than TurboQuant’s, and this deploy’s single-request ceiling dropped from 256K (262,144) to 131K (131,072) that day. That 131,072 cap was this spike’s deliberate conservative setting, carried into production — not a proven hard limit, and it didn’t stay a limit for long: pushed to the model’s full native 262,144 the very next day, KV pool unchanged (see the Update below). The aggregate budget itself (270,592 tokens) is unchanged either way — that’s the part of this tradeoff that’s actually load-bearing, not the per-request ceiling.
• No speculative decoding on the new stack, as of this writing. Every new-stack number above is draft-none. The C=12 aggregate comparison (110.92 vs. 15.21 tok/s) pits a no-draft config against old-stack production suffix decoding and is explicitly not apples-to-apples. Resolved the next day — ngram speculative decoding shipped to production; see the Update below.
• Stack-vs-stack, not an isolated attention-kernel benchmark. Attention backend, weight quant, and KV scheme all changed together; the measured deltas can’t be cleanly attributed to FlashAttention alone.
• C=1 is the primary story; exactly one C=12 aggregate cell per stack. No broader concurrency sweep.
• 5-prompt samples, not 16. Same noise caveat as the April 16 post — expect roughly ±10-15% at this sample size.
• Old stack’s 131,072-context prefill and decode rows are CITED, not re-run this session — the known ~20-minute O(N²) cold-eager-prefill wedge risk. The new stack’s largest measured context is 122,880, not literally 131,072 — close, not identical.
• Idle-node protocol, stated plainly: drained production via docker compose stop vllm open-webui (health-drain — the cluster router’s health check fails and reroutes), never iptables -j REJECT (a prior session found that triggers a reconnect storm that loads the host CPU and skews prefill timing). Verified idle before benching: connections on host :8000 dropped to only loopback TIME-WAIT entries, the legacy 192.168.0.62 feeder showed zero established connections, and host load average was quiescent. Old and new stacks were benched sequentially on dark port 8001, never concurrently, and production was restored on 8000 — then migrated — as an explicit exit step, not left drained.
• XPU-specific. Ratios here won’t transfer to other hardware. Battlemage-specific fragility this project has documented before (SIGABRT on B-series model inspection, TP=2 GP-fault) wasn’t triggered by this workload — that isn’t the same as saying it’s fixed.

Repro

Drain / verify / restore (scripts/node_dark.sh):

cd /apps/b70-vllm/turboquant-xpu/.worktrees/vllm-023-spike
scripts/node_dark.sh drain     # docker compose stop vllm open-webui on :8000
scripts/node_dark.sh verify    # confirms ~0 conns on :8000, legacy feeder quiet, load avg low
# ... bring the stack-under-test up on dark port 8001, bench, tear down ...
scripts/node_dark.sh restore   # brings current prod stack back up (compose up), polls /health up to 12 min — not a version rollback

Old stack on 8001:

cd /apps/b70-vllm
./switch-model.sh qwen3-30b-tq-256k
VLLM_PORT=8001 VLLM_SPEC_CONFIG='' docker compose --profile gpu up -d --force-recreate vllm

New stack on 8001 (deploy/docker-compose.vllm023.yml + deploy/.env + deploy/vllm023.env):

cd /apps/b70-vllm/turboquant-xpu/.worktrees/vllm-023-spike/deploy
docker compose -f docker-compose.vllm023.yml up -d

Bench harness (scripts/bench_vllm023.py — new; splits prefill and decode instead of conflating them like bench_c1_context.py):

.venv-sycl/bin/python scripts/bench_vllm023.py --stack old --metric prefill \
  --endpoint http://localhost:8001 --model-alias qwen3-30b-tq-256k \
  --ctx 8192 --n-prompts 5 --output docs/tuning/c1_vllm023_old_2026-07-04.txt

.venv-sycl/bin/python scripts/bench_vllm023.py --stack new --metric aggregate \
  --endpoint http://localhost:8001 --model-alias qwen3-30b-w4a16 \
  --ctx 8192 --concurrency 12 --max-tokens 200 --note nodraft,C12 \
  --output docs/tuning/c1_vllm023_new_2026-07-04.txt

Chart: scripts/plot_vllm023_compare.py --inputs docs/tuning/c1_vllm023_old_2026-07-04.txt docs/tuning/c1_vllm023_new_2026-07-04.txt --out site/assets/vllm023_compare_2026-07-04.png.

Raw data (committed): c1_vllm023_old_2026-07-04.txt / .kvlog, c1_vllm023_new_2026-07-04.txt / .kvlog.

Rollback, proven working during this same session (450s to healthy):

cp /apps/b70-vllm/docker-compose.yml.tq-rollback /apps/b70-vllm/docker-compose.yml
cp /apps/b70-vllm/.env.tq-rollback /apps/b70-vllm/.env
cd /apps/b70-vllm && ./switch-model.sh qwen3-30b-tq-256k
docker compose --profile gpu up -d --force-recreate vllm

Full deployment source: github.com/bryanvine/turboquant-xpu. The upstream Marlin-MoE carve-out patch lives at patches/vllm023_upstream_fixes/marlin_utils.py in the worktree (mirrored to production at /apps/b70-vllm/vllm-patches/marlin_utils.py, host-specific, not in-repo).

Update (2026-07-05): pushing to 262K and settling speculative decoding

Two of the items flagged above in “Honest limits” and “What’s next” didn’t survive 24 hours as open questions. The conservative 131K ceiling got tested properly, and the “no speculative decoding” gap got closed — though not cleanly on the first attempt at closing it.

The ceiling was self-imposed, not measured

131,072 was flagged above as “this spike’s deliberate conservative setting… not a proven hard limit,” with the fp8 pool already sized at more than double that. It held up under an actual test: --max-model-len went to 262,144 — Qwen3-30B-A3B’s native max context (~270K) — and the stack booted and served cleanly, no code changes required.

[cache.py:261] Using fp8 data type to store kv cache. It reduces the GPU memory footprint...
[xpu.py:96] Using Flash Attention backend.
[flash_attn.py:641] Using FlashAttention version 2
[kv_cache_utils.py:1710] GPU KV cache size: 270,592 tokens
[kv_cache_utils.py:1711] Maximum concurrency for 262,144 tokens per request: 1.03x

The number that matters here is the one that didn’t move: the KV pool is still 270,592 tokens — identical to the figure measured at the old 131,072 max-len earlier in this post. --max-model-len sets the per-request ceiling, not the pool size; the pool is fixed by --gpu-memory-utilization and the model/KV-dtype footprint, independent of it. So the “single-request ceiling 256K → 131K” framing above is now stale: the new stack’s ceiling is 262K, matching the old TurboQuant stack’s, at 1.03x concurrency for full-length requests. The real, still-standing gap between the two stacks is what it always was — the ~40%-smaller aggregate pool (270,592 vs. 453,184 tokens) — not the per-request cap.

Speculative decoding: three tried, one ships

• suffix — doesn’t boot. SpeculativeConfig.__post_init__ hard-fails at config validation (config/speculative.py:769), before the engine starts: ImportError: Arctic Inference is required for suffix decoding. Install via 'pip install arctic-inference==0.1.1'. The package isn’t in intel/vllm:latest. Not an XPU or FlashAttention incompatibility — a packaging gap.
• EAGLE3 — a dead end at both context settings tried. At 262,144 it doesn’t boot at all: loading the draft model brings total model memory to 16.06 GiB and leaves only 11.74 GiB free for KV cache (down from 12.39 GiB with no draft), and vLLM’s own pre-flight check refuses to start a config that can’t fit one full-length request — it needs 12.25 GiB and estimates a max feasible len of 251,200. Retried at 131,072, where it does boot — and decodes slower than no-SD (12.04 vs. 12.79 tok/s): draft acceptance runs 40% on short prompts but collapses to 0.0% at 8K context, almost certainly because the draft’s own config.json caps max_position_embeddings at 2048 and it’s being fed positions far outside that range. Output stays correct regardless (a rejected draft just falls back to the target model), but a 0%-acceptance draft is pure overhead.
• ngram — works, and costs nothing: it’s prompt-lookup, no draft weights, no draft KV, so the pool stays at the full 270,592 tokens regardless of whether it’s enabled. Whether it’s worth shipping turned out to depend entirely on concurrency.

The concurrency curve, and nearly shipping the wrong answer

Aggregate tok/s with --max-model-len set to 262,144, actual prompt context 8,192 tokens, warm/steady-state. (The literal first request after a cold boot pays a one-time ~20s JIT/compile tax that has nothing to do with speculative decoding — measured, confirmed via a warm re-run, and controlled for symmetrically on both sides below.)

ngram forces two things at dispatch time, unconditionally: Async scheduling not supported with ngram-based speculative decoding and will be disabled, and max_num_scheduled_tokens is set to 2048. Neither costs anything at C≤4 — a decode step at that concurrency needs on the order of tens of scheduled token-slots, nowhere near the cap. But it’s the same mechanism either way, and once concurrency is high enough for async-scheduling-off and the 2048 cap to actually bind, the win inverts hard: by C=12 the scheduling tax outweighs ngram’s per-token benefit by a wide margin. The shape is clean and monotonic — +24% → +14% → +12% → +2% → −45% — exactly what you’d expect from a roughly fixed per-token speculative win competing against a scheduling cost that scales with concurrency.

That shape is also why this nearly shipped the wrong config. The first pass at this decision measured exactly two points — C=1 and C=12 — saw the C=12 number, and came within a hair of shipping no speculative decoding at all, on the strength of a single high-concurrency cell. The gap: this node’s actual traffic is C=1 through C=4, usually C=1, and effectively never C=12. That’s precisely the regime where ngram wins at every measured point — and wins biggest exactly where most of the traffic lives (C=1, +24%). The lesson isn’t “ngram is good” or “ngram is bad.” It’s measure at your actual concurrency, not a convenient round number. C=1 and C=12 were convenient to bench; they aren’t what this box serves.

Final config

Production now runs 262K context + ngram speculative decoding — --speculative-config '{"method":"ngram","num_speculative_tokens":5,"prompt_lookup_max":4,"prompt_lookup_min":2}' — on top of the same W4A16 + fp8 KV + FlashAttention stack described above. Post-change verification: tool-calling still comes back clean (a get_weather-style call returns a well-formed finish_reason:"tool_calls"), and the KV pool is unchanged at 270,592 tokens, 1.03x concurrency at the full 262,144 ceiling — ngram genuinely costs zero pool capacity, confirmed rather than assumed. Everything else in the production command is byte-identical to the config described earlier in this post; ngram’s config block is the only addition. A rollback snapshot (docker-compose.yml.pre-ngram) exists on the same pattern as the TQ rollback in Repro, and wasn’t needed.

Follow-up (2026-07-05): the newer hybrid models — Qwen3.5 & 3.6

The whole premise of this migration was riding whatever frontier move a B70 can actually ride. So the obvious next question: now that we’re on upstream, can the upgraded stack run the newer open models — Qwen3.5-35B-A3B and Qwen3.6-35B-A3B? Both are hybrid Gated-DeltaNet + sparse-MoE architectures (linear-attention layers interleaved 3:1 with full attention), and getting Qwen3.5 to run at all was a 17-task custom-image project back in April. Short version: one of them works, neither is worth swapping to, and the reason is instructive.

The good news: native GDN on XPU is real now

In April, the Gated-DeltaNet layers had no XPU dispatcher — they fell back to a Triton/FLA path that produced gibberish, and getting Qwen3.5 to ~7 tok/s took a hand-built 0.19.1rc0 image with an unmerged kernel PR. That work is now upstream: intel/vllm:latest ships torch.ops.vllm.gdn_attention_core_xpu, a native Battlemage GDN kernel wired into the model’s forward_xpu. (The boot log’s Using Triton/FLA GDN prefill kernel line is a red herring — the XPU platform hook reassigns the forward method to the native op right after it prints.) Confirmed working by Qwen3.5 producing coherent output — the thing April hand-built now comes in the box.

Qwen3.5-35B-A3B (official GPTQ-Int4): works, but slow and fragile

The official Qwen GPTQ checkpoint is a genuine multimodal build — text weights, fp16 vision tower, MTP draft layers — so it loads on the stock, fully-wired Qwen3_5MoeForConditionalGeneration class with zero code patches. It runs, and it’s coherent (it’s a thinking model — emits a visible reasoning trace before answering). But it took a specific, constrained config to get there:

• --enforce-eager is mandatory. With torch.compile on (the production default), boot hangs — one core pegged for 20+ minutes compiling the GDN splitting_ops, never reaching serving. Eager mode boots, but forfeits the compile speedup that the 30B stack enjoys.
• No prefix caching. Enabling it forces an experimental “align” mamba cache mode whose state-copy path crashes on this build (get_mamba_state_copy_func() returns None → TypeError: object of type 'NoneType' has no len()).
• GPTQ is fp16-only (it rejects bfloat16 outright) and boots with a "4-bit gptq_gemm kernel is buggy, switch to gptq_marlin" warning — and Marlin is CUDA-only on this card. It runs anyway; that’s just not a reassuring banner.
• ~5.4 min cold boot, and only 6.46 GiB left for KV after the 23 GB of weights + the vision tower.

Throughput at the concurrency this box actually serves (C=1–4):

3.5’s aggregate cells are measured at 2K context vs. the 30B curve’s 8K, so those two columns aren’t a clean overlay — but the single-stream C=1 decode, the cleanest comparison, is unambiguous: ~7.3 vs ~13 tok/s, roughly half. That’s the --enforce-eager tax, mostly — the 30B gets torch.compile; the hybrid can’t.

Qwen3.6-35B-A3B (community CT-INT4): boots, generates garbage

3.6 is the newer model and the one I actually wanted. The only checkpoint on disk that could run on XPU is a community compressed-tensors INT4 quant, and it turned out to be a text-only-weights build wearing a multimodal config. Upstream’s text-only Qwen3.5/3.6 classes are only half-wired, so getting it to boot took a stack of patches: register the text-only class in the model registry, tag it IsHybrid (so mamba_block_size gets set at all), add the two get_mamba_state_*_from_config classmethods the hybrid-KV setup calls, and implement get_mrope_input_positions (Qwen3.5/3.6 use M-RoPE even for text). After all that it boots, profiles, and generates fast — 80 tok/s — but the output is word-salad ("om garò prezi domesticeder ò…"). fp16 vs. bf16 changed the flavor of garbage (!!!! → word-salad) but not the fact of it; keeping vs. stripping M-RoPE made no difference. Weights load with no missing-tensor warnings, so it’s either a bad community quant or a subtle fault in the text-only load path — and I couldn’t isolate which, because the only other 3.6 checkpoint available is a GGUF, which doesn’t run on XPU at all. Unresolved, and not deployable with the weights on hand.

The call: kept the 30B

Both hybrids share one disqualifying trait for this box: the torch.compile hang forces --enforce-eager, which roughly halves decode throughput versus the compiled 30B. Stack that on top of 3.5’s thinking-model verbosity (more tokens per answer, so felt latency is worse than the raw tok/s gap already shows), its GPTQ fragility and tiny KV budget, and 3.6 being outright broken — and there’s no version of “swap to a 35B” that comes out ahead for a C=1-dominant interactive box that cares about latency. The intelligence edge of a newer 35B-A3B is real, but it doesn’t buy back half the speed plus the operational tax.

Same lesson as the speculative-decoding curve above, from the other direction: the 30B wins where the traffic actually lives (C=1–4). The migration’s verdict stands — the 30B stays.

Two things would flip it, and I’ll re-check both on the next image bump: a bf16/W4A16 (non-GPTQ) Qwen3.5/3.6 checkpoint that loads via the stock multimodal class (no text-only patch surface, no fp16-only constraint), or torch.compile no longer hanging on the GDN layers — which would hand the hybrids their missing ~2x and likely put a 35B-A3B ahead of the 30B outright.

Follow-up (2026-07-06): the speculative-decoding call was wrong — EAGLE-3 ships

The Update above wrote EAGLE-3 off as “a dead end at this model’s context lengths” and shipped ngram — with one hedge buried in the diagnosis: acceptance “collapses to 0.0% at 8K context, almost certainly because the draft’s own config.json caps max_position_embeddings at 2048.” That hunch was right. It was also the entire story, and chasing it down flipped the production decision.

“~0% acceptance at long context” was a config bug, not a model limit

Put EAGLE-3 back on an isolated port, swept draft acceptance against prompt length. It isn’t a slope, it’s a cliff:

Acceptance drops to exactly zero the moment the prompt crosses 2048 tokens, and stays pinned there. A weak draft still lands the occasional common token; zero across ~1,000 proposals is a switch flipping off. The draft’s config.json says max_position_embeddings: 2048, vLLM honours it, so past position 2048 the draft’s RoPE is garbage and the target rejects 100% of it. (The head’s README even ships a SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 escape hatch — a known sharp edge, just not one vLLM lifts for you.) And it is not chunked prefill: raising --max-num-batched-tokens to 32,768 so a 25K prompt prefills in one chunk changed nothing. The cliff tracks absolute context, not batching.

The fix: one field

Copy the head, set max_position_embeddings to the target’s 262K ceiling, point --speculative-config at the copy. No training, no long-context draft, no kernel work. Acceptance returns immediately — flat ~1.0–1.25 out to 20K, then a graceful decay at 50–74K (the genuine “this head was trained at short context” effect, but irrelevant here — see the traffic below). And it’s nearly free: the draft trims the KV pool from 270,592 to 265,664 tokens, ~1.8%. The old “EAGLE eats your pool” worry was real only for the 262K boot that couldn’t fit draft-weights-plus-a-full-length-request; sized normally, the 366 MB head barely registers.

What actually favors ngram vs EAGLE-3 (I had this backwards)

Before replaying real traffic I predicted ngram would win, because the traffic looked repetitive — the same few automated system prompts all day. Wrong axis. The rule isn’t “repetitive input → ngram.” It’s:

Does the output echo the input?

ngram is prompt-lookup — it can only propose tokens it finds in the context. On code continuation, where the output copies identifiers and boilerplate already on screen, it drafts 3–5 tokens at a time for free and beats EAGLE handily. On novel generation — a fresh answer, a sentiment verdict, a JSON object that isn’t sitting in the prompt — it has nothing to copy and stalls near one token, while EAGLE-3 (predicting from the model’s own hidden states) keeps drafting. Starkest single case in a content sweep: “write an original fable” → EAGLE 0.76 vs ngram 0.062. The shape of the input isn’t the question; whether the output is in there to be copied is.

Measuring it: a day of real traffic

Rather than argue from synthetic prompts, I put a fail-open capture proxy in front of the server — tees every /v1 request body to disk, forwards to vLLM, and if anything in the capture path throws it forwards anyway (never break prod for a measurement). A full business day: ~1,900 real requests, and the shape was its own surprise — ~99% automated structured-JSON classification (financial-news sentiment, sports-betting research; all response_format: json_object or guided_json), every one under 2K context, and exactly two human chat messages all day. Then I replayed a sample through both configs verbatim — same messages, same json_object/guided_json, same temperatures — so it’s real work, not normalized toy prompts. Faithfulness check: ngram’s replayed acceptance (1.00) matches ngram measured live on the full day (1.045).

The result, and the switch

EAGLE-3 wins 1.6x at C=1 and 1.4x at C=4 — the margin narrows under load (both batch better as concurrency climbs) but never flips, and acceptance holds flat at ~1.8 across the range. The “repetitive input → ngram” instinct was exactly backwards here: the inputs repeat, but the outputs — the JSON verdicts — are novel every time, which is EAGLE’s court. Production now runs EAGLE-3: the patched draft on a stable path, --enforce-eager (same torch.compile caveat as the hybrids, but here it costs far less than the ~2x it buys), everything else byte-identical to the ngram config it replaced. Rollback to ngram is one docker compose up against the saved snapshot.

Two things I got wrong

Both were decisions, not just facts, so worth stating plainly:

1. I wrote EAGLE-3 off as “a dead end at this model’s context lengths.” It wasn’t the model or the context — it was a four-digit number in a JSON file. The 0%-acceptance symptom was load-bearing and I’d stopped one diagnostic step short of the cause.
2. I predicted ngram would win the real traffic, and it lost — because I reasoned from the shape of the input instead of the output. The replay is the only reason the right config shipped; the prediction would have kept the wrong one.

The ngram concurrency curve’s lesson was “measure at your real concurrency.” This one’s is adjacent: measure on your real traffic. The synthetic sweep got the mechanism right (novel output favors EAGLE); only the captured day got the weighting right — this box’s traffic is almost entirely novel output.

What’s next

1. Speculative decoding on the new stack — resolved, then re-resolved. ngram shipped first (see the 07-05 Update), but the 2026-07-06 follow-up above reversed it: the “EAGLE3 dead end” was the draft’s max_position_embeddings: 2048 cap, a one-field fix, and on a captured day of real traffic EAGLE-3 beat ngram 1.4–1.6x and now ships in production. Still open: suffix decoding, which needs a derived image with arctic-inference baked in (this image doesn’t carry it) — untested against EAGLE-3, and only likely to matter if the traffic mix ever shifts toward output-copies-input (code editing, RAG-quoting), where prompt-lookup regains its edge.
2. Whether to reclaim context if the 270,592-token aggregate budget turns out to bite. The single-request ceiling is now the model’s own native max (262K — see the Update above), so there’s no higher per-request context left to chase on this model; the only remaining lever is aggregate concurrency headroom (1.03x at full-length requests, as deployed). XQuant — the KV-compression idea parked from the never-built April 26 “nextgen stack” series — is the fallback if a real workload needs more concurrent full-length requests than that. fp8 already halves KV cost against fp16; XQuant would be stacking a second compression layer on top of it, the way TurboQuant did for the old stack.
3. Push --max-model-len closer to the measured fp8 ceiling — done. It goes all the way to 262,144, the model’s native max, with the KV pool unchanged (see the Update above). No further headroom to chase on this specific model; a higher single-request ceiling would need a different model or a context-extension trick, not just a flag change.
4. Re-check Qwen3.5 / 3.6 on the next image bump. The native GDN kernel works now (see the follow-up above); the blockers are the torch.compile hang that forces --enforce-eager (~halving throughput) and, for 3.6, a checkpoint that generates garbage. A bf16/W4A16 non-GPTQ checkpoint that loads via the stock multimodal class, or a compile path that doesn’t hang on GDN, would make a 35B-A3B genuinely competitive with — probably faster than — the 30B.

Fourth post in the Arc B70 series. Repo: github.com/bryanvine/turboquant-xpu. Prior posts: April 14, April 15, April 16.

• Deployed Qwen3-30B-A3B + TurboQuant k3v4_nc + speculative decoding on the Arc Pro B70 (32 GiB Xe2 Battlemage, vllm-xpu 0.19). Six integration fixes were needed before the stack would serve a single token, and one more fused-kernel finding turned up during end-to-end validation.
• EAGLE3 + TurboQuant works. The prior BENCHMARK_QWEN3_30B.md had flagged this combination as “unintegrated — would require the draft model’s attention backend to also support TQ cache reads.” The real answer: the draft reuses a separate FP16 cache path; nothing special needs to happen on the TQ side for EAGLE3 target-model verification to succeed.
• The fused-N_spec Triton kernel has a correctness regression on XPU. It compiles, the micro-bench’s numerical check passes, but in real deployment it produces garbage tokens. Details below. Kernel is gated off in the production config (TQ_USE_FUSED_SPEC=0) pending a follow-up fix.
• C=1 throughput matrix across {suffix, EAGLE3} × {TQ k3v4_nc, FP16} at 8K and 32K contexts below. TL;DR for picking a config: FP16 wins throughput, TQ wins memory ceiling, suffix wins when the prompt is repetitive, EAGLE3 wins when acceptance has to hold up on prose — pick your tradeoff.
• Third post in this Arc B70 series. Previous posts: the first SYCL PoC + 2× Triton fix, three SYCL attempts and the gap to Triton.

Why C=1 matters on B70

The B70 saturates around C=12–20 regardless of preset, so production scale-out isn’t the interesting question on this silicon. What is interesting is how much context a single user can work with — RAG windows, long documents, multi-turn coding sessions. Gemma4-31B + suffix at FP16 holds 88K ctx. TurboQuant promises ~8.5× KV capacity on Qwen3-30B-A3B thanks to uniform head_dim=128 + strong GQA + MoE sparsity. If that compounds with speculative decoding, we should be able to serve 128K–256K single-user contexts with headroom.

This post is about the gap between that promise and what actually shipped.

The six integration fixes

vllm-xpu 0.19 predates the upstream TurboQuant PR (#38479). The turboquant-xpu/patches/ directory contains the vLLM core-file overlays and the turboquant_register.py monkey-patch module that bridges the two. Six distinct problems had to be solved before the stack would even parse its own arguments.

1. find_module → find_spec (Python 3.12 deprecation)

sitecustomize.py installs a meta_path import hook that patches vllm.config.cache.CacheDType before vLLM reads it. The hook used the legacy find_module/load_module API. Python 3.12 no longer calls that interface during normal imports — it uses find_spec/exec_module. So our hook sat on sys.meta_path but never fired. Argparse kept seeing the pre-patch CacheDType Literal and rejected turboquant_k3v4_nc as an invalid --kv-cache-dtype value. Rewrote the hook to implement find_spec; argparse then recognized the preset.

2. Patching dataclasses.Field.type (not just the annotation)

After fix 1, typing.get_args(vllm.config.cache.CacheDType) correctly returned the TQ-extended Literal, but --kv-cache-dtype turboquant_k3v4_nc still failed. vllm.engine.arg_utils._compute_kwargs reads argparse choices from dataclasses.fields(CacheConfig)[i].type, which is the original Literal captured at class-definition time — patching CacheConfig.__annotations__["cache_dtype"] has no effect on Field objects. Had to walk dataclasses.fields(CacheConfig) and mutate f.type directly.

3. Workers don’t inherit monkey-patches

VLLM_WORKER_MULTIPROC_METHOD=spawn means the engine-core subprocess starts with a fresh Python interpreter. Our sitecustomize.py auto-loads via PYTHONPATH, but turboquant_register.apply_all_patches() doesn’t run in workers unless someone explicitly imports it. The engine core OOM’d on STR_DTYPE_TO_TORCH_DTYPE["turboquant_k3v4_nc"] (a KeyError the main-process patches had fixed but the worker hadn’t). Added a retrying meta_path hook that imports turboquant_register as soon as vllm.config.cache is in sys.modules — works for both the main process and every subprocess worker.

4. import vllm resolved from /workspace/vllm instead of site-packages

The base image’s WORKDIR is /workspace/vllm, which contains a vllm/ subdirectory. sys.path[0]='' means “cwd”, so import vllm found /workspace/vllm/vllm/ before /opt/venv/lib/python3.12/site-packages/vllm/. Our bind-mounts went to the site-packages copy, so the running vLLM ignored them entirely. The backend file was there; Python just never imported it. The entrypoint shim now cd /tmp before exec python, which pushes cwd off the vLLM package path.

5. TURBOQUANT enum missing from the backend registry

vllm.v1.attention.backends.registry._Backend is a string enum. Without TURBOQUANT as a member, _Backend("TURBOQUANT") raises Unknown attention backend. turboquant_register.py patches XPUPlatform.get_attn_backend_cls to return the backend path, but the enum lookup happens earlier. Mounted patches/vllm_mounts/registry.py statically on top of vLLM’s own registry.py. Did the same for five other core files (torch_utils, xpu, attention, attention_config, cache) as belt-and-suspenders so the working state doesn’t depend on whether monkey-patches fired in a given subprocess.

6. --max-num-seqs 4 pre-allocates 4× the per-request KV budget

Gemma4 production runs with --max-num-seqs 4 at --max-model-len 90112, which reserves KV for 4 concurrent 90K-ctx requests. Qwen3-30B-A3B + TQ at --max-model-len 262144 with the same --max-num-seqs 4 reserves ~1M tokens of KV cache. Prefill activations at even 8K tokens then can’t fit in what’s left, and the engine OOMs on the first prompt. Dropping --max-num-seqs 1 (we’re benching C=1 anyway) freed enough memory for prefill. Production deployment at max concurrency would need per-request context caps significantly below 262K.

Benchmark matrix

2×2 matrix at C=1 with 5-prompt amortization and prefix caching enabled. Prompts are the 16-prompt mixed set from BENCHMARK_QWEN3_30B.md (code / math / translation / prose / QA), truncated to 5 for this run. Harness: scripts/bench_c1_context.py. Raw results: docs/tuning/c1_context_sweep_2026-04-15.txt.

Surprises:

• suffix+FP16 hits 32.44 tok/s at 32K context. The repetitive system-prompt padding used to hit the target context size is deeply suffix-tree-friendly — acceptance rate blows up once the tree has seen the padding, and single-request wall time drops. Anyone running a real repetitive-context workload (multi-turn chat against a stable system prompt, RAG against a cached document) may see something similar. For less repetitive workloads, expect numbers closer to the 8K figures.

• EAGLE3 + TQ is the slowest C=1 config at 8K. Draft-model compute overhead + TQ dequant on the target. Neither speculative decoding method comes free on TQ. At 32K, the combo OOMs — the EAGLE3 draft’s own KV cache plus TQ-target prefill activations exceed what’s left after weights.

• TQ’s promised 262K context ceiling never materialized in this session. Not because KV capacity is the binding resource (the KV budget held 543K tokens of TQ cache at --max-num-seqs 1) but because prefill activations for a 32K+ prompt exceed the remaining memory on this particular vllm-xpu image. Prefill buffers are proportional to prompt length, not KV compression ratio. A smaller --max-num-batched-tokens chunk size (chunked prefill is on but defaults are too large) is the likely fix — follow-up work.

The fused-N_spec correctness regression

The fused-N_spec kernel (_tq_decode_stage1_spec in turboquant_xpu.kernels.triton_decode) was the headline of the previous post: 2.04× speedup at the backend-integration layer for k3v4_nc spec-verify, validated against a looped baseline at atol=5e-3, rtol=1e-2. Shipping it uncovered two issues that the micro-bench didn’t catch.

First, the dispatch rate is zero under default suffix params. Suffix decoding with num_speculative_tokens=8, max_spec_factor=2.0 emits q_len=3 consistently when the model’s actual acceptance rate is low (~13-22% per-position on our mixed-prompt set). The fused kernel’s tl.arange(0, N_SPEC) requires N_SPEC to be a power of 2 — Triton raises arange's range must be a power of 2 for N_SPEC in {3, 5, 6, 7}. So the kernel never fires through the normal suffix path. Added a gate in the backend ((q_len & (q_len - 1)) == 0) so non-power-of-2 q_lens fall back to the looped kernel transparently instead of crashing.

Second, when forced to fire, the kernel produces wrong outputs. Bumping num_speculative_tokens=7, max_spec_factor=30.0 forces suffix to emit q_len=8 (7 proposed + 1 verified), which is a power of 2. The fused kernel compiles, the Triton artifact (_tq_decode_stage1_spec.spv) appears in the cache — and the model generates "One, two, three, four, five,!!!!!!!!!!!!!!!..." — garbage tokens. Flipping TQ_USE_FUSED_SPEC=0 while keeping the same suffix params produces the correct "One, two, three, four, five, six, seven, eight, nine, ten, ..." output. The kernel itself, not the dispatch setup, has a numerical regression on XPU.

Why the micro-bench missed it: the bench’s numerical assertion is against a looped-baseline output tensor using torch.allclose(atol=5e-3). That tolerance is generous enough to pass the fused output even when it’s silently skipping or mis-masking positions. The deployment-layer failure mode is the next token being wrong, which compounds quickly — fused kernel is flagged as known-broken on XPU until someone tracks down whether it’s a causal-mask broadcast issue, an FP8 bitcast alignment issue, or something else. On the looped path (q_len handled one at a time with per-token cached_len+n+1 seq_lens) generation is correct.

Bottom line: the 2.04× spec-verify speedup from fused-N_spec is not a shipped optimization on XPU today. What is shipped is the rest of the TurboQuant stack — cache compression, attention backend, suffix/EAGLE3 integration — all of which work correctly at C=1 across the contexts we measured.

Decision matrix: which config to pick at C=1

For an Open WebUI / coding-assistant use case, suffix + FP16 is the best-available single-user config today. For the long-context demo the original spec asked for, the work isn’t done: --max-num-batched-tokens tuning is the next lever, followed by revisiting --max-num-seqs once we know the real prefill ceiling per config.

Honest limits

• C=1 only. Concurrency story is in BENCHMARK_QWEN3_30B.md.
• 5-prompt samples, not 16. Numbers have ±10-15% noise at this sample size.
• Prefix caching active — this is a realistic chat/RAG condition but pessimistic for fresh-prefill-per-request workloads.
• 128K and 256K contexts couldn’t complete this session. Prefill OOM, not model architecture.
• TTFT not reported. The bench’s streaming TTFT measurement was crashing the engine core under some conditions; the data I have is non-streaming wall time only.
• XPU-specific. NVIDIA ratios will differ, particularly for TQ’s dequant cost relative to FP16.
• The fused-N_spec kernel correctness issue is an XPU finding; NVIDIA numerical behavior is presumably still fine.

Repro

Production config writes through switch-model.sh. Four modes for the bench:

cd /apps/b70-vllm
./switch-model.sh qwen3-30b-tq         # suffix + TQ k3v4_nc
./switch-model.sh qwen3-30b-fp16       # suffix + FP16 (65K max-ctx)
./switch-model.sh qwen3-30b-eagle3     # EAGLE3 + FP16 (65K max-ctx)
./switch-model.sh qwen3-30b-eagle3-tq  # EAGLE3 + TQ k3v4_nc (131K max-ctx)
./switch-model.sh gemma4               # rollback to prior production

The compose file pins --max-num-seqs 1 for C=1 bench. For production with concurrency, raise it (with per-request max-ctx sized appropriately).

Bench command:

cd /apps/b70-vllm/turboquant-xpu
.venv-sycl/bin/python scripts/bench_c1_context.py \
  --mode <mode> --contexts 8192,32768 \
  --n-prompts 5 --skip-ttft \
  --output docs/tuning/c1_context_sweep_<date>.txt

Full deployment source: github.com/bryanvine/turboquant-xpu. Integration fixes live in patches/sitecustomize.py and patches/vllm_mounts/. Mount list is in /apps/b70-vllm/docker-compose.yml (not in this repo — that directory is host-specific).

What’s next

Two threads open:

1. Long-context deployment isn’t there yet. --max-num-batched-tokens tuning + maybe a targeted prefill-activation audit is required to actually deliver the 128K+ single-user contexts that TurboQuant’s KV compression makes architecturally possible on 32 GiB silicon. The KV budget is there; the prefill path is where the constraint lives today.

2. Fused-N_spec kernel correctness. The XPU Triton build compiles the kernel, passes the micro-bench numerical check, and produces garbage tokens in deployment. Bisecting the inner loop (causal mask broadcast? FP8 bitcast? per-query scratch register pressure spilling?) is the follow-up. Until then, looped is the default.

Both are concrete next steps for a follow-up post.

Third post in the Arc B70 series. Repo: github.com/bryanvine/turboquant-xpu. Prior posts: April 14, April 15.

• Follow-up to /2026/04/14/spec-decode-intel-arc/ — two more SYCL attempts after the first PoC’s NO-GO. Every number below is at PoC shape: N_spec=8, B=4, Hq=32, Hk=4, D=128, seqlen=8192, causal, cached_len=8184.
• ESIMD via stock 2025.3 xmx::dpas (Option 5, esimd-poc branch): MARGINAL at 186-192 ms. Faster than scalar SYCL (0.88× zc_scalar on k8v4) but 38-60× slower than fused Triton. Ablation: scalar softmax is 55% of wall time, not the matmul.
• joint_matrix + split-KV via intel/llvm nightly (Option 4 phase a, sycl-jointmatrix-splitkv branch, tag phase-a-decision-2026-04-15): NO-GO at 96.921 ms, 30× slower than fused Triton’s 3.229 ms. Full DPAS on Q·Kᵀ and P·V fires, correctness 4/4, perf wall is structural.
• Gap-to-fused progression on k8v4: SYCL scalar 68× → ESIMD 60× → joint_matrix 30×. ESIMD closed ~14% of the original wall-time gap; joint_matrix+split-KV roughly halved what ESIMD left. The 30× that remains is out of reach for phase (b)’s plausible 3-10× combined.
• Three hardware/toolchain findings worth documenting: nightly header requires jm::layout::dynamic on accumulator fragments, joint_matrix_load/store reject private_space via static_assert (forced SLM staging + lane-0 serialization), and — most important — Intel’s Triton XPU backend already emits DPAS, so custom-DPAS isn’t the lever.
• Fused Triton causal at 3.229 ms stays load-bearing for production. Repo at github.com/bryanvine/turboquant-xpu; the three branches (sycl-poc, esimd-poc, sycl-jointmatrix-splitkv) are preserved with decision writeups.

Part 1: Context recap

The original post covered the first SYCL PoC (scalar + DPAS on the sycl-poc branch, NO-GO at commit 796f7df), the Triton profile that exposed a 24% Level-Zero dispatch tax, and the fused-N_spec Triton kernel that closed it — 2.04× on k3v4_nc and 1.07× on k8v4 at the backend-integration layer. That post closed with “thesis untested”: the PoC’s scalar baseline was missing split-KV, SIMD cooperation, and SLM reuse, so showing DPAS on top didn’t prove anything about whether a production-grade SYCL kernel could beat Triton.

Two follow-ups took opposite paths at resolving that ambiguity.

ESIMD (Option 5) sidestepped the toolchain problem. Stock oneAPI 2025.3’s libsycl.so.8 doesn’t have BMG-G31 in get_matrix_combinations() for joint_matrix, but xmx::dpas intrinsics work without needing the joint_matrix entry point. No nightly, no ABI split, no subprocess bridge. If xmx::dpas alone closed the gap, that was the cheapest answer possible.

joint_matrix + split-KV (Option 4 phase a) built what the original PoC skipped. Two-stage kernel with NUM_KV_SPLITS=8 parallel work-items per (b, h_q), portable joint_matrix DPAS for both Q·Kᵀ and P·V, running on the intel/llvm nightly (with the same subprocess-bridge workaround as the first PoC). The feasibility doc’s CUSTOM_KERNEL_FEASIBILITY.md projection was 2.5-4× over Triton; the phase-a gate was set at ≤30 ms wall time as a less ambitious stepping stone.

Both follow-ups were contained go/no-go PoCs — each answered one scoped question and stopped. This post covers both, summarizes what each measured, maps the findings against the feasibility doc’s projection, and closes with the lessons that apply to anyone attempting BMG-G31 kernel work.

Part 2: ESIMD via stock 2025.3 xmx::dpas

Why ESIMD. Stock oneAPI 2025.3’s icpx compiles xmx::dpas<8,8,float,float,half,half> on BMG-G31 cleanly — no nightly toolchain, no LIBUR_LOADER version conflict, no subprocess bridge. The first PoC needed the intel/llvm nightly and its libsycl.so.9 precisely because joint_matrix wasn’t in 2025.3’s get_matrix_combinations() table for BMG-G31; xmx::dpas intrinsics bypass that entry point entirely. If xmx::dpas alone closed the gap, that was the simplest answer possible.

What landed. The pybind module turboquant_xpu_esimd (at sycl/esimd/, built with stock 2025.3 icpx) launches one ESIMD thread per (b, h_q) — for the PoC shape that’s B·Hq = 128 SIMD threads total. K and V tiles live in per-thread registers (simd<half, 2048> fits within the per-thread register budget, trading SLM staging for simpler code). Both Q·Kᵀ and P·V use xmx::dpas<8,8,float,float,half,half>; softmax runs scalar. All 8 correctness parametrizations pass (2 presets × parallel+causal × small+poc shape, tolerance atol=5e-3, rtol=1e-2). Branch: esimd-poc.

Bench table. Causal mode, PoC shape (N_spec=8, B=4, Hq=32, Hk=4, D=128, seqlen=8192, cached_len=8184). Warmup 5 / timed 20 (Triton legs timed 10).

ESIMD beat scalar SYCL by 12–25% — DPAS is doing useful work. It did not meet the plan’s 2× bar over zc_scalar, and it landed 39–60× slower than fused Triton. The plan’s hard GO criterion (ESIMD ≤ 0.5× zc_scalar) is not met; the hard NO-GO criterion (>0.8× on both presets) is also not quite met at 0.75× on k3v4_nc.

The diagnostic — ablation profile. Ablating each component one at a time (disable, measure, restore) against the 190 ms wall time: ~55% scalar softmax, ~48% K dequant, ~47% V dequant, ~51% DPAS Q·Kᵀ, ~31% DPAS P·V. The percentages sum above 100% because ablation doesn’t serialize perfectly — each figure is that component’s measured contribution when it was the one disabled. The reading is unambiguous: DPAS fires, DPAS is not the bottleneck. Scalar softmax and the two dequant paths account for roughly half of wall time each, and they run outside the matrix units entirely.

The mid-PoC 2.4× fix. The initial kernel body was adapted from Task 5’s correctness-only scaffold, which included a lane != 0 return guard — a pattern that makes sense in a correctness test that wants exactly one result but is a disaster in production: 15 of the 16 SIMD lanes in every work-group returned immediately, leaving the hardware 94% idle. Removing the guard cut wall time from 436 ms to ~190 ms, a 2.4× improvement with no correctness change. The lesson: correctness scaffolds accumulate perf disasters by design, and every one of them needs an explicit audit before any benchmark is meaningful.

Decision: MARGINAL, leaning NO-GO. The ESIMD writeup’s own honest-unknowns section flags the structural ceiling: Intel’s Triton XPU backend already emits DPAS. Triton’s fused kernel at 3–5 ms encodes much of what ESIMD+DPAS is supposed to enable, and without a specific piece of ESIMD-only hardware control that DPAS-via-Triton can’t access, the realistic upper bound for a fully optimized ESIMD port is much closer to Triton’s 3 ms than to the 0.5–1 ms that dedicated ESIMD could theoretically reach with perfect register utilization and vectorized softmax. Branch parked with docs/ESIMD_POC_RESULTS.md recording the decision.

Part 3: joint_matrix + split-KV via intel/llvm nightly

Why try again. The ESIMD ablation was instructive: DPAS fires, but the 55% scalar-softmax and dual dequant bottlenecks are structural, not tuning nits. ESIMD deliberately avoided three structural optimizations — split-KV parallelism across work-items, cross-thread SLM K-tile reuse, and vectorized softmax — trading SLM for register-resident tiles to keep the first PoC simple. Phase (a) was scoped as “build the structural prerequisites, measure, decide” with a ≤30 ms gate. That gate is a less ambitious stepping stone than the feasibility doc’s 2.5–4× over-fused projection — the plan explicitly sets a three-band rubric (GO / MARGINAL / NO-GO) with 30 ms as the cutoff for continuing to phase (b).

What landed. Two-stage kernel: stage 1 dispatches NUM_KV_SPLITS=8 parallel work-items per (b, h_q) — each work-item handles its seqlen slice, computes Q·Kᵀ via joint_matrix DPAS, runs softmax, then computes P·V via joint_matrix DPAS, and writes partials to USM; stage 2 is a scalar log-sum-exp reduce over the 8 splits. Toolchain: intel/llvm nightly 2026-04-13 (clang 23). A subprocess bridge keeps the nightly’s libsycl.so.9 out of the torch-XPU process; .venv-jm/ is numpy-only — no torch dependency in the kernel harness. Tag: phase-a-decision-2026-04-15. Execution via superpowers:subagent-driven-development. Correctness: 4/4 parametrizations pass at atol=5e-3, rtol=1e-2 (2 shapes × parallel + causal, k8v4 preset).

Toolchain findings worth writing down. Three API constraints the feasibility doc couldn’t predict — each caught during implementation and each changed the kernel in a concrete way.

jm::layout::dynamic required for accumulator fragments. The nightly’s matrix-unified.hpp binds the accumulator-variant joint_matrix_load and joint_matrix_store signatures to layout::dynamic. Omitting the layout template parameter is a compile error, not a silent default. Caught during smoke-test compilation; fixed by adding the explicit layout::dynamic parameter to every accumulator load and store.

joint_matrix_load/store reject private_space via hard static_assert. Three separate static_asserts in the nightly header all reject access::address_space::private_space with the message “Joint Matrix doesn’t support load from private memory!”. Stack-allocated scratch arrays fail at compile — there is no runtime fallback. Task 7’s kernel swapped every scratch buffer to sycl::local_accessor with local_space address casts. Total SLM for Task 8 = 12.75 KB per sub-group across 9 buffers (2 KB Q, 4 KB K tile, 512 B b_tile transpose, 512 B scores, 4 KB V tile, 256 B P buffer, 512 B b_pv transpose, 512 B acc_in, 512 B acc_out) — well within the 64 KB BMG-G31 budget, but the change is materially structural: lane-0-only SLM fills plus sycl::group_barrier(sg) after each write replace per-work-item private registers, serializing the sub-group on every tile load.

joint_matrix_mad arg order is (sg, D, A, B, C) for D = A*B + C. Verified in the nightly header; confirmed via the smoke test’s max_err=0 against a CPU reference. Mentioned here because older intel/llvm examples had the arg order reversed — silent wrong answers, not a compile error.

Per-task timing progression. Commit-by-commit at PoC shape (N_spec=8, B=4, Hq=32, Hk=4, D=128, seqlen=8192, causal, cached_len=8184, k8v4). Source: docs/tuning/sycl_jm_per_task_timings_2026-04-15.md.

Task 6 → Task 7 is a small regression. DPAS for Q·Kᵀ replaced an 8192-iteration scalar inner loop, but the lane-0-only SLM fill pattern the nightly’s static_assert forced — Q, K tile, and per-d_slice b_tile pre-transpose, each followed by a sub-group barrier — added enough serialized overhead to net-regress by 21 ms. Task 7 → Task 8 is the opposite: DPAS for P·V replaced a 1024-iteration scalar acc[n][d] += p*v[d] accumulation per row per KV block. That inner loop dominated Task 7’s wall time, and replacing it was decisive — a 2.5× drop from 244.3 ms to 96.9 ms.

Bench at PoC shape, k8v4 causal. Source: docs/tuning/sycl_jm_bench_2026-04-15.txt. Warmup 5 / timed 20.

sycl_jm / fused_trit = 30.02×. The phase-a gate was ≤30 ms. Missed by 3.2×.

Decision: PHASE (A) NO-GO. Reference: docs/SYCL_JM_POC_RESULTS.md. At 96.921 ms, sycl_jm is technically in the plan’s 30–100 ms MARGINAL band — but the 30× gap to fused Triton, not the raw millisecond figure, drives the decision. The plan’s MARGINAL-to-GO path requires a specific, plausible route to a large speedup. Phase (b) has five candidate optimizations: SIMD16 cooperation for dequant + softmax, SLM K-tile reuse across queries, keeping mc_out[N_D_SLICES] in fragments across KV blocks (eliminating the acc_scalar round-trip), joint_matrix_apply for in-fragment rescale, and NUM_KV_SPLITS autotune. Published-literature range for each: 1.5–3×. Combined best case: 3–10× — they share overhead, so not multiplicative. Even a 10× combined improvement brings sycl_jm to ~9.7 ms, still 3× slower than fused Triton’s 3.229 ms. Reaching parity requires ~30× from phase (b) alone. That is not achievable from the phase (b) menu. Branch and tag pushed to origin; phase (b) is not scheduled.

Part 4: The diagnostic map

Each attempt closed some of the gap to fused Triton but hit a different ceiling. I put the three attempts side-by-side at the k8v4 preset — the only shape all three legs cover.

The two transitions did not return equal amounts. From zero-copy scalar to ESIMD — swapping the scalar inner products for xmx::dpas register-resident Q·Kᵀ and P·V tile operations — wall time dropped from ~219 ms to ~192 ms, a 14% reduction and a gap compression from 68× to 60×. DPAS fired, the register-resident K and V tiles were coherent, and correctness held across all 8 parametrizations. But the structural work — split-KV parallelism, SLM K-tile sharing, vectorized softmax — was explicitly absent, and the ablation made clear where the 190 ms was going: scalar softmax alone accounted for ~55% of wall time.

The move from ESIMD to joint_matrix + split-KV delivered the much larger structural win. The 8× grid expansion (NUM_KV_SPLITS=8 work-items per (b, h_q)), combined with joint_matrix DPAS collectives for both Q·Kᵀ and P·V replacing their respective scalar inner loops, cut wall time from ~192 ms to 96.9 ms — a 49% wall reduction and a gap compression from 60× to 30×. That is roughly halving what ESIMD left. The task-level timing makes the P·V contribution concrete: Task 7 → Task 8 (adding P·V DPAS) was a 2.5× drop from 244.3 ms to 96.9 ms, while Task 6 → Task 7 (adding Q·Kᵀ DPAS) actually regressed by 21 ms.

The ceilings at each step map to different hardware resources. Scalar SYCL: D=128 FMAs per token in scalar — no matrix unit contact, no SLM reuse, no cross-work-item parallelism. ESIMD: DPAS fires but scalar softmax runs outside the matrix units, and ~128 threads across 32 Xe-cores leaves roughly 4 SIMD threads per core. Per-element simd<> VNNI packing serializes inside each lane. joint_matrix + split-KV: the nightly’s static_assert rejection of private_space forces lane-0-only SLM fills plus a sub-group barrier after each tile write, serializing every load. The per-d_slice acc_scalar round-trip (fragment → private scalar array → back) adds a register-to-ALU-to-register cycle per slice; the acc_scalar[M_TILE=8][D_DIM=128] fp32 array is 4 KB per work-item — enough to trigger register spill on BMG-G31’s 256 KB register file shared across the sub-group.

None of the three attempts were throughput-bound on DPAS. Every ceiling — scalar inner products, scalar softmax, serial SLM fills, acc_scalar round-trips — is non-DPAS work. The feasibility doc’s 2.5–4× projection cited DPAS as the primary lever. All three attempts agree it is not. I keep coming back to that consensus: the lever the feasibility doc identified is not the limiting one.

Part 5: Revisiting the feasibility doc’s 2.5-4× projection

The feasibility doc’s 2.5-4× projection was reasonable given its assumptions. The three attempts disagreed with four of those assumptions. Each optimism source below is grounded in measured evidence.

Reason 1: Baseline anchor moved. The 2.5-4× was against the looped Triton path (~9-14 ms at the time of the doc). Fused Triton — shipped with the original post at 425fc5c — is 3.229 ms on k8v4 causal, a 2.7-4.2× reduction that captured most of the projected gain from a Python→kernel-fusion angle. Custom SYCL now has to beat ~3 ms to show value, not ~10 ms. That is a 3-4× tighter budget than the projection assumed.

Reason 2: Triton on Xe2 already emits DPAS. Intel’s intel-xpu-backend-for-triton lowers tl.dot to joint_matrix operations on Xe2. Custom SYCL DPAS isn’t bringing a hardware capability Triton lacks — it’s bringing finer manual control. The ESIMD writeup said so explicitly: “If Triton’s DPAS is already optimal for this shape, ESIMD’s upper bound is much closer to Triton’s 3 ms than the 0.5-1 ms that dedicated ESIMD could theoretically deliver.” Phase (a) confirmed it from the joint_matrix side: explicit joint_matrix_mad for both GEMMs landed 30× off fused Triton. DPAS is not the lever that moves the number.

Reason 3: Scalar softmax and dequant are the real wall. The ESIMD ablation decomposed wall time at 55% scalar softmax, 47% K dequant, and 47% V dequant — each individually larger than the DPAS contributions (51% Q·Kᵀ, 31% P·V). Phase (a) measured the same thing from the toolchain side: lane-0 serial SLM fills for Q, K dequant, V dequant, and p_buf, plus acc_scalar round-trips for rescale, all pile up faster than the DPAS mads drain. The feasibility doc implicitly assumed DPAS-centric work dominated; the measurement says it’s closer to 50/50, and DPAS isn’t the majority.

Reason 4: Hardware constraints only visible once you hit them. Three surfaced in phase (a). First: joint_matrix_load/store reject private_space via hard static_assert — every scratch buffer has to live in SLM, forcing lane-0-only fills plus barriers, serializing what was supposed to be collective work. Second: acc_scalar[M_TILE=8][D_DIM=128] fp32 per work-item is 4 KB of per-thread live state; Xe2’s per-thread register budget is ~8 KB at full occupancy — probable spill, though I didn’t extract a register report. Third: the lane-0-serialized fill pattern turns a 16-lane sub-group into 1-lane-plus-15-waiting for the non-DPAS stages. The ESIMD PoC hit the same trap in a different shape — the mid-PoC 2.4× fix removed a lane != 0 return guard that was idling 15 of 16 threads; phase (a) hit it again because writing cooperative fills is a rewrite, not a knob.

The 2.5-4× projection assumed (a) the looped-Triton baseline, (b) DPAS as the dominant lever, (c) scalar softmax as negligible, and (d) independent compounding optimizations. All four failed under measurement. The projection wasn’t wrong in spirit — a production-grade SYCL kernel with all five structural wins plus vectorized softmax plus AOT-tuned SLM reuse could potentially beat Triton on Xe2. What the three PoCs established is that no scoped go/no-go milestone inside the phased commitment reached that ceiling: each PoC answered its question and each returned NO-GO. A full production build would be a larger scope — the joint_matrix spec labels phase (c) “open-ended” — and against a fused Triton target that keeps improving, the three attempts never found an intermediate decision point where the numbers justified the next step.

Part 6: Lessons

• Sidestepping toolchain issues can also sidestep the problem. ESIMD stayed on stock oneAPI 2025.3 + xmx::dpas and avoided the nightly/ABI split entirely. It also avoided every structural optimization that gave Triton its edge. Toolchain cost and algorithmic cost aren’t independent; the path of least toolchain resistance often skips the algorithm that would have mattered. Choosing ESIMD for stability was defensible — but the choice bought predictability at the price of the structural optimizations that were the whole point.

• The scalar softmax is the real ceiling. Both the ESIMD ablation and the joint_matrix phase (a) timing point at the same place: non-DPAS compute — softmax, dequant, rescale round-trips — dominates once the matrix contractions are fast. A custom kernel wins when it beats Triton across the whole pipeline, not just the GEMM slices. Any future attempt should scope vectorized softmax into the first PoC, not defer it as phase (b) cleanup. It’s the gate, not the garnish.

• Register budget and SLM discipline compound — in the wrong direction. The acc_scalar[8][128] fp32 stack array in phase (a) is roughly 4 KB per work-item and probable spill territory on Xe2’s ~8 KB per-thread budget at full occupancy. The phase (b) plan’s “keep mc_out in fragments across KV blocks” avoids exactly this pattern. Structural choices aren’t autotuneable; you pay for them whether or not you’ve measured them, and the cost compounds across every iteration of the outer KV loop.

• Correctness scaffolds turn into perf disasters. The ESIMD PoC’s lane != 0 return guard — inherited from the single-lane correctness scaffold — idled 15 of 16 SIMD threads per work-group for the entire benchmark. Removing it cut wall time 2.4× with no correctness change. The guard made sense when validating output; it became a silent performance floor once validation passed. Prune correctness-only guards before running the perf benchmark, not after you’ve already filed the results.

• Publish each negative result with its specific next-step cost. Tag phase-a-decision-2026-04-15 and SYCL_JM_POC_RESULTS.md record the exact phase (b) optimization list with projected per-item speedups. Anyone resuming — or Intel’s team looking at the work — knows what to try and what it would plausibly be worth before committing effort. A negative result without a cost-tagged next step is just a dead end; a negative result with one is a decision tree node.

Part 7: Closing

Three attempts, one pattern: DPAS fires every time, and non-DPAS work is the ceiling every time. Scalar softmax at 55% of wall time, lane-0 SLM fills, acc_scalar round-trips — none of these are what the feasibility doc pointed at, and none of them yielded to DPAS tuning. The fused Triton kernel at 3.229 ms stays load-bearing. The 2.04× on k3v4_nc and 1.07× on k8v4 at the backend-integration layer are the numbers that matter, and TQ_USE_FUSED_SPEC=1 remains the recommended production setting.

Issue #271 on vllm-project/vllm-xpu-kernels is the plausible path that actually beats Triton — a SYCL kernel with vectorized softmax, AOT-tuned SLM reuse, and persistent-fragment accumulators is what none of the three PoCs delivered. The caveat is honest: the three attempts together suggest the path is narrower than the feasibility doc estimated, and the target keeps moving — Intel’s Triton XPU backend’s DPAS lowering keeps improving, which means every iteration of custom SYCL has to clear a bar that Triton raises from below. If Intel ships that kernel, that’s the one to watch.

All three branches are preserved with decision writeups: sycl-poc, esimd-poc, sycl-jointmatrix-splitkv. Repo: github.com/bryanvine/turboquant-xpu.

• TurboQuant KV cache quantization on Intel Arc Pro B70 (BMG-G31, Xe2) trades 3-8x KV capacity for 0.27-0.47x throughput vs FP16. Speculative decoding amplifies the gap: FP16+spec hits 2.37x, TQ+spec only 1.34x.
• I tried a custom SYCL kernel using joint_matrix (DPAS) for M=N_spec=8 verification. First bench: 50-60x slower than Triton — per-call PCIe transfers (~400 MB/call) dominated.
• Rebuilt with zero-copy USM pointers. Gap closed from 50-60x to 20-25x, but the remainder was algorithmic: the scalar baseline had no split-KV, no SIMD cooperation, no SLM reuse. Decision: NO-GO (796f7df); thesis untested.
• Profiling the Triton baseline (b69399a) revealed 24% of wall time in Level-Zero submission overhead across the N_spec loop, 6.2% FP32 compute utilization, and 3.9% BW utilization — not throughput-bound, dispatch-bound.
• A single fused Triton kernel (425fc5c, causal mode c0a69a3) gives a 2.04x backend-layer speedup on k3v4_nc and 1.07x on k8v4. The asymmetry is the point: k8v4’s cheap FP8 dequant leaves almost nothing to share across queries; k3v4_nc’s MSE-centroid gather is expensive per tile and amortizes 8x over the spec window.
• Everything is in github.com/bryanvine/turboquant-xpu. Intel’s team acknowledged TQ on XPU in vllm-xpu-kernels issue #271; a SYCL port is on their roadmap.

Context: TurboQuant, speculative decoding, and the gap

TurboQuant (DeepMind, ICLR 2026; upstream vLLM PR #38479) compresses LLM KV caches to 3-4 bits via a Walsh-Hadamard rotation plus a precomputed Lloyd-Max codebook for keys and uniform scalar quantization for values. On an Intel Arc Pro B70 (32 GB GDDR6, 32 Xe2 cores, 256 XMX engines), the three hot Triton kernels (fused store, decode stage 1, stage 2 reduction) compile cleanly via the intel-xpu-backend-for-triton with every correctness gate passing.

Capacity is excellent. At max-model-len=262144 on Qwen3-30B-A3B, turboquant_k3v4_nc (3-bit MSE keys + 4-bit values + norm correction) holds 549,888 KV tokens in 11.8 GiB versus ~65K for FP16 — roughly 8.5x. On Gemma4-31B it’s 4.83x (10,240 to 49,408 tokens).

Throughput is less happy. On Qwen3-30B at C=20, TQ runs at 141.1 tok/s against FP16+EAGLE3’s 298.5 tok/s — 0.47x. That’s the steady-state cost of bit-unpacking keys and gathering centroids on every attention call, in kernels tuned for Ampere/Hopper and cross-compiled to Xe2.

Speculative decoding is where the picture gets awkward. FP16+suffix on Gemma4 jumps from 51.2 to 121.5 tok/s (2.37x) by verifying N_spec=8 tokens in one forward pass. TQ+suffix on the same model only goes from 27.0 to 36.3 tok/s — 1.34x. Spec helps TQ much less than FP16, and the relative slowdown widens from 0.53x to ~0.30x under spec.

That gap is the target. Two candidates surveyed in the feasibility study: a custom SYCL kernel using DPAS (Xe2’s 2D systolic array) with M=N_spec=8 to match the native tile shape, or a Triton restructure that fuses the N_spec=8 loop into one dispatch. The feasibility doc projected 2.5-4x from the SYCL route. This post is about trying both.

Part 1: The SYCL attempt

The PoC was scoped to be a go/no-go data point, not a full port. Fourteen tasks, three weeks, two variants:

1. SYCL scalar: one work-item per (n_spec, b, hq), online softmax over the sequence dimension, no matrix-unit involvement. Correctness anchor.
2. SYCL DPAS: Q·K^T tiled via sycl::ext::oneapi::experimental::matrix::joint_matrix using 8x16 x 16x16 fp16 tiles targeting the B70’s XMX engines. The design bet: M=N_spec=8 exactly matches the native joint_matrix tile, so the systolic array finally runs at full M-utilization during spec verification (vs M=1 for standard decode).

PoC shape: N_spec=8, B=4, Hq=32, Hk=4, D=128, seqlen=8192. Same as the projected production spec-verify batch.

Toolchain: the libsycl ABI split

Both DPAS variants required the intel/llvm nightly (clang 23, dated 2026-04-13). Stock oneAPI 2025.3’s libsycl.so.8 has a get_matrix_combinations() table that does not list BMG-G31 — joint_matrix_load on a B70 throws “no matrix hardware on the target device” at runtime. The nightly ships libsycl.so.9 with the BMG-G31 DPAS definition included.

The catch: the nightly’s libsycl.so.9 requires LIBUR_LOADER_0.12 and torch-XPU’s libsycl.so.8 requires LIBUR_LOADER_0.11. Strict ABI requirements, no LD_LIBRARY_PATH ordering reconciles them. The workaround was a subprocess bridge: split the benchmark into two child processes with distinct LD_LIBRARY_PATH settings, coordinated by a JSON-passing orchestrator. Triton timings from the torch-XPU leg; SYCL timings from the nightly leg.

Correctness passed. Performance did not.

Every correctness gate went green: joint_matrix_smoke() on 8x16x16 fp16 GEMM, both presets passing test_decode_spec_scalar.py and test_decode_spec_dpas.py at small and PoC shapes. Task 13’s BLK_KV sweep identified BLK_KV=16 as the scalar-path optimum.

Then the go/no-go bench:

The DPAS kernel was slower than the scalar kernel. Both were 50-60x slower than Triton. The go-criterion (DPAS speedup >= 2x) was nowhere in sight.

The root cause was not in the kernel. The Python binding matched the numpy reference’s pack_cache_for_kernel layout, accepting host-side numpy arrays. Each call did sycl::malloc_device plus a blocking memcpy for the KV cache — B * seqlen * Hk * D = 4 * 8192 * 4 * 128 = 16.8 MB per buffer, and with fp32 Q, FP8 keys, norms, scales, and zeros, roughly 400 MB of host-to-device transfer per call. At the B70’s PCIe 5.0 x16 bandwidth that is around 200 ms, exactly what the scalar numbers show. The kernel compute was 1-2 ms buried inside a 240 ms wall.

The Triton baseline, by construction, pre-allocates all XPU tensors once and loops N_spec calls each touching only device-resident buffers. Its 10-14 ms wall is true dispatch latency. I had benchmarked interface overhead against compute, not compute against compute.

Zero-copy: the second data point

Option A was rewriting the binding to accept torch.Tensor.data_ptr() USM pointers, so the SYCL kernel reads from the same device buffer Triton does. This also meant rebuilding the module against stock 2025.3 icpx (not the nightly), so the .so links against torch-XPU’s libsycl.so.8 ABI — the subprocess bridge goes away for the scalar path.

Results after removing every byte of PCIe traffic:

The 50-60x gap closed to 20-25x. Still nowhere near the required >=0.5x. Removing data transfer revealed a structural gap in the kernel itself. Inspecting sycl/zc/src/tq_decode_spec_zc.cpp against the Triton kernel spells out what’s missing:

1. No split-KV. Triton uses NUM_KV_SPLITS=32 to partition seqlen across parallel WGs plus a stage-2 log-sum-exp reduction. The SYCL scalar kernel assigns one work-item per (n_spec, b, hq) and iterates serially over all 8192 positions — ~1024 work-items each doing 8192 sequential FMAs, a long critical path rather than a tree.
2. No SIMD / sub-group cooperation. Each work-item issues scalar FMAs. Xe2’s SIMD16 units have 16 lanes waiting for work the kernel never lets them do.
3. No SLM reuse across queries. Each work-item loads K and V independently from HBM; multiple work-items sharing the same KV head all re-read and re-dequant the same key vectors.
4. No prefetch, no online-softmax pipelining. Triton uses tl.dot with async-load hints and an online softmax tuned for the Xe2 memory hierarchy. The scalar SYCL is a straight textbook online softmax.

The PoC ran a DPAS proof against a scalar baseline that was algorithmically uncompetitive with Triton. Showing DPAS beats a bad SYCL scalar would prove nothing about whether DPAS beats Triton.

Honest verdict: NO-GO, thesis untested

The feasibility doc’s 2.5-4x projection assumed a production-grade SYCL baseline: split-KV parallelism, SIMD16 vectorization over D, SLM-staged K dequant shared across the KV group, and only then DPAS on top for the matrix contractions. The PoC built the DPAS contribution in isolation. The data doesn’t prove the thesis, and it doesn’t disprove it either — it measures a kernel missing the structural work the feasibility doc itself called prerequisite. The only honest path to test the 2.5-4x claim is a production-grade SYCL re-implementation — a 4-5 month commitment that should not start until the libsycl ABI split is resolved upstream.

The PoC did produce concrete deliverables: a working CMake + pybind11 + SYCL build recipe for BMG-G31, environment notes for the SO-version gotchas, 14 passing correctness tests, a subprocess-bridge harness, and confirmation that joint_matrix is accessible on B70 given the right toolchain. Merged to main as 796f7df with a NO-GO flag and a pointer forward.

Part 2: Profiling the actual Triton baseline

The next question was obvious: is Triton actually fast, or just fast relative to our bad SYCL? If the real bottleneck is something a Triton tweak could fix, the 14-task SYCL bet was the wrong tool.

VTune was the plan. VTune 2025.10 reports "This analysis type is not applicable to the system because VTune Profiler cannot recognize the processor" when you point gpu-hotspots, gpu-offload, or xpu-offload at the B70 — BMG-G31 is too new for the 2025.10 PMU database. Fell back to torch.profiler with ProfilerActivity.XPU (PTI-backed via libpti_view.so), wall-clock timing with and without xpu.synchronize(), and roofline math against the B70’s 608 GB/s HBM peak and 8 TFLOPS FP32 peak.

At the PoC shape (N_spec=8, k8v4, seqlen=8192):

Each call to the decode op dispatches 2 kernels (stage1 + stage2), so the N_spec=8 loop is 16 Level-Zero submissions. Stage 2 takes <1% of GPU time — it’s not on the hot path.

Roofline

KV cache read per stage 1 call with slot_size_aligned=196:

2048 blocks * 16 tokens * 4 Hk * 196 bytes = 25.7 MB

At 1,087 us that’s 23.6 GB/s — 3.9% of 608 GB/s. Nowhere near bandwidth-bound.

Stage 1 FLOPs (QK + AV matmuls):

2 * B * Hq * seqlen * D * 2 = 0.54 GFLOP

At 1,087 us that’s 0.49 TFLOPS — 6.2% of 8 TFLOPS FP32. Nowhere near compute-bound either.

Arithmetic intensity is 20.9 FLOP/byte against a ridge point of 13.2 FLOP/byte — nominally compute-side of the roofline, but 6.2% of peak is not a compute-bound workload in any useful sense. It is an EU-underoccupancy workload that happens to sit above the ridge line. The real question is why EUs are idle, not whether arithmetic density is high enough.

The 24% dispatch tax

Breaking out the 8.922 ms N_spec loop:

CPU dispatch and GPU execution overlap (the driver pipelines ahead), so these don’t sum naively. The aggregate is that ~24% of wall time is Level-Zero submission and Python glue around 16 launches — a fixed per-dispatch cost that no kernel-level change can reduce because it exists outside the kernel.

Classification: compute-underutilized / serialized-dispatch. Two structural wins are visible:

1. Eliminate most of the launches. Sixteen dispatches could become two (one stage 1, one stage 2 for all N_spec). Hard lower bound: ~1.22 ms saved (14 x 87 us).
2. Share per-tile work across queries. All N_spec=8 queries attend to the same KV cache; loading and dequanting each K tile once instead of once per query amortizes the bit-unpack and centroid gather 8x.

Neither needs SYCL. Both are single-file changes to the Triton kernel. Commit b69399a for the profile artefacts.

Part 3: The fused Triton kernel

_tq_decode_stage1_spec takes Q of shape [N_spec, B, Hq, D] and processes all queries in a single dispatch. The grid stays (B, Hq, NUM_KV_SPLITS) — N_spec is handled by a loop inside each work-group, sharing the K dequant per BLOCK_KV tile across queries that live together in registers.

The load-bearing structure is about this:

@triton.jit
def _tq_decode_stage1_spec(Q_rot_ptr, KV_cache_ptr, ..., N_SPEC: tl.constexpr):
    bid = tl.program_id(0); hid = tl.program_id(1); sid = tl.program_id(2)
    n_idx = tl.arange(0, N_SPEC)                     # [N_SPEC]

    # Load all N_spec queries for this (b, h) once, stay in registers:
    q_all = tl.load(Q_rot_ptr + n_idx[:, None] * stride_q_nspec
                    + bid * stride_qb + hid * stride_qh
                    + d_offs[None, :], ...)          # [N_SPEC, D]

    m_prev = tl.full([N_SPEC], -float("inf"), tl.float32)
    l_prev = tl.zeros([N_SPEC], tl.float32)
    acc    = tl.zeros([N_SPEC, BLOCK_D], tl.float32)

    for start_n in range(split_start, split_end, BLOCK_KV):
        # ONE K dequant per tile, shared across all N_SPEC queries:
        k_tile = <bit-unpack + centroid gather + norm correction>  # [BLOCK_KV, D]

        # [N_SPEC, BLOCK_KV] = [N_SPEC, D] x [BLOCK_KV, D]^T:
        scores = tl.sum(q_all[:, None, :] * k_tile[None, :, :], axis=2) * ATTN_SCALE

        # Per-query online softmax, vectorized over N_SPEC:
        m_new = tl.maximum(m_prev, tl.max(scores, axis=1))
        ...

The q_all[:, None, :] * k_tile[None, :, :] broadcast sum is the load-bearing piece. It expresses the M=N_spec GEMM without any DPAS-specific intrinsic, but it gives the Triton-on-Xe2 compiler an 8-wide contraction it can schedule against SIMD16 lanes more naturally than 8 separate 1-wide contractions in a Python loop. The same pattern repeats for P·V, with online-softmax state vectorized over N_SPEC: m_prev [N_SPEC], l_prev [N_SPEC], acc [N_SPEC, BLOCK_D], all in registers.

Micro-bench (commit 425fc5c)

PoC shape, same-seq_len-per-query baseline (parallel-completion semantics):

Clears the projected 35-55% reduction by a wide margin. Commit 425fc5c.

Post-fusion profile (d6b6afb)

Running the same profiling script against the fused kernel:

Launch overhead fell from 2.10 ms to 0.20 ms — from 24% of wall time to under 10%. Compute utilization roughly tripled to 14-18% (bandwidth utilization fell, because per-tile K data is now read once and reused instead of reloaded per query — arithmetic intensity jumps accordingly).

Attribution: of the ~5.9 ms k8v4 wall-time reduction, about 1.9 ms (62%) is direct launch-overhead elimination; the remaining ~40% comes from K-dequant sharing across the N_spec window. The per-preset asymmetry this sharing produces matters more at the integration layer — covered in Part 6.

Part 4: The causal correction

With the fused numbers on hand, I posted them to the vllm-xpu-kernels issue #271 thread (Intel’s SYCL team was following). Then I read the integration site more carefully and realized the micro-bench was wrong.

In _prefill_attention, the spec-verify continuation path doesn’t use a single uniform seq_len for all N_spec queries. It synthesizes per-query lengths with synth_seq_lens = torch.arange(cached_len + 1, seq_len + 1). Query n attends to cached_len + n + 1 tokens — the whole point of causal verification is scoring P(token_{n+1} | token_{0..n}, cache) for each n, not P(token_{n+1} | token_{0..N_spec}, cache). The “parallel completion” semantics the micro-bench used — all queries sharing one seq_len — would leak future candidate tokens into earlier queries’ attention.

My kernel was semantically wrong for spec-verify. The numbers were real but measuring the wrong thing.

The fix (commit c0a69a3)

Added a CAUSAL: tl.constexpr mode and a scalar cached_len: tl.constexpr. When CAUSAL=1, compute per-query effective end positions once and mask inside the hot loop:

if CAUSAL:
    eff_end_per_query = tl.minimum(cached_len + n_idx + 1, seq_len)  # [N_SPEC]

# ...inside the hot loop, after scoring:
if CAUSAL:
    causal_mask = kv_mask[None, :] & (kv_offs[None, :] < eff_end_per_query[:, None])
    scores = tl.where(causal_mask, scores, -float("inf"))

CAUSAL=0 stays byte-identical to the original kernel, so the parallel-completion path is preserved. About 6 hours of work including new tests. Causal-mode micro-bench:

Slightly below parallel-completion (2.71x / 4.22x), because the per-query mask adds a broadcast comparison and tl.where inside the hot loop, and because the looped causal baseline does marginally less work on average (earlier queries have shorter context). Still comfortably above the 1.3x threshold I had mentally set for “fusion worth the code.”

Posted a correction to the GH issue. Being upfront about your own mistakes matters more than being right first; Intel was going to look at the code eventually and better for them to see corrected numbers from me than find the bug themselves. Commit c0a69a3; test tests/test_fused_nspec.py::test_fused_causal_matches_looped.

Part 5: Autotune and the register wall

The post-fusion profile suggested projected gains from three knobs: BLOCK_KV 4 -> 16/32 (1.5-2.5x from more instruction-level parallelism), num_warps 1 -> 2/4 (1.2-1.5x from hardware thread interleaving), and NUM_KV_SPLITS 32 -> 8/16 (1.1-1.3x from per-split cache reuse).

A 144-config sweep (BLOCK_KV ∈ {4, 8, 16, 32}, num_warps ∈ {1, 2, 4}, NUM_KV_SPLITS ∈ {8, 16, 32}, across two presets and two causal modes) ran cleanly — zero skips, zero compiler spill errors. Winners:

BLOCK_KV=4 and num_warps=1 won every category. The projected 1.5-2.5x from larger tiles was nowhere. Only meaningful win from the whole sweep: NUM_KV_SPLITS=8 for k8v4 causal (+0.20x, because splits=32 over-spreads at this shape).

The explanation is register budget. The kernel carries q_all[8, 128] (4 KB fp32), acc[8, 128] (4 KB), and scores[8, BLOCK_KV] (512 B at BLOCK_KV=4) in registers — about 8.5 KB of per-thread live state. Xe2’s register file is 64 KB per XVE with 8 hardware threads at full occupancy, giving each thread ~8 KB of effective budget. BLOCK_KV=4 is already at the edge. Doubling to 8 pushes scores to 1 KB; total goes to ~9.5 KB and starts spilling to scratch — throughput collapses even though the compiler reports no error. num_warps > 1 splits the thread budget further with the same effect.

Lesson: projected multiplicative wins from unrelated knobs do not compound when one dimension is register-budget-capped. The sweep can only find what the hardware allows.

Commit 8b4291f, full numbers in docs/tuning/fused_nspec_sweep_2026-04-14.txt. Launcher reads env-var overrides for future A/B testing.

Part 6: Backend integration — the real number

Integrating into _prefill_attention is a small Python patch. TurboQuantAttentionImpl.forward branches on attn_metadata.is_prefill, which is true whenever max_query_len > 1. During spec-verify the scheduler sends query_lens == N_spec, so is_prefill=True and spec-verify routes to _prefill_attention. A continuation-chunk branch there handles q_len <= _CONTINUATION_DECODE_THRESHOLD (128) by calling triton_turboquant_decode_attention in a Python loop with synth_seq_lens = arange(cached_len+1, seq_len+1). That’s the looped path.

The integration patch (commit 9974d8e) adds a gate in that branch:

use_fused = (
    _USE_FUSED_SPEC
    and _FUSED_SPEC_AVAILABLE
    and 1 < q_len <= _FUSED_SPEC_MAX_QLEN  # <= 8
)
if use_fused:
    q_spec = q_seq.unsqueeze(1)              # (q_len, Hq, D) -> (N_spec, B=1, Hq, D)
    sl_single = torch.tensor([seq_len], ...)
    out_spec = _decode_attention_spec_fused(
        query=q_spec, kv_cache=kv_cache, ...,
        causal=True, cached_len=cached_len,
    )
    out = out_spec.squeeze(1)                # -> (q_len, Hq, D)
else:
    # existing looped path, unchanged
    ...

Gated behind TQ_USE_FUSED_SPEC so users can A/B per preset. The looped path below the else: is preserved exactly, so q_len==1 (pure decode) and q_len>8 (outside the fused range — register pressure at N_spec=16 is ~16 KB, past the B70 wall) cannot regress. A new test (test_turboquant_attn_fused_path.py) calls both paths with identical inputs and asserts torch.testing.assert_close(atol=5e-3, rtol=1e-2) — both presets pass with NaN parity.

The backend-layer bench

bench_backend_integration.py times the looped path against the fused path with realistic inputs — single request, q_len=N_spec=8, with the looped path passing the real synth_seq_lens that _prefill_attention actually builds. Warmup 5, N_timed 20 per measurement:

These are substantially smaller than the kernel-alone causal micro-bench (2.75x / 2.99x). The reason: the production baseline isn’t 8 separate kernel calls. _prefill_attention makes one call with B=N_spec=8 and incrementing seq_lens, delegating the per-spec-token dimension to the batch axis of the existing single-query kernel. That already pays the dispatch cost once and exploits cross-query batch parallelism — so the launch-overhead elimination that drove ~60% of the micro-bench speedup doesn’t exist at the integration layer.

What remains is K-dequant sharing. For k3v4_nc (MSE-centroid rotation + norm correction per tile — expensive) sharing across 8 queries is worth 2.04x. For k8v4 (a single FP8 bitcast per tile — cheap) there’s almost nothing to share, so the fused kernel barely beats the batched looped kernel: 1.07x.

Enabling fusion is worth 2x for k3v4_nc — also the preset that matters most, giving 3.7-8.5x KV capacity depending on model. k8v4 is effectively neutral.

What isn’t measured

End-to-end server-layer tokens/sec is not in this post. The running vllm-xpu container currently serves Gemma4-31B GPTQ with FP16 KV cache, not a TurboQuant model — the docker-compose file only mounts the GPTQ XPU regression patch, not the TurboQuant backend files. A full offline engine bench requires a TQ-compatible GPTQ checkpoint on disk plus a compose update and model reload with --kv-cache-dtype turboquant_k3v4_nc. Blocked on checkpoint availability. The backend-layer 2.04x is the honest measurement that’s available today.

Part 7: Working with Intel

Opened issue #271 on vllm-project/vllm-xpu-kernels with the feasibility report: TurboQuant works on XPU via the upstream Triton path, here are the benchmarks, here is what we think is leaving performance on the table, here is a proposed SYCL optimization plan. Intel’s yma11 replied acknowledging TQ on XPU and stating that a SYCL port is on their roadmap (no timeline).

Five follow-up comments over the two weeks: the torch.profiler / PTI profile data with the 24% dispatch breakdown, the parallel-completion micro-bench (4.22x), the causal-mode correction (2.90x), and the final backend-layer numbers (2.04x / 1.07x). Intel has the data if they want it.

The work stayed in our repo rather than going upstream as a PR. The fused-N_spec Triton kernel is complementary to Intel’s future SYCL port, not competing with it — Intel’s SYCL work will eventually replace the Triton decode path on XPU, at which point the fused Triton kernel becomes a bridge rather than a destination. Kept the PR offer open but didn’t block on acceptance.

Part 8: Lessons

Profile before porting. The PoC committed 14 tasks to a custom SYCL kernel when a simpler Triton restructure would have done the job. The 6.2% compute utilization and 24% launch overhead from the first profile would have pointed straight at fusion. The order of operations in this post — SYCL first, profile second — is the order I did it in, not the order I would do it in again.

Validate the baseline you’re beating. The first micro-bench compared against a same-seq_len-per-query baseline. That’s correct for scoring N alternative completions against the same prefix, and it’s semantically wrong for causal spec-verify, which is what the production code actually does. I didn’t notice until I read _prefill_attention carefully. Always run against what the code does in production, not a convenient approximation that happens to share a function signature.

Measure at the boundary that matters. Kernel-alone 2.9x and backend-boundary 2.04x are both real measurements, but only one is what users see. The gap is the production baseline’s batch-axis trick, which wasn’t visible from the kernel interface. A writeup that reports only the kernel number is overselling.

Know your register budget. The autotune found nothing because BLOCK_KV, num_warps, and NUM_KV_SPLITS were all gated on the same per-thread register budget. When projected compounding wins fail to compound, something is gating all of them — usually a single resource like registers, SLM, or dispatch bandwidth. Check the gating resource before running the sweep.

Negative results are worth writing down. The SYCL PoC returned NO-GO. Documenting why — and being specific about what would be needed to actually test the thesis — is more valuable than hiding it. The Triton profile that produced the 2x fix wouldn’t have happened without the SYCL NO-GO making me look harder at the Triton path.

Closing

From NO-GO SYCL PoC to a 2.04x backend-layer speedup for the k3v4_nc preset, in two weeks. The fused-N_spec kernel is one file diff in src/turboquant_xpu/kernels/triton_decode.py, one integration gate in patches/vllm_mounts/backends/turboquant_attn.py, and three commits: 425fc5c, c0a69a3, 9974d8e. The SYCL PoC is on the sycl-poc branch merged at 796f7df, preserved with its NO-GO note for if the libsycl ABI split ever resolves upstream.

The surprise was that the register file, not the arithmetic pipes, was the wall. Launch overhead was the obvious bottleneck and closing it drove most of the gain — but what now gates further fused-kernel wins is a resource that neither VTune nor a roofline plot points at. You learn that kind of thing by running the sweep and watching every knob fail in the same direction.

Intel is planning a SYCL port (issue #271); the Triton win is complementary and will eventually be superseded. Until then, TQ_USE_FUSED_SPEC=1 is a 2x improvement on the preset that needed it most. Repo at github.com/bryanvine/turboquant-xpu with benchmarks and tests reproducible on a single B70.