DSpark with the Vision-Exp support model on Metal: ~half the accepted tokens per cycle of the 0731 pair, a wash in ds4-bench and a net loss through ds4-server at every confidence (M5 Max)

[nazeshinjite]

Summary

With the DeepSeek V4 Flash Vision-Exp target + its own DSpark support model, DSpark on Metal accepts far fewer draft tokens than the 0731 pair does on the same workload on the same machine: ~1.2–1.3 accepted tokens per cycle on pure code (2.27 for 0731), ~0.4 on reasoning text. That is break-even at best (ds4-bench: 38.41 → 38.69 t/s) and a loss through ds4-server (43.06 → 33.8 t/s, −21%, pure code; 44.15 → 39.3, −11%, thinking). The 0731 pair on this box measured +18.8% on the same kind of prompt.

I could not find an existing report with acceptance numbers for the Vision-Exp support model, and QA_BEFORE_RELEASES.md asks Metal DSpark changes to be tested against the Vision-Exp pair without recording expected numbers — so I'm asking whether these figures are expected for the released Vision-Exp head, or whether something is off in the Metal Vision-Exp + DSpark path.

Setup

• Apple M5 Max, 128 GB, macOS 26.6.2, Apple clang 21.0.0; plain make (Metal).
• ds4 6289c51 (current main, unmodified — see "ruled out" for the other builds).
• Target: DeepSeek-V4-Flash-Vision-Exp-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8.gguf (97.6 GB, sha256 cded4517bb9d…c51b02, from antirez/deepseek-v4-gguf; header carries quantize.imatrix.*, deepseek4.checkpoint_variant = vision-exp, general.source.revision = e46e16bf…).
• Support model: DeepSeek-V4-Flash-Vision-Exp-DSpark-support.gguf (6.0 GB, sha256 0807a67f…bc877). Loads cleanly: DSpark support model detected (stages=3 block=5 markov_rank=256), target-hidden capture enabled: layers=40,41,42; the checkpoint/revision pin passes.
• Encoder: DeepSeek-V4-Flash-Vision-Encoder.gguf (--vision), loaded in every server arm; no image in any request.
• Server: ds4-server -m <target> --vision <encoder> --metal -c 262144 --mtp-model <support> --dspark [--dspark-confidence F | --dspark-strict], DS4_DSPARK_STATS=1. All requests temperature 0, max_tokens 1800, one prompt (a ~100-token Python codegen task: KV store with TTL/LRU + pytest tests, "code only"). Baseline arms use --dspark-strict (support model loaded, target-only decode — identical memory layout). Each arm is a fresh server; t/s is the server's own avg= from its decode log; counters from the shutdown DSpark stats line.

Measurements

Thinking on (reasoning_effort: low; the whole 1800-token output was reasoning text on this checkpoint):

arm	t/s	Δ	accept_rate	avg_accept	no_draft	accepted_len_hist
strict	44.15	—
opp conf 0.45	39.34	−11%	37.68%	0.394	997	0:1062 1:84 2:56 3:44 4:17 5:22
opp conf 0.6 (default)	35.75	−19%	43.94%	0.354	1036
opp conf 0.3	33.83	−23%	41.45%	0.626	768

Thinking off (reasoning_effort: none; pure code, finished at natural stop ~1640 tokens):

arm	t/s	Δ	accept_rate	avg_accept	no_draft	accepted_len_hist
strict	43.06	—
opp conf 0.45	33.82	−21%	50.32%	1.156	384	0:428 1:77 2:81 3:66 4:33 5:58
opp conf 0.6	33.23	−23%	54.71%	1.275	320	0:366 1:91 2:89 3:89 4:39 5:45

ds4-bench --metal --chat-prompt-file <same prompt> --ctx-start 100 --ctx-max 100 --gen-tokens 1200 (no thinking):

	gen_steady_tps	counters
no support model	38.41
--mtp-model <support> --dspark --dspark-confidence 0.45	38.69	accept 51.12%, avg_accept 1.260, no_draft 256, hist 0:279 1:60 2:67 3:50 4:35 5:36

Reference, same machine, 0731 pair (ds4 ab06d19, 2026-08-31, isolated 1147-token structured codegen, same method): opportunistic 52.17 t/s vs strict 43.90 (+18.8%), accept_rate=84.88% avg_accept=2.271 net_saved=+3593ms, accepted_len_hist=0:129,1:22,2:35,3:45,4:30,5:90. A confidence sweep that day put the peak at 0.45 (+10.9% mean over two prompts).

So on comparable code generation the Vision-Exp support model produces about half the accepted tokens per cycle (1.16–1.28 vs 2.27) and roughly twice the zero-accept cycles, which lands at the scheduler's break-even in the CLI path and below it in the server path.

Ruled out

• Confidence threshold: 0.3 / 0.45 / 0.6 all lose (thinking and code).
• Workload: reasoning text vs pure code both lose; pure code is where 0731 wins.
• PR dspark: fix indexer sparse-threshold mismatch + skip batched verify for a bare draft_n==1 accept #915 (indexer sparse-threshold fix for batched verify): merged onto 6289c51 and rebuilt — counters bit-identical (37.68% / 0.394 / 997).
• Support-model structure: header-parsed both files — 81 tensors each, identical type mix (31×Q8_0, 34×F32, 6×IQ2_XXS, 3×Q2_K, 7×F16), same dspark.* params. The Vision-Exp file only adds the variant/revision pin.
• Unit tests: ./ds4_test --dspark-verify-depth and --mtp-verify-depth pass on this pair; the full suite passes apart from local-golden-vectors, which is the 0731 fixture compared against a different checkpoint (top-1 equal, top-20 18/20).
• Reproduced with a local branch carrying an unrelated reasoning-effort patch and on unmodified main: identical counters.

Not testable here

• Target-quant fidelity for Vision-Exp: I have no reference logits for this checkpoint (tests/test-vectors/flash-vision-exp/official/*.json were captured via OpenRouter/Novita and carry 0 logprob steps), so I can't say whether the IQ2_XXS/Q2_K target simply disagrees with the BF16 argmax more often than the 0731 quant does.
• CUDA/ROCm behaviour of the same pair.

Questions

1. Do you have Metal acceptance/net_saved numbers for the Vision-Exp pair from the QA runs? Is ~50% / 1.2 tokens per cycle on code what the released Vision-Exp support model is expected to do, or does the 0731-level 85% / 2.3 apply?
2. If the head is simply weaker, would it make sense for --dspark to log a warning (or for metal: bypass DSpark speculation once a request proves unprofitable #965's bypass) when the lifetime avg_accept sits below the break-even for the backend, so a Vision-Exp user doesn't silently lose 20%?

Side note: greedy output length differed between strict and opportunistic on the pure-code prompt (1642 vs 1601 tokens), the same identity break as #658.

Happy to run anything else on this machine.

[Flor1an-B]

We ran into the same weakness independently a week earlier, from a different angle — sharing in case the cross-reference helps narrow this down.

Related threads: #913 (temp>0 root-cause investigation on the opportunistic accept path) and #695 (original observation by alangiu-gif that a custom higher-precision drafter conversion beats the officially published support GGUF).

Our own greedy (temp=0) numbers for the Vision-Exp support model, M5 Max, main@110afdd at the time (before #915):

content	baseline	official drafter (IQ2_XXS/Q2_K, 5.6 GiB)	our Q4_K rebuild (11.4 GiB)
code (Python if/elif)	43.92 t/s	46.17–46.53 t/s, 65.52% accept (net gain)	49.46–47.68 t/s, 72.22% accept
prose (water cycle)	44.65 t/s	35.38 t/s, 39.64% accept (net loss, −21%)	36.31 t/s, 40.53% accept (still a loss, −19%)

Two things that might be useful against your data:

1. We saw a net gain on code at temp=0 with the stock support model (+5–6%, 65.52% accept), which doesn't match "a loss at every confidence" — our code prompt was a short if/elif chain, yours a longer KV-store-with-tests task. If the weak spot is workload/length-dependent rather than uniformly bad, that's worth separating out; happy to re-run our exact prompt on current main if it helps triangulate.
2. Re-quantizing the drafter's routed experts to Q4_K measurably raises acceptance (65.52%→72.22% on code) but does not flip prose from a loss. We also tried Q8_0 for the experts: zero further gain over Q4_K (byte-identical accept/stats), so the ceiling isn't a quantization artifact past 4-bit — it's the draft head's own predictive ceiling on that content type. That's a partial answer to your Q1: on code the released head is usably weaker than achievable (a better-trained/quantized head measurably helps), but on prose the shortfall looks structural, not a quantization bug.

Side note unrelated to speed, in case it matters for your setup: we also hash-diffed strict-vs-opportunistic output at temp=0 and found the official drafter is not always bit-exact vs a no-draft greedy reference either (1/3 prose prompts diverged for us) — DS4's Metal batched-verify kernel isn't strictly batch-invariant, so a longer accepted run has more chances to land on a near-tied logit position. Not the cause of your throughput numbers, just flagging it's a pre-existing property of the Metal DSpark path, not something specific to Vision-Exp.