Server: multimodal sessions lose all KV reuse on a tail token mismatch — no common-prefix rewind, no disk-KV fallback, no continued disk stores

[graymount]

Title: Server: multimodal sessions lose all KV reuse on a tail token mismatch — no common-prefix rewind, no disk-KV fallback, no continued disk stores

Observed on: 6289c51 (current main), DeepSeek V4 Flash Vision-Exp MXFP4, --vision server mode, --batched-session 2, disk KV enabled (--kv-disk-dir, 1 TB budget).

Summary

For multimodal (vision) chat requests, KV reuse works perfectly while the
conversation round-trips exactly (multimodal live kv hit identity=fingerprint-match,
sub-second turns on a ~335k-token prompt). But as soon as one request diverges
from the live session by even a handful of tokens at the tail, the server falls
back to a full cold prefill from token 0 — three reuse mechanisms that exist for
text sessions are all unavailable or inert for multimodal ones:

1. No common-prefix rewind on live token-mismatch. The live-prefix check
   computes the common prefix length but only accepts it on an exact full match:

   ds4-server: live kv cache miss live=340263 prompt=340551 common=340144 vision=match reason=token-mismatch
   ds4-server: chat ctx=0..340551:340551 TOOLS prompt start
   

   99.96% of the prompt (340,144 / 340,551 tokens) was identical to the live
   session, yet the server restarted prefill from 0. The GLM path right above
   it does exactly this kind of rewind (rewound GLM live prefix from %d to %d,
   memory-rewind cache source). The DS4 path is all-or-nothing:

   cached = common == old_pos && j->req.prompt.len >= old_pos ? common : 0;

2. Disk KV load is gated off for multimodal requests (ds4_server.c, request
   path around the live-miss fallback):

   if (!multimodal && s->kv.enabled && cached == 0 && old_pos >= s->kv.opt.min_tokens) {
       kv_cache_store_current(s, slot, "evict");
   }
   if (!multimodal && cached == 0) {
       disk_cached = kv_cache_try_load(...);
   }

   So a vision session that misses the live cache never consults the disk store
   at all, even when a matching checkpoint exists.

3. No continued disk checkpoints are written during multimodal cold
   prefills. On c0a6119 the same workload logged
   kv cache stored tokens=... reason=continued every ~30k tokens during a long
   prefill. On 6289c51, three full ~330k-token prefills (18 minutes each)
   wrote zero checkpoints — so even if (2) were reachable, there would be
   nothing fresh to load. (The text-prefix memory path did resume one later
   turn at 100,352 of 346,533 tokens, so partial reuse is not entirely dead,
   but the deep disk checkpoints are never written.)

Trigger

A coding-agent client (OpenAI chat-completions, tools, one screenshot per turn)
occasionally re-serializes the previous assistant reply slightly differently
than the server's generated token stream (thinking/DSML round-trip). The
divergence lands inside the previous reply's tail (e.g. 119 tokens before the
end of a 340,263-token history). With text sessions this costs nothing; with a
vision session every such turn is a full ~330k prefill ≈ 18 minutes on an
M3 Ultra.

Suggested fixes

• On live token-mismatch with vision=match, rewind the DS4 session to
  common (same approach as the existing GLM memory-rewind path) instead of
  discarding the whole session.
• Allow disk-KV load/store for multimodal sessions (store text-prefix KV even
  if image embeddings can't be serialized; the embedding cache already covers
  re-encoding cheaply).

Sanitized log excerpt

21:47:24 multimodal live kv hit images=1 cached=334535 prompt=334603 identity=fingerprint-match
21:47:25 chat ctx=334535..334603:68 TOOLS prompt done 1.041s        # ~1 s turns while round-trip is exact
...
21:55:10 live kv cache miss live=340263 prompt=340551 common=340144 vision=match reason=token-mismatch
21:55:10 chat ctx=0..340551:340551 TOOLS prompt start               # full 18-min prefill, zero continued stores

(Happy to provide longer excerpts privately if useful.)