Skip to content

feat(turbomind): MTP speculative decoding and SM75 verification attention - #5146

Draft
richardclli wants to merge 6 commits into
InternLM:mainfrom
richardclli:main-mtp-stack
Draft

richardclli wants to merge 6 commits into
InternLM:mainfrom
richardclli:main-mtp-stack

Conversation

@richardclli

Copy link
Copy Markdown

Summary

This consolidates the MTP speculative-decoding work into one merge-ready stack:

The upstream PRs are functional but do not pass the full CI individually. This
branch carries the same work as four main commits plus two CI fixups on current
main, with the original authors preserved, and is intended to pass every check
before merge.

Commits

  1. feat(turbomind): EAGLE3 and Qwen3.5 MTP speculative decoding — Li Zhang
  2. fix(turbomind): address MTP CI failures — bltcn + Richard Li
  3. fix(turbomind): SM75 (pre-SM80) fixes for MTP speculative decoding — bltcn
  4. fix(turbomind): fix MTP lint after the SM75 changes — bltcn + Richard Li
  5. fix(turbomind): harden MTP/speculative serving — Richard Li
  6. feat(turbomind): native SM75 verification attention — Richard Li

Validation

  • SM75 MTP on Qwen3.6-35B-A3B FP8: smoke, 120-request soak, needle 4K/32K/128K.
  • Verification attention: parity fp16 + int4; int4 bench 4K-256K.
  • GitHub CI: <fill from the checks>.

Provenance

The original PRs remain open until this lands; they will be closed as superseded
afterward. NVFP4 (#5142) and the d256 prefill (#5139) are separate and not part of
this stack; the SM75 evidence here uses an FP8 checkpoint.

- Fixed-chain speculator base with EAGLE3 and Qwen3.5 MTP draft models,
  target-native speculation, and method-keyed draft weight registry
- Block speculative verification: paged CuTe verification attention,
  SM90 GDR verify and commit kernels with persistent pipelining
- Unified parameterized batch-op pipeline; executor forward split into
  target pass and speculative round; host batch ops moved engine-side
- Scheduler: unified required admission, simplified rollback, submitted
  rows carry producer-set effects instead of a kind tag
- Op-level pytest suites: verification attention, draft carry,
  speculative sampling, speculative sequence, target hidden projection,
  copy
@richardclli
richardclli marked this pull request as ready for review November 17, 2026 18:05
@richardclli
richardclli marked this pull request as draft November 17, 2026 18:05
@richardclli

Copy link
Copy Markdown
Author

@lzhangzz Can you permit me to run the workflow? It is based on what you have started

bltcn and others added 5 commits November 19, 2026 14:53
CI fixes for the MTP feature commit:

- Use the canonical 'from lmdeploy.turbomind import _tm' import in the
  TurboMind tests and apply the ruff fixes inherited from the feature PR.
- Apply clang-format to the files introduced or changed by the feature.
- Inject the engine symmetric buffer into the pass environment before the
  vision model runs, so VLM TP with cuda-ipc keeps its symmetric buffer.
- Keep the previous whole-archive link order on MSVC, whose linker does not
  support LINK_GROUP:RESCAN.

Co-authored-by: Richard Li <richard.li@ces.hk>
- Normalize qwen3_5_mtp/hy3_mtp/deepseek_mtp to the registered 'mtp' name
  before building the draft model and assigning EngineConfig.spec_method.
- Accept half_t for the GDN recurrent state in the accepted-state commit
  kernel, so speculative rounds run with the float16 dtype on pre-SM80
  GPUs.
Keep the normalize_spec_method docstring within the lint summary width and
apply the lint fixes for the files touched by the SM75 MTP changes.

Co-authored-by: Richard Li <richard.li@ces.hk>
- MoE gate: initialize routing slots to a skip sentinel and bounds-check the
  expert id, so a valid token that selects fewer than top_k experts no longer
  indexes masks[] out of range. Adds a regression test.
- Synchronize the target-pass logits before publishing them; the sampler runs
  on another stream and the first step could hang on stale data.
- Reserve 2*(max_proposals+1) rows below session_len at admission so a
  speculative round cannot write past the allocated KV columns.
Add a native paged verification-attention path for SM75 so MTP no longer
falls back to the prefill path:

- policy_sm75.cuh + kernel_sm75.cu, compiled 75-real only; the Turing MMA
  path reuses the m16n8k8 emulation and keeps k16 tiling.
- int4/uint4 KV reader and dequant (quant_policy 4).
- Per-arch static libraries plus a host dispatch_arch.cc; the SM90 WGMMA
  path is namespaced and behavior-unchanged.
- SM75 split tuning and an int4 verification-attention bench.
@richardclli

Copy link
Copy Markdown
Author

@lzhangzz I have tried to fix the build, need approval again

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants