Repository navigation
feat(turbomind): MTP speculative decoding and SM75 verification attention - #5146
Draft
richardclli wants to merge 6 commits into
Draft
richardclli wants to merge 6 commits into
richardclli wants to merge 6 commits into
Conversation
- Fixed-chain speculator base with EAGLE3 and Qwen3.5 MTP draft models, target-native speculation, and method-keyed draft weight registry - Block speculative verification: paged CuTe verification attention, SM90 GDR verify and commit kernels with persistent pipelining - Unified parameterized batch-op pipeline; executor forward split into target pass and speculative round; host batch ops moved engine-side - Scheduler: unified required admission, simplified rollback, submitted rows carry producer-set effects instead of a kind tag - Op-level pytest suites: verification attention, draft carry, speculative sampling, speculative sequence, target hidden projection, copy
richardclli
marked this pull request as ready for review
November 17, 2026 18:05
richardclli
marked this pull request as draft
November 17, 2026 18:05
Author
|
@lzhangzz Can you permit me to run the workflow? It is based on what you have started |
CI fixes for the MTP feature commit: - Use the canonical 'from lmdeploy.turbomind import _tm' import in the TurboMind tests and apply the ruff fixes inherited from the feature PR. - Apply clang-format to the files introduced or changed by the feature. - Inject the engine symmetric buffer into the pass environment before the vision model runs, so VLM TP with cuda-ipc keeps its symmetric buffer. - Keep the previous whole-archive link order on MSVC, whose linker does not support LINK_GROUP:RESCAN. Co-authored-by: Richard Li <richard.li@ces.hk>
- Normalize qwen3_5_mtp/hy3_mtp/deepseek_mtp to the registered 'mtp' name before building the draft model and assigning EngineConfig.spec_method. - Accept half_t for the GDN recurrent state in the accepted-state commit kernel, so speculative rounds run with the float16 dtype on pre-SM80 GPUs.
Keep the normalize_spec_method docstring within the lint summary width and apply the lint fixes for the files touched by the SM75 MTP changes. Co-authored-by: Richard Li <richard.li@ces.hk>
- MoE gate: initialize routing slots to a skip sentinel and bounds-check the expert id, so a valid token that selects fewer than top_k experts no longer indexes masks[] out of range. Adds a regression test. - Synchronize the target-pass logits before publishing them; the sampler runs on another stream and the first step could hang on stale data. - Reserve 2*(max_proposals+1) rows below session_len at admission so a speculative round cannot write past the allocated KV columns.
Add a native paged verification-attention path for SM75 so MTP no longer falls back to the prefill path: - policy_sm75.cuh + kernel_sm75.cu, compiled 75-real only; the Turing MMA path reuses the m16n8k8 emulation and keeps k16 tiling. - int4/uint4 KV reader and dequant (quant_policy 4). - Per-arch static libraries plus a host dispatch_arch.cc; the SM90 WGMMA path is namespaced and behavior-unchanged. - SM75 split tuning and an int4 verification-attention bench.
richardclli
force-pushed
the
main-mtp-stack
branch
from
November 19, 2026 07:13
658afbe to
b569f57
Compare
Author
|
@lzhangzz I have tried to fix the build, need approval again |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This consolidates the MTP speculative-decoding work into one merge-ready stack:
The upstream PRs are functional but do not pass the full CI individually. This
branch carries the same work as four main commits plus two CI fixups on current
main, with the original authors preserved, and is intended to pass every check
before merge.
Commits
Validation
<fill from the checks>.Provenance
The original PRs remain open until this lands; they will be closed as superseded
afterward. NVFP4 (#5142) and the d256 prefill (#5139) are separate and not part of
this stack; the SM75 evidence here uses an FP8 checkpoint.