Skip to content

Fix GLM 5.2 mxfp4 MTP loading issue#1022

Open
amd-xiaoyu12 wants to merge 1 commit into
ROCm:mainfrom
amd-xiaoyu12:xiaoyu/glm-5.2-mtp
Open

Fix GLM 5.2 mxfp4 MTP loading issue#1022
amd-xiaoyu12 wants to merge 1 commit into
ROCm:mainfrom
amd-xiaoyu12:xiaoyu/glm-5.2-mtp

Conversation

@amd-xiaoyu12

@amd-xiaoyu12 amd-xiaoyu12 commented Jun 25, 2026

Copy link
Copy Markdown

Summary

  • Fix Quark quantization exclusion handling for GLM-5.2 MTP layers.
  • Treat excluded aggregate MoE modules as unquantized when their child expert projections are listed in quantization_config.exclude.
  • This prevents the GLM-5.2 MTP layer from being allocated with MXFP4-packed expert tensors when the checkpoint stores that layer in bf16.

Motivation

GLM-5.2 MXFP4 checkpoints can keep the MTP layer unquantized while quantizing the main MoE layers. The Quark config currently honors exclude for linear layers, but the MTP draft layer is constructed as a FusedMoE module. As a result, vLLM allocates MXFP4-packed expert weights for an unquantized bf16 MTP layer and fails during weight loading with:

RuntimeError: The size of tensor a (256) must match the size of tensor b (512) at non-singleton dimension 1

Test Plan

Verified GLM-5.2 MXFP4 MTP serving on MI355X, TP=4:

vllm serve /data/colizeng/GLM-5.2-mxfp4-ptpcfp8-attn/
--quantization quark
--trust-remote-code
--tensor-parallel-size 4
--speculative-config '{"method":"mtp","num_speculative_tokens":5}'
--gpu-memory-utilization 0.55
--max-model-len 32768
--port 8011
--no-enable-prefix-caching
Confirmed server reached /health.

Confirmed old MoE load failure no longer occurs:

RuntimeError: The size of tensor a (256) must match the size of tensor b (512)
Ran completion smoke test and confirmed spec decode metrics increased.

Ran TP=4 MTP throughput smoke benchmark:

vllm bench serve
--model /data/colizeng/GLM-5.2-mxfp4-ptpcfp8-attn/
--base-url http://localhost:8011
--dataset-name random
--random-input 8000
--random-output 1024
--request-rate 10
--num-prompts 32
--ignore-eos
Benchmark completed with 32/32 successful requests and emitted MTP counters:

Test Result

Draft tokens: 119285
Accepted tokens: 8899
Acceptance rate: 7.46%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant