Skip to content

[Cherry-Pick][Optimization] Use triton qk_norm both in Prefill and Decode.(#7213) #7305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
EmmonsCurse wants to merge 1 commit into
base: release/2.5
Choose a base branch
from
Open

[Cherry-Pick][Optimization] Use triton qk_norm both in Prefill and Decode.(#7213) #7305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion fastdeploy/model_executor/layers/normalization.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ def forward(
forward_meta,
proxy_rmsnorm=None,
) -> paddle.Tensor:
if proxy_rmsnorm is None and self.qk_norm_fused and forward_meta.step_use_cudagraph:
if proxy_rmsnorm is None and self.qk_norm_fused:
qkv_out = qk_rmsnorm_fused(
qkv_out,
self.q_norm.weight,
Expand Down
2 changes: 1 addition & 1 deletion tests/e2e/test_Qwen3VL_serving.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ def test_consistency_between_runs(api_url, headers, consistent_payload):
content1 = result1["choices"][0]["message"]["content"]

# base result
content2 = "视频中手机支架的颜色是黑色的。"
content2 = "视频中手机支架的颜色是黑色。"
Copy link
Copy Markdown

@PaddlePaddle-bot PaddlePaddle-bot Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 建议 测试预期输出从 "视频中手机支架的颜色是黑色的。" 修改为 "视频中手机支架的颜色是黑色。"，这个变更应该在 PR 描述中说明原因。

建议在 PR 描述中补充说明：由于改用 triton qk_rmsnorm_fused kernel 替代 paddle 算子，导致模型输出在端到端测试中存在微小差异（属于正常范围内的浮点计算差异）。


# Verify that result is same as the base result
assert content1.startswith(content2), content1
Expand Down
Loading