[None][chore] Add failed cases into waives.txt #10302

xinhe-nv · 2025-12-25T18:06:50Z

waive failed cases.

Summary by CodeRabbit

Tests
- Updated test tracking for known issues with specific model serving configurations.

Note: These changes are internal quality assurance updates with no direct impact on end-user features or functionality.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Signed-off-by: xinhe-nv <[email protected]>

…ION_TEST_1788 Signed-off-by: xinhe-nv <[email protected]>

xinhe-nv · 2025-12-30T02:44:25Z

/bot run --skip-test

coderabbitai · 2025-12-30T02:46:22Z

📝 Walkthrough

Walkthrough

Added test-skip entries to the integration test waives configuration file for specific test parameterizations in the Llama3.1 8B disaggregated serving accuracy test, referencing associated bug tracker issues.

Changes

Cohort / File(s)	Summary
Test skip configuration `tests/integration/test_lists/waives.txt`	Added SKIP annotations for `TestLlama3_1_8BInstruct::test_tp_pp_symmetric` with GSM8K and MMLU parameterizations, linked to bug references (https://nvbugs/5773047, https://nvbugs/5596337)

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

[TRTLLM-8638][fix] Add failed cases into waives.txt #10129 — Modifies the same test waives file to skip test parameterizations for TestLlama3_1_8BInstruct::test_tp_pp_symmetric with different dataset configurations
[None][chore] Add failed cases into waives.txt #10025 — Adds SKIP entries to the same waives.txt file for disaggregated serving accuracy tests
[None][chore] Add failed cases into waives.txt #10240 — Updates the same file to skip test variants of the TestLlama3_1_8BInstruct class with different parameter configurations

Suggested reviewers

crazydemo
LarryXFly
jieli-matrix
StanleySun639

Pre-merge checks

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is extremely minimal and lacks required sections from the template (Description, Test Coverage, and PR Checklist).	Expand the description to include: detailed explanation of which test cases are being waived and why, relevant test coverage information, and completion of the PR Checklist items.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding failed test cases to the waives.txt file.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4944192 and 5e6df82.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

🧰 Additional context used

🧠 Learnings (7)

📓 Common learnings

Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 7781
File: tests/integration/test_lists/waives.txt:313-313
Timestamp: 2025-09-17T02:48:52.732Z
Learning: In TensorRT-LLM, `tests/integration/test_lists/waives.txt` is specifically for waiving/skipping tests, while other test list files like those in `test-db/` and `qa/` directories are for different test execution contexts (pre-merge, post-merge, QA tests). The same test appearing in both waives.txt and execution list files is intentional - the test is part of test suites but will be skipped due to the waiver.

Learnt from: pengbowang-nv
Repo: NVIDIA/TensorRT-LLM PR: 7192
File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72
Timestamp: 2025-08-26T09:49:04.956Z
Learning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

Learnt from: fredricz-20070104
Repo: NVIDIA/TensorRT-LLM PR: 7645
File: tests/integration/test_lists/qa/llm_function_core.txt:648-648
Timestamp: 2025-09-09T09:40:45.658Z
Learning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Learnt from: EmmaQiaoCh
Repo: NVIDIA/TensorRT-LLM PR: 7370
File: tests/unittest/trt/model_api/test_model_quantization.py:24-27
Timestamp: 2025-08-29T14:07:45.863Z
Learning: In TensorRT-LLM's CI infrastructure, pytest skip markers (pytest.mark.skip) are properly honored even when test files have __main__ blocks that call test functions directly. The testing system correctly skips tests without requiring modifications to the __main__ block execution pattern.

Learnt from: fredricz-20070104
Repo: NVIDIA/TensorRT-LLM PR: 9511
File: tests/integration/defs/examples/serve/test_serve.py:136-186
Timestamp: 2025-11-27T09:23:18.742Z
Learning: In TensorRT-LLM testing, when adding test cases based on RCCA commands, the command format should be copied exactly as it appears in the RCCA case, even if it differs from existing tests. For example, some RCCA commands for trtllm-serve may omit the "serve" subcommand while others include it.

Learnt from: nvpohanh
Repo: NVIDIA/TensorRT-LLM PR: 7478
File: tests/unittest/_torch/modeling/test_modeling_llama_min_latency.py:286-308
Timestamp: 2025-09-03T13:16:38.028Z
Learning: In test files, temporary monkey-patches for upstream bugs can be kept simple when they are explicitly intended to be removed soon, rather than investing effort in making them more robust.

Learnt from: galagam
Repo: NVIDIA/TensorRT-LLM PR: 6487
File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12
Timestamp: 2025-08-06T13:58:07.506Z
Learning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

📚 Learning: 2025-09-17T02:48:52.732Z

Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 7781
File: tests/integration/test_lists/waives.txt:313-313
Timestamp: 2025-09-17T02:48:52.732Z
Learning: In TensorRT-LLM, `tests/integration/test_lists/waives.txt` is specifically for waiving/skipping tests, while other test list files like those in `test-db/` and `qa/` directories are for different test execution contexts (pre-merge, post-merge, QA tests). The same test appearing in both waives.txt and execution list files is intentional - the test is part of test suites but will be skipped due to the waiver.

Applied to files:

tests/integration/test_lists/waives.txt

📚 Learning: 2025-09-09T09:40:45.658Z

Learnt from: fredricz-20070104
Repo: NVIDIA/TensorRT-LLM PR: 7645
File: tests/integration/test_lists/qa/llm_function_core.txt:648-648
Timestamp: 2025-09-09T09:40:45.658Z
Learning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Applied to files:

tests/integration/test_lists/waives.txt

📚 Learning: 2025-08-29T14:07:45.863Z

Learnt from: EmmaQiaoCh
Repo: NVIDIA/TensorRT-LLM PR: 7370
File: tests/unittest/trt/model_api/test_model_quantization.py:24-27
Timestamp: 2025-08-29T14:07:45.863Z
Learning: In TensorRT-LLM's CI infrastructure, pytest skip markers (pytest.mark.skip) are properly honored even when test files have __main__ blocks that call test functions directly. The testing system correctly skips tests without requiring modifications to the __main__ block execution pattern.

Applied to files:

tests/integration/test_lists/waives.txt

📚 Learning: 2025-08-26T09:49:04.956Z

Learnt from: pengbowang-nv
Repo: NVIDIA/TensorRT-LLM PR: 7192
File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72
Timestamp: 2025-08-26T09:49:04.956Z
Learning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

Applied to files:

tests/integration/test_lists/waives.txt

📚 Learning: 2025-07-28T17:06:08.621Z

Learnt from: moraxu
Repo: NVIDIA/TensorRT-LLM PR: 6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Applied to files:

tests/integration/test_lists/waives.txt

📚 Learning: 2025-08-18T08:42:02.640Z

Learnt from: samuellees
Repo: NVIDIA/TensorRT-LLM PR: 6974
File: tensorrt_llm/serve/scripts/benchmark_dataset.py:558-566
Timestamp: 2025-08-18T08:42:02.640Z
Learning: In TensorRT-LLM's RandomDataset (tensorrt_llm/serve/scripts/benchmark_dataset.py), when using --random-token-ids option, sequence length accuracy is prioritized over semantic correctness for benchmarking purposes. The encode/decode operations should use skip_special_tokens=True and add_special_tokens=False to ensure exact target token lengths.

Applied to files:

tests/integration/test_lists/waives.txt

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

GitHub Check: Pre-commit Check

🔇 Additional comments (1)

tests/integration/test_lists/waives.txt (1)

523-523: Entry format is consistent and properly structured.

The new waive entry follows the established pattern in the file, includes a valid bug tracker reference, and correctly extends the skip list for the Llama3.1 8B disaggregated serving test with the GSM8K dataset variant on sm89 hardware. The entry mirrors the structure of related test parameterizations already in the file (MMLU variants at lines 299 and 417) and integrates cleanly with the existing waive configuration.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2025-12-30T02:50:29Z

PR_Github #30113 [ run ] triggered by Bot. Commit: 5e6df82

tensorrt-cicd · 2025-12-30T03:34:40Z

PR_Github #30113 [ run ] completed with state FAILURE. Commit: 5e6df82
/LLM/main/L0_MergeRequest_PR pipeline #23174 (Partly Tested) completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

…ION_TEST_1788

…ION_TEST_1788 Signed-off-by: xinhe-nv <[email protected]>

xinhe-nv · 2025-12-30T05:28:14Z

/bot run --skip-test

tensorrt-cicd · 2025-12-30T05:34:03Z

PR_Github #30134 [ run ] triggered by Bot. Commit: faae14a

…ION_TEST_1788

tensorrt-cicd · 2025-12-30T07:18:39Z

PR_Github #30134 [ run ] completed with state SUCCESS. Commit: faae14a
/LLM/main/L0_MergeRequest_PR pipeline #23188 (Partly Tested) completed with status: 'SUCCESS'

xinhe-nv · 2025-12-30T07:24:56Z

/bot reuse-pipeline

tensorrt-cicd · 2025-12-30T07:30:29Z

PR_Github #30145 [ reuse-pipeline ] triggered by Bot. Commit: c282862

…ION_TEST_1788

xinhe-nv · 2025-12-30T07:41:17Z

/bot reuse-pipeline

tensorrt-cicd · 2025-12-30T07:47:23Z

PR_Github #30147 [ reuse-pipeline ] triggered by Bot. Commit: 3e4db0f

tensorrt-cicd · 2025-12-30T07:47:27Z

PR_Github #30145 [ reuse-pipeline ] completed with state ABORTED. Commit: c282862
Can't reuse PR_Github #30134 (Partly Tested) with status: SUCCESS

tensorrt-cicd · 2025-12-30T08:11:50Z

PR_Github #30147 [ reuse-pipeline ] completed with state SUCCESS. Commit: 3e4db0f
Reusing PR_Github #30134 (Partly Tested) for commit 3e4db0f

update waive list

8b5aad5

Signed-off-by: xinhe-nv <[email protected]>

xinhe-nv requested review from LarryXFly and crazydemo December 25, 2025 18:06

xinhe-nv added 3 commits December 30, 2025 10:42

Update waives.txt

684901a

Signed-off-by: xinhe-nv <[email protected]>

Update waives.txt

decc1e5

Signed-off-by: xinhe-nv <[email protected]>

Merge branch 'main' into user/qa/post_update_waive_20251226_LLM_FUNCT…

5e6df82

…ION_TEST_1788 Signed-off-by: xinhe-nv <[email protected]>

xinhe-nv marked this pull request as ready for review December 30, 2025 02:44

xinhe-nv enabled auto-merge (squash) December 30, 2025 02:44

xinhe-nv added 2 commits December 30, 2025 12:24

Merge branch 'main' into user/qa/post_update_waive_20251226_LLM_FUNCT…

27ed6d2

…ION_TEST_1788

Merge branch 'main' into user/qa/post_update_waive_20251226_LLM_FUNCT…

faae14a

…ION_TEST_1788 Signed-off-by: xinhe-nv <[email protected]>

xinhe-nv added 2 commits December 30, 2025 14:05

Merge branch 'main' into user/qa/post_update_waive_20251226_LLM_FUNCT…

38cee25

…ION_TEST_1788

Merge branch 'main' into user/qa/post_update_waive_20251226_LLM_FUNCT…

c282862

…ION_TEST_1788

LarryXFly approved these changes Dec 30, 2025

View reviewed changes

Merge branch 'main' into user/qa/post_update_waive_20251226_LLM_FUNCT…

3e4db0f

…ION_TEST_1788

xinhe-nv merged commit 6accdbc into NVIDIA:main Dec 30, 2025
5 checks passed

xinhe-nv deleted the user/qa/post_update_waive_20251226_LLM_FUNCTION_TEST_1788 branch December 30, 2025 08:12

coderabbitai bot mentioned this pull request Jan 5, 2026

[TRTLLM-8638][fix] Add failed cases into waives.txt #10384

Merged

[None][chore] Add failed cases into waives.txt #10302

[None][chore] Add failed cases into waives.txt #10302

Uh oh!

Conversation

xinhe-nv commented Dec 25, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

xinhe-nv commented Dec 30, 2025

Uh oh!

coderabbitai bot commented Dec 30, 2025

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Pre-merge checks

Uh oh!

tensorrt-cicd commented Dec 30, 2025

Uh oh!

tensorrt-cicd commented Dec 30, 2025

Uh oh!

xinhe-nv commented Dec 30, 2025

Uh oh!

tensorrt-cicd commented Dec 30, 2025

Uh oh!

tensorrt-cicd commented Dec 30, 2025

Uh oh!

xinhe-nv commented Dec 30, 2025

Uh oh!

tensorrt-cicd commented Dec 30, 2025

Uh oh!

xinhe-nv commented Dec 30, 2025

Uh oh!

tensorrt-cicd commented Dec 30, 2025

Uh oh!

tensorrt-cicd commented Dec 30, 2025

Uh oh!

tensorrt-cicd commented Dec 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xinhe-nv commented Dec 25, 2025 •

edited by coderabbitai bot

Loading