DAOS-19248 test: Fix no tag match for Functional VM stages by phender · Pull Request #18597 · daos-stack/daos

phender · 2026-07-01T18:05:54Z

Define the stages in the Test stage as scripted stages to enable running the
Functional Test stages run on VM with the getFunctionalTestStage() method.
This will enable using testsInStage() to verify the test tags will result in at
least one test to be run by this stage - fixing a current issue where VM
Functional test stages can fail with no tests matching the tags. To run the
Test RPM and Fault injection testing stages as scripted stages two new
pipeline-lib methods are being utilized: scriptedTestRpmStage() and
scriptedUnitTestStage().

Additional fixes/changes:

fix running the Test RPMs on EL 9.6 stage on EL 9.6 instead of 9.7
rename Test RPMs on EL 9.6 stage to Test RPMs on EL 9
rename Test RPMs on Leap 15.5 to Test RPMs on Leap 15
add collecting a log for the Test RPMs stages and log the distro
groovyLint cleanup in the Jenkinsfile

Test-tag: test_always_passes_hw
Skip-functional-on-el-8: false
Skip-functional-on-el-9: false
Skip-functional-on-leap-15: false
Skip-functional-on-sles-15: false

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Ensure Functional VM stages are skipped if no test tags match. Test-tag: test_always_passes_hw Skip-functional-on-el-8: false Skip-functional-on-el-9: false Skip-functional-on-leap-15: false Skip-functional-on-sles-15: false Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

github-actions · 2026-07-01T18:09:29Z

Ticket title is 'Update testing branches'
Status is 'Resolved'
https://daosio.atlassian.net/browse/DAOS-19248

Skip-test-hardware: true Test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

skip-unit-tests: true skip-test-hardware: true test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-07-02T03:37:45Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/5/display/redirect

daosbuild3 · 2026-07-02T04:22:43Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/7/display/redirect

skip-unit-tests: true skip-test-hardware: true test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-07-02T20:01:41Z

Test stage Functional on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/13/execution/node/1065/log

Skip-cancel-previous-builds: true Skip-functional-test: true SKip-fault-injection-testing: true Skip-unit-tests: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-07-02T23:03:57Z

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/15/execution/node/831/log

daosbuild3 · 2026-07-02T23:41:23Z

Test stage Functional on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/15/execution/node/944/log

phender · 2026-07-03T02:38:57Z

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/15/execution/node/831/log

In Build 15 the Test RPMs on Leap 15.5 was corrected to install and run on 15.5 - previously (in build 13) it was installing and running on 15.6. Now the stage fails with:

[2026-07-02T23:03:56.240Z] Error: 
[2026-07-02T23:03:56.240Z]  Problem: package daos-server-tests-2.9.100-4.50.g33213bba.suse.lp156.x86_64 requires daos-server = 2.9.100-4.50.g33213bba.suse.lp156, but none of the providers can be installed
[2026-07-02T23:03:56.240Z]   - package daos-server-2.9.100-4.50.g33213bba.suse.lp156.x86_64 requires libpmemobj1 = 2.1.3-2.suse.lp156, but none of the providers can be installed
[2026-07-02T23:03:56.240Z]   - conflicting requests
[2026-07-02T23:03:56.240Z]   - nothing provides libc.so.6(GLIBC_2.38)(64bit) needed by libpmemobj1-2.1.3-2.suse.lp156.x86_64

According to https://daosio.atlassian.net/browse/DAOS-18925, we need pciutils-3.13.0-150300.13.12.1.x86_64 but that's only available in 15.6.

~> sudo dnf provides pciutils
Last metadata expiration check: 0:01:33 ago on Fri Jul  3 03:30:09 2026.
allow_vendor_change is disabled. This option is currently not supported for downgrade and distro-sync commands
pciutils-3.5.6-150300.13.6.1.x86_64 : PCI utilities for the Linux Kernel
Repo        : opensuse-oss-proxy-artifactory
Matched from:
Provide    : pciutils = 3.5.6-150300.13.6.1

pciutils-3.13.0-150300.13.12.1.x86_64 : PCI utilities for the Linux Kernel
Repo        : @System
Matched from:
Provide    : pciutils = 3.13.0-150300.13.12.1

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

sjip-cancel-previous-builds: true skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-9: true skip-fault-injection-testing: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-07-03T03:35:35Z

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/17/display/redirect

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-9: true skip-fault-injection-testing: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-07-03T04:20:05Z

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/18/display/redirect

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-07-03T05:26:03Z

Test stage Functional on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/19/execution/node/948/log

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-07-06T18:12:58Z

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/20/execution/node/973/log

daosbuild3 · 2026-07-06T18:32:13Z

Test stage Functional on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/20/execution/node/1087/log

phender · 2026-07-07T02:44:41Z

Test stage Functional on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/20/execution/node/1087/log

This failure is expected until https://daosio.atlassian.net/browse/DAOS-15233 has been resolved.

phender · 2026-07-07T02:51:04Z

Note the improved artifacts for the RPM Test RPM stages:

daosbuild3 · 2026-07-16T08:15:38Z

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/45/execution/node/1511/log

daosbuild3 · 2026-07-16T09:17:31Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/45/execution/node/1608/log

daosbuild3 · 2026-07-16T11:21:12Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/45/execution/node/1743/log

daosbuild3 · 2026-07-16T11:41:44Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/45/execution/node/1792/log

daosbuild3 · 2026-07-16T15:01:23Z

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/46/execution/node/728/log

daosbuild3 · 2026-07-16T16:00:48Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/46/execution/node/910/log

daosbuild3 · 2026-07-16T16:03:53Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/46/execution/node/920/log

daosbuild3 · 2026-07-16T16:09:37Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/46/execution/node/1007/log

phender · 2026-07-20T13:49:10Z

Testing passed in https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/50/stages where:

all the Functional stages ran successfully (excluding Ubuntu) using getFunctionalTestStage()
the Fault Injection testing stage ran successfully using scriptedUnitTestStage()
all the Test RPM stages ran successfully using scriptedTestRpmStage()

daltonbohning

Is this working? The latest CI seems to still be failing due to "no tests in stage"

daltonbohning · 2026-07-20T14:59:38Z

-                            unstashTests: false,
-                            imageVersion: 'el9.7',
-                            provEnvVars: 'VM_CPUS=14',
+                            unitTestArgs: [


Seeing it this way, it seems odd that a function called scriptedUnitTestStage has an argument unitTestArgs because I would assume all args to scriptedUnitTestStage are "unit test" arguments :)

I can see where you're coming from, but the Unit Test Stage encompasses more than just the unitTest function; it calls:

the unitTest function

the unitTestPost function

the archiveArtifacts function

the jobStatusUpdate function

so, its arguments are not just arguments exclusive to the unitTest function.

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender · 2026-07-20T21:37:50Z

Is this working? The latest CI seems to still be failing due to "no tests in stage"

If we're talking about https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/50/stages, then yes. The build was run with:

skip-unit-tests: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

The test_load_mpi tag matches one test that uses :avocado: tags=vm, so it should only be run on the Functional VM stages. The four Functional VM stages passed and ran the harness/basic.py:HarnessBasicTest.test_load_mpi test:

Functional on EL 8
Functional on EL 9
Functional on Leap 15
Functional on SLES 15

None of the Functional HW stages ran - as expected: 4 stages due to no tags match; 5 stages due to default run parameters (e.g. the runStage argument):

[Functional Hardware Medium] Stage skipped by runStage=false
[Functional Hardware Medium MD on SSD] Skipping the stage due to detecting no tests matching the 'test_load_mpi,hw,medium,-provider' tags
[Functional Hardware Medium VMD] Stage skipped by runStage=false
[Functional Hardware Medium Verbs Provider] Stage skipped by runStage=false
[Functional Hardware Medium Verbs Provider MD on SSD] Skipping the stage due to detecting no tests matching the 'test_load_mpi,hw,medium,provider' tags
[Functional Hardware Medium UCX Provider] Stage skipped by runStage=false
[Functional Hardware Large] Stage skipped by runStage=false
[Functional Hardware Large MD on SSD] Skipping the stage due to detecting no tests matching the 'test_load_mpi,hw,large' tags
[Functional Cluster Box Medium MD on SSD] Skipping the stage due to detecting no tests matching the 'test_load_mpi,cb,medium' tags

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2026-07-23T21:55:11Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/54/execution/node/1702/log

daosbuild3 · 2026-07-23T21:55:19Z

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/54/execution/node/1518/log

…18715) Define the stages in the Test stage as scripted stages to enable running the Functional Test stages run on VM with the getFunctionalTestStage() method. This will enable using testsInStage() to verify the test tags will result in at least one test to be run by this stage - fixing a current issue where VM Functional test stages can fail with no tests matching the tags. To run the Test RPM and Fault injection testing stages as scripted stages two new pipeline-lib methods are being utilized: scriptedTestRpmStage() and scriptedUnitTestStage(). Additional fixes/changes: - fix running the Test RPMs on EL 9.6 stage on EL 9.6 instead of 9.7 - rename Test RPMs on EL 9.6 stage to Test RPMs on EL 9 - rename Test RPMs on Leap 15.5 to Test RPMs on Leap 15 - add collecting a log for the Test RPMs stages and log the distro - groovyLint cleanup in the Jenkinsfile Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender requested a review from daltonbohning July 1, 2026 18:06

daltonbohning reviewed Jul 1, 2026

View reviewed changes

Comment thread Jenkinsfile Outdated

Use scripted pipeline methods for the Test stages.

ab22926

Skip-test-hardware: true Test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender mentioned this pull request Jul 1, 2026

DAOS-19247 test: Support scripted Test stages daos-stack/pipeline-lib#526

Merged

Updates.

e2c7eda

skip-unit-tests: true skip-test-hardware: true test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender added 2 commits July 2, 2026 01:24

Updates.

231152d

skip-unit-tests: true skip-test-hardware: true test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Updates

9658ef1

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender added 2 commits July 2, 2026 17:56

Test artifact dir name.

d8d25ab

Skip-cancel-previous-builds: true Skip-functional-test: true SKip-fault-injection-testing: true Skip-unit-tests: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Test RPM updates

33213bb

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender added 2 commits July 2, 2026 22:48

Hack to install pciutils-3.13

e57a1e0

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Collect log for test_daos_node.sh execution

7b33944

sjip-cancel-previous-builds: true skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-9: true skip-fault-injection-testing: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Log collection hardening

a643198

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-9: true skip-fault-injection-testing: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Remove hack

19629f9

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Retain test_daos_node.sh exit status

dcfa210

skip-unit-tests: true skip-test-hardware: true skip-functional-on-el-8: false skip-functional-on-leap-15: false skip-functional-on-sles-15: false test-tag: test_load_mpi test_setup_vm Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Merge branch 'master' into hendersp/DAOS-19247-fix

076812e

phender requested review from daltonbohning and grom72 July 16, 2026 04:56

daltonbohning reviewed Jul 20, 2026

View reviewed changes

phender added 2 commits July 20, 2026 13:42

Merge branch 'master' into hendersp/DAOS-19247-fix

b5b3ef6

Removing pipeline-lib PR reference

fe2999a

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daltonbohning previously approved these changes Jul 21, 2026

View reviewed changes

ryon-jensen previously approved these changes Jul 22, 2026

View reviewed changes

Merge branch 'master' into hendersp/DAOS-19247-fix

0c55229

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender dismissed stale reviews from ryon-jensen and daltonbohning via 0c55229 July 23, 2026 00:42

phender requested review from daltonbohning and ryon-jensen July 23, 2026 00:43

ryon-jensen approved these changes Jul 23, 2026

View reviewed changes

daltonbohning approved these changes Jul 23, 2026

View reviewed changes

phender requested a review from a team July 23, 2026 21:49

phender merged commit 9cef688 into master Jul 24, 2026
48 of 52 checks passed

phender deleted the hendersp/DAOS-19247-fix branch July 24, 2026 13:21

Uh oh!

Conversation

phender commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

daosbuild3 commented Jul 2, 2026

Uh oh!

daosbuild3 commented Jul 2, 2026

Uh oh!

daosbuild3 commented Jul 2, 2026

Uh oh!

daosbuild3 commented Jul 2, 2026

Uh oh!

daosbuild3 commented Jul 2, 2026

Uh oh!

phender commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jul 3, 2026

Uh oh!

daosbuild3 commented Jul 3, 2026

Uh oh!

daosbuild3 commented Jul 3, 2026

Uh oh!

daosbuild3 commented Jul 6, 2026

Uh oh!

daosbuild3 commented Jul 6, 2026

Uh oh!

phender commented Jul 7, 2026

Uh oh!

phender commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

phender commented Jul 20, 2026

Uh oh!

daltonbohning left a comment

Choose a reason for hiding this comment

Uh oh!

daltonbohning Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

phender Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phender commented Jul 20, 2026

Uh oh!

daosbuild3 commented Jul 23, 2026

Uh oh!

daosbuild3 commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

phender commented Jul 1, 2026 •

edited

Loading

github-actions Bot commented Jul 1, 2026 •

edited

Loading

phender commented Jul 3, 2026 •

edited

Loading

phender commented Jul 7, 2026 •

edited

Loading

phender Jul 20, 2026 •

edited

Loading