DAOS-623 test: instrument NLT fault-injection sweep hang - #18756
Conversation
|
Ticket title is 'Generic ticket for minor code cleanup and improvement' |
github-actions
Bot
added
the
priority
|
Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18756/1/execution/node/868/log |
mjmac
changed the title
b306f51 to
1e4d15e
Compare
|
Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18756/8/execution/node/535/log |
|
Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18756/11/execution/node/534/log |
A hung child stalls the sweep until the CI stage is killed, which discards the artifacts and leaves nothing to diagnose. Also fixes two bugs found while investigating: a wedged mount kept the process alive past the end of the run, and a build without fault injection support reported success without running anything. Signed-off-by: Michael MacDonald <github@macdonald.cx>
A run that stops making progress shows from outside only as a futex wait, which does not say which wait it is. SIGUSR1 now dumps every thread's Python stack, so a stall can be read rather than guessed at. Signed-off-by: Michael MacDonald <michael.macdonald@hpe.com>
cleanup() runs from __del__, so it can fire during garbage collection, where Thread.start() waits for a thread the collector will not schedule. The thread existed to put a timeout on the flush, so the mechanism meant to bound a hang was the hang: every run whose test failed stopped there, holding the stage open for hours with its bzip2 children unreaped. Bound the waits directly instead, which needs no thread. Signed-off-by: Michael MacDonald <michael.macdonald@hpe.com>
CI Pylint spell-checks comments, but local hooks skip that check unless python3-enchant is installed, so these only fail after a push. Signed-off-by: Michael MacDonald <github@macdonald.cx>
879ee0c to
01b0f11
Compare
|
Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18756/15/execution/node/866/log |
The CI wrapper passes the variable through as an empty string when it is unset, which is not the same as absent, so the fault injection container-create test died before it ran a single iteration. Signed-off-by: Michael MacDonald <github@macdonald.cx>
2 participants
Draft — diagnostic instrumentation, not for review or merge.
Purpose: capture evidence for the intermittent NLT hang that has aborted the NLT stage
~13 times in 5 days across 10 unrelated PRs (#18709, #18731, #17550, #18707, #18729,
#18718, #18733, #18737, #18740, #18741). Each occurrence costs the full stage timeout and
discards all artifacts, so no build so far has produced any diagnostic data.
What hangs
test_alloc_fail_cont_createin the NLT stage (which runs modeall). dfuse stopsanswering FUSE requests it has already accepted; every client using the mount then blocks
in the kernel indefinitely, and NLT's sweep loop waits forever on children that can never
finish. Reproduced twice on a 16-core cloud client; intermittent (a second cluster ran the
same sweep 9 times without hanging).
The cause is not known. The obvious explanation — dfuse's threads all tied up
servicing stalled clients — is contradicted by the only thread stacks captured so far
(from a
--thread-count 2wedge, so not necessarily the same bug): the dfuse worker wasidle in
read()while the kernel held 5 queued requests, i.e. requests accepted and neveranswered rather than threads busy. Getting the equivalent stacks from a
default-configuration hang is exactly what this branch is for.
What this branch adds
NLT_FI_STALL_SECS(default 300), dump NLT thread stacks, per-child/procstate(
wchan/syscall/State) and gdb backtraces of dfuse, the agent, the engine and thestalled children — then kill the wedged children so the run fails fast with artifacts
intact instead of being aborted with none. Bounded by a wall-clock deadline, writes to a
file as well as stdout, skips targets gdb cannot attach to, resumes any target a debugger
left stopped, never blocks on children that outlive SIGKILL, and reports its own failure
rather than aborting silently.
DFuse.stop()query. Teardown currently queries the very mount that may bewedged with no timeout, which is why a hang consumes the whole stage rather than one test.
fi/allis requested but the build has no fault injectioncompiled in. Today that prints one line and exits green with the requested tests
silently skipped.
WARN, so the prime suspect in a dfuse-side wedge logs nothing at all.NLT-repeat: 5is set to get multiple chances at an intermittent failure per build. Skippragmas restrict this to builds plus NLT.
Independent of this investigation, the watchdog and the bounded teardown would convert any
future NLT hang from a lost CI run into a fast, self-documenting test failure. If they look
useful beyond the debugging, they are worth proposing separately against master.