Skip to content

DAOS-19173 pool: destroy timeout tiering and serialization - #18523

Open
kccain wants to merge 1 commit into
release/2.8from
kccain/daos_19173_rel2p8
Open

DAOS-19173 pool: destroy timeout tiering and serialization#18523
kccain wants to merge 1 commit into
release/2.8from
kccain/daos_19173_rel2p8

Conversation

@kccain

@kccain kccain commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

For MD on SSD configurations, engine local SCM capacity based corpc timeout configuration:

  • Add pool_destroy_local_scm_size() get engine-local SCM size
  • Add pool_destroy_rpc_timeout() to customize destroy CoRPC timeout for ds_mgmt_tgt_pool_destroy_ranks()
  • non-MD-on-SSD case falls back to default timeout

And serialize pool target destroy handling for when a previous handler invocation (whose CoRPC initiator timed out) is still busy performing expensive subtree destruction / file unlinking:

  • Refactor ds_pooltgts with dual synchronization domains: Create: dpt_create_mutex/cv + dpt_creates_ht (existing create-cancel) Destroy: dpt_destroy_mutex/cv + dpt_destroys_ht (new serialization)
  • Add ds_pooltgts_destroy_rec for destroy-in-flight tracking hash table
  • Add to ds_mgmt_hdlr_tgt_destroy() the destroy serialization

Features: pool

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@github-actions

github-actions Bot commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Ticket title is 'dmg pool destroy very slow'
Status is 'In Progress'
Labels: 'request_for_2.8'
https://daosio.atlassian.net/browse/DAOS-19173

@kccain
kccain force-pushed the kccain/daos_19173_rel2p8 branch from 2a34b97 to ab8f393 Compare June 23, 2026 13:48
@daosbuild3

daosbuild3 commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18523/2/testReport/

@Michael-Hennecke
Michael-Hennecke self-requested a review June 23, 2026 14:42
@daosbuild3

daosbuild3 commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18523/2/execution/node/1116/log

@kccain
kccain marked this pull request as ready for review July 6, 2026 13:54
@kccain
kccain requested review from liw and wangshilong July 6, 2026 13:54
@kccain

kccain commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

reviewing the patch in parallel to @Michael-Hennecke patch deployment and test. Potentially a candidate for 2.8-rc2 (subject to approval to merge)

@kccain
kccain removed request for liw and wangshilong July 6, 2026 14:05
@kccain
kccain marked this pull request as draft July 6, 2026 14:06
@kccain kccain mentioned this pull request Jul 6, 2026
Merged
6 tasks
@kccain
kccain marked this pull request as ready for review July 10, 2026 12:23
@kccain
kccain marked this pull request as draft July 10, 2026 12:29
@kccain
DAOS-19173 pool: destroy timeout tiering and serialization
7da9c16 
For MD on SSD configurations, engine local SCM capacity
based corpc timeout configuration:
- Add pool_destroy_local_scm_size() get engine-local SCM size
- Add pool_destroy_rpc_timeout() to customize destroy CoRPC timeout
  for ds_mgmt_tgt_pool_destroy_ranks()
- non-MD-on-SSD case falls back to default timeout

And serialize pool target destroy handling for when a previous
handler invocation (whose CoRPC initiator timed out) is still
busy performing expensive subtree destruction / file unlinking:
- Refactor ds_pooltgts with dual synchronization domains:
  Create: dpt_create_mutex/cv + dpt_creates_ht (existing create-cancel)
  Destroy: dpt_destroy_mutex/cv + dpt_destroys_ht (new serialization)
- Add ds_pooltgts_destroy_rec for destroy-in-flight tracking hash table
- Add to ds_mgmt_hdlr_tgt_destroy() the destroy serialization

Features: pool

Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
@kccain
kccain force-pushed the kccain/daos_19173_rel2p8 branch from ab8f393 to 7da9c16 Compare July 10, 2026 12:31
@daosbuild3

daosbuild3 commented Jul 10, 2026 •
edited by kccain
Loading

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18523/3/execution/node/1649/log

container/boundary.py failure is an instance of known issue https://daosio.atlassian.net/browse/DAOS-18610 (fix not yet landed to release/2.8 branch)

erasurecode/online_rebuild_mdtest.py tests passed, though there was an CI problem processing collected logfiles after the run

Error running /usr/lib/daos/TESTING/ftest/cart/cart_logtest.py on the /var/tmp/daos_testing/*log* files on hdr-147

@kccain kccain added the clean-cherry-pick Cherry-pick from another branch that did not require additional edits label Jul 14, 2026
@kccain
kccain marked this pull request as ready for review July 14, 2026 15:00
@kccain
kccain requested a review from liw July 14, 2026 18:09
@kccain
kccain requested a review from a team July 15, 2026 16:00
@kccain

kccain commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@daos-stack/daos-gatekeeper , understand that this PR cannot be merged to release/2.8 until merge approval is granted in https://daosio.atlassian.net/browse/DAOS-19173

@daltonbohning daltonbohning added this to the release-2.8.1 milestone Jul 15, 2026
@daltonbohning daltonbohning added the waiting-for-merge-approval Waiting for merge approval label Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liw liw liw approved these changes
@wangshilong wangshilong wangshilong approved these changes
@Michael-Hennecke Michael-Hennecke Awaiting requested review from Michael-Hennecke

Assignees

No one assigned

Labels

clean-cherry-pick Cherry-pick from another branch that did not require additional edits waiting-for-merge-approval Waiting for merge approval

Milestone

release-2.8.1

Development

Successfully merging this pull request may close these issues.

5 participants

@kccain @daosbuild3 @liw @wangshilong @daltonbohning