fix(fuse): handle shutdown run_all errors (#1082)#1175
Conversation
yangcx000
left a comment
There was a problem hiding this comment.
LGTM — nicely scoped fix for a real bug, verified against the code.
Root cause confirmed. run_all returns CommonResult<()> and is tokio::spawn-ed, so run_all_handle.await is Result<CommonResult<()>, JoinError> — genuinely double-layered. The old if let Err(e) = run_all_handle.await in both signal arms matched only the outer JoinError and silently dropped an inner Ok(Err(..)). Good that the tokio::select! first arm already destructures all three cases, so scoping the fix to the two re-await sites (and leaving that arm untouched, since it needs the distinct RUN_ALL_ERROR/RUN_ALL_PANIC metrics) is exactly right.
The two arms diverge correctly. TERM logs and continues to unmount, while SIGUSR1 refuses to persist and returns — the safe split: a hot-upgrade persist should not serialize potentially-inconsistent state, and falling back to a clean unmount via the mnts drop is the right default. I verified the auto_unmount timing the comment describes: FuseMnt::auto_unmount defaults to true and is only flipped to false inside persist_inner, so the return Err path drops mnts with auto-unmount still on (mounts torn down), while a persist error happens after that flag is cleared (mounts stay). RunCleanupGuard's early-return Drop is correct too. err.into() resolves cleanly to CommonError (JoinError: Error + Send + Sync), and the three unit tests cover all three branches.
One optional, non-blocking nit: the helper's doc-comment says the flatten guards against "a clean task join [hiding] a real receiver/sender error." Looking at run_all, the functional receiver/sender errors (start().await returning Err) are already logged-and-swallowed inside the spawned closures, which return (). The only way run_all yields an inner Err is handle.await? propagating a child accept/send task that panicked or was aborted — so the wording is slightly off; consider "so a clean join of the run_all task cannot hide a panic in one of its child accept/send tasks." The fix's value (surfacing those panics at shutdown) is real regardless. Also join_error_is_preserved's contains("cancelled") softly couples to tokio's Display wording — fine to keep, or relax to is_err(). Neither is blocking.
Summary
Inspect both result layers when awaiting the spawned
run_alltask during signal shutdown. Termination shutdown now logs receiver/sender failures, while SIGUSR1 refuses to persist state after a failed shutdown.Issue Describe / Design
Closes #1082
Root cause: the TERM and SIGUSR1 branches only checked the outer Tokio
JoinErrorand silently ignored an innerOk(Err(...))returned byrun_all.Fix approach: flatten the nested join result in one helper and use it in both signal branches. TERM records the failure and continues normal unmount; SIGUSR1 returns the failure without persisting potentially unreliable state.
Changes
curvine-fuse/src/session/fuse_session.rsrun_allresult and add regression testsTest verified
cargo test -p curvine-fuse --lib run_all_shutdown_result_tests -- --nocapturecargo test -p curvine-fuse --lib session::fuse_session -- --nocapturecargo clippy -p curvine-fuse --lib --tests -- -D warningsmake format/dev/fusemount and SIGTERM smoke testDependencies