fix: hold strong references to lifespan background tasks#6629
Open
sricursion wants to merge 1 commit into
Open
fix: hold strong references to lifespan background tasks#6629sricursion wants to merge 1 commit into
sricursion wants to merge 1 commit into
Conversation
|
|
dosubot
Bot
added
the
size:S
asyncio.create_task() results were discarded in startup() (maintenance window watcher) and lifespan() (pending-tasks debug logger). The event loop only keeps weak references to tasks, so either task could be garbage-collected and silently cancelled mid-execution. Keep the tasks in a module-level set with a done-callback discard, per the pattern recommended in the asyncio docs. A separate set is used rather than the request-scoped background_tasks set so the debug logger's pending-events count is not skewed by long-lived service tasks. Fixes keephq#6552
sricursion
force-pushed
the
fix/6552-background-task-strong-refs
branch
from
July 12, 2026 14:38
55c041a to
3483497
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked Issue
Fixes #6552
Description
startup()andlifespan()create fire-and-forget tasks withasyncio.create_task(...)and discard the returned Task:async_process_watcher)check_pending_tasksdebug loggerThe event loop keeps only weak references to tasks, so either task can be garbage-collected and silently cancelled mid-execution — no traceback; the watcher just stops running and maintenance-window alert recovery silently breaks.
This change adds a module-level
service_tasksset and acreate_service_task()helper that holds a strong reference and discards the task on completion, per the pattern recommended in the asyncio docs. Both call sites now go through it.A separate set is used instead of the request-scoped
background_tasksset on purpose: that set's length is whatcheck_pending_tasksreports as pending events (androutes/alerts.pyadds event-processing tasks to it), so long-lived service tasks in the same set would permanently skew the count.Test
Added
tests/test_service_tasks.py: the task is strongly referenced while running and discarded on completion.Also verified under forced GC (all local references dropped,
gc.collect()×3 — the task survives and runs to completion) and on a live server boot withWATCHER=trueandKEEP_DEBUG_TASKS=true: the watcher completed multiple full cycles and the debug logger ran continuously for 2+ minutes with zerotask was destroyed/exception was never retrievedlog entries. Fullpytest --non-integrationsuite passes with no regressions.ruff checkpasses.