Repository navigation
Return 5xx on database deadlock when updating task state - #75368
Merged
Merged
Conversation
Lee-W
force-pushed
the
ti-update-state-deadlock-retry
branch
from
October 22, 2026 07:08
c695588 to
f5a37b3
Compare
A deadlock is transient and the task would succeed on a retry, but the catch-all around the state update marked the task instance FAILED, which is permanent. The Task SDK client already retries 5xx responses from the Execution API, so letting the deadlock surface as a 500 gives the worker that retry. Only deadlocks are treated this way. Recognising other lock errors would change how existing callers of is_lock_not_available_error behave.
Lee-W
force-pushed
the
ti-update-state-deadlock-retry
branch
from
October 22, 2026 10:35
f5a37b3 to
5dd4665
Compare
Lee-W
marked this pull request as ready for review
October 22, 2026 12:47
ashb
approved these changes
Oct 26, 2026
amoghrajesh
approved these changes
Oct 26, 2026
Lee-W
enabled auto-merge (squash)
October 26, 2026 10:49
kaxil
approved these changes
Oct 26, 2026
Contributor
|
Hi maintainer, this PR was merged without a milestone set.
|
Contributor
Backport successfully created: v3-4-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
github-actions Bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 26, 2026
…pache#75368) A deadlock is transient and the task would succeed on a retry, but the catch-all around the state update marked the task instance FAILED, which is permanent. The Task SDK client already retries 5xx responses from the Execution API, so letting the deadlock surface as a 500 gives the worker that retry. Only deadlocks are treated this way. Recognising other lock errors would change how existing callers of is_lock_not_available_error behave. (cherry picked from commit 840d3a4) Co-authored-by: Wei Lee <weilee@apache.org>
aws-airflow-bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Oct 26, 2026
…pache#75368) A deadlock is transient and the task would succeed on a retry, but the catch-all around the state update marked the task instance FAILED, which is permanent. The Task SDK client already retries 5xx responses from the Execution API, so letting the deadlock surface as a 500 gives the worker that retry. Only deadlocks are treated this way. Recognising other lock errors would change how existing callers of is_lock_not_available_error behave. (cherry picked from commit 840d3a4) Co-authored-by: Wei Lee <weilee@apache.org>
vatsrahul1001
pushed a commit
that referenced
this pull request
Oct 26, 2026
…75368) (#75652) A deadlock is transient and the task would succeed on a retry, but the catch-all around the state update marked the task instance FAILED, which is permanent. The Task SDK client already retries 5xx responses from the Execution API, so letting the deadlock surface as a 500 gives the worker that retry. Only deadlocks are treated this way. Recognising other lock errors would change how existing callers of is_lock_not_available_error behave. (cherry picked from commit 840d3a4) Co-authored-by: Wei Lee <weilee@apache.org>
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
related: #71074
Why
A database deadlock is transient, but the catch-all in
ti_update_statemarks the task instance FAILED, which is permanent. The Task SDK client already retries Execution API 5xx responses, so letting the deadlock surface as a 500 gives the update another chance instead of failing the task.What
ti_update_statenow re-raises a deadlock error instead of marking the task instance FAILED, so the Execution API responds with a 500 and the Task SDK client retries. A newis_deadlock_errorhelper inairflow.utils.sqlalchemyrecognizes Postgres40P01(psycopg2pgcodeand psycopg3sqlstate) and MySQL1213.Only deadlocks are handled, so the existing callers of
is_lock_not_available_errorkeep their behavior. Other database errors still mark the task instance FAILED as before.is_deadlock_errorreturns False on SQLite, which has no deadlock (onlydatabase is locked), so that case keeps the previous behavior.Was generative AI tooling used to co-author this PR?
Generated-by: [Claude] following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.