Repository navigation
Respect remaining execution_timeout when a deferred task resumes - #75317
sandeep-paluru wants to merge 3 commits into
Conversation
Resuming a deferred task restarted the timeout at its full duration, so a task could run longer than the limit configured for it.
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
|
Note for maintainers: I think this should wait until #74869 is merged, since the design around Under that shape, I'd argue the remaining-time-after-deferral computation belongs outside the Python task runner. |
|
Thanks, that makes sense. I read through #74869 and agree the runner-side math won't fit once the supervisor owns the deadline. Once it lands, I think the natural spot is I'll hold this one until #74869 merges and then rework it into that shape, with the tests moved to the execution API side. Happy to hear if a maintainer would rather see it go a different way. |
…ion-timeout Co-authored-by: Sandeep Paluru <sandeep-paluru@users.noreply.github.com>
A deferred task was given a fresh full timeout every time it resumed, so it could run longer than the limit configured for it.
|
Reworked this on top of #74869 now that it's merged. The run route sends the remaining timeout on resume through |
|
@jason810496 since you reviewed and merged #74869, would you mind taking a look at this when you get a chance? It builds directly on the TIRunContext.execution_timeout change from that PR. No rush, thanks. |
A deferred task was given a fresh
execution_timeoutevery time it resumed. Time already spent, including the deferral, did not count, so the task could run longer than the limit configured for it.Since #74869 the supervisor arms the duration on
TIRunContext.execution_timeout, and every SDK uses that number. When the task is resuming (next_methodis set), the run route now sends the time still left: the configured timeout minus the time since the originalstart_date, clamped at 0. A task that is not resuming still gets the full timeout. If no timeout is set, the field stays unset. A value of 0 means the limit has already elapsed, so the task fails as a timeout promptly. Astart_dateahead of the clock does not extend the timeout.The Python runner applies those seconds to its in-process timeout, which is what raises
AirflowTaskTimeoutand runson_kill. It uses the task's ownexecution_timeoutonly when the server omits the field, and it no longer subtracts elapsed time itself. How long a trigger may run while the task is still deferred is unchanged.No new execution API version.
execution_timeoutis already onTIRunContextfrom #74869 with the same type, and older clients still have the field stripped.Tests in
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.pyusetime_machine: a fresh run keeps the full timeout, a resume with time left gets the remainder, a clock ahead ofstart_datedoes not extend it, an already elapsed resume returns 0 with the field present, and a task with no timeout omits the field. Task SDK tests check that the runner uses the server seconds, falls back when the field is omitted, and raisesAirflowTaskTimeoutbefore the callable when the server sends 0.prek pre-commit checks pass, including airflow-core and task-sdk mypy, ruff, and the execution API version check.
closes: #73475
Was generative AI tooling used to co-author this PR?
Generated-by: AI coding assistant following the guidelines