Skip to content

Fix Edge workers not reporting the final state of deadline callbacks - #74784

Merged
Lee-W merged 4 commits into
apache:mainfrom
FrankYang0529:fix-edge3-callback-log-push
Oct 22, 2026
Merged

Lee-W merged 4 commits into
apache:mainfrom
FrankYang0529:fix-edge3-callback-log-push

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Why

  • On Airflow 3.3+ and edge3 3.8.0+ run executor callbacks, such as the SyncCallback of a Deadline Alert, on Edge workers. With the default [edge] push_logs = True, the worker never reports whether the callback succeeded or failed.
  • The worker pushes the callback's log with the job key that the Edge executor builds for callbacks. That key has dag_id="ExecuteCallback". The log route on the API server looks up a task instance for that key. No such task instance exists, so the route fails with HTTP 500. Even a callback whose code logs nothing leaves a non-empty log file, so the push fails for every callback.

How

  • API server: for a callback job, the log route reads log_path from the workload JSON that the scheduler stored in the command column of the edge_job row. The worker reads the callback log from that same relative path. For a callback whose row or log_path is missing, both log routes return 404 instead of 500.
  • Worker: pushing logs is now best effort. The worker logs an error from a push and still reports the final state. If pushes keep failing with 5xx, each push still waits for the client's retries, so the state can arrive a few minutes late. The worker change also handles an older API server that does not have the log route change above.

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

@boring-cyborg boring-cyborg Bot added area:providers provider:edge Edge Executor / Worker (AIP-69) / edge3 labels Oct 15, 2026
@FrankYang0529
FrankYang0529 force-pushed the fix-edge3-callback-log-push branch from 678a65a to b43c75e Compare October 16, 2026 03:24
@FrankYang0529
FrankYang0529 marked this pull request as ready for review October 16, 2026 06:31
Comment thread providers/edge3/src/airflow/providers/edge3/worker_api/routes/logs.py Outdated
Comment thread providers/edge3/src/airflow/providers/edge3/worker_api/routes/logs.py Outdated
Comment thread providers/edge3/src/airflow/providers/edge3/worker_api/routes/logs.py Outdated
@FrankYang0529
FrankYang0529 force-pushed the fix-edge3-callback-log-push branch from dfcf3e7 to af38628 Compare October 22, 2026 04:40
@Lee-W
Lee-W merged commit 7483c23 into apache:main Oct 22, 2026
72 checks passed
@FrankYang0529
FrankYang0529 deleted the fix-edge3-callback-log-push branch October 22, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:edge Edge Executor / Worker (AIP-69) / edge3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants