Fix task config defaults for language SDKs by methatp0 · Pull Request #75267 · apache/airflow

methatp0 · 2026-10-21T05:54:58Z

Preserve explicitly configured task fields that match schema defaults in the language SDK serializers.

Populate missing task defaults from Airflow configuration without overriding values already supplied by the SDK. Add regression tests and update serialization conformance checks and the SDK parsing ADR.

Closes: #74094

Was generative AI tooling used to co-author this PR?

Yes (GPT-6)

Generated-by: OpenAI ChatGPT (GPT-6) following "the guidelines" (https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)

boring-cyborg · 2026-10-21T05:55:03Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

jason810496

Thanks for the PR, the overall direction is correct but there're some places to refine.

jason810496 · 2026-10-21T08:20:01Z

+	// keepDefaults preserves an explicitly set task field when it equals the schema default.
+	keepDefaults bool


A per-SDK flag for a rule that should hold everywhere. Dropping it here, in taskSpecRules and in the write condition leaves schemaDefault and isSameJSON with no reader.

Suggested change

// keepDefaults preserves an explicitly set task field when it equals the schema default.

keepDefaults bool

jason810496 · 2026-10-21T08:20:01Z

+	fields:       taskSpecFields,
+	keepDefaults: true,


Suggested change

fields: taskSpecFields,

keepDefaults: true,

fields: taskSpecFields,

jason810496 · 2026-10-21T08:20:02Z

+// writeSpecFields writes into data each field of spec that rules does not skip. It leaves out
+// fields that are unset, and normally those that hold their schema default. Task rules preserve
+// explicitly set schema-default values so Airflow config defaults cannot replace them. A field
+// is unset when it holds its zero value. A pointer field is unset only when nil, so it can explicitly
+// set the zero value of its type, such as false or 0.


Suggested change

// writeSpecFields writes into data each field of spec that rules does not skip. It leaves out

// fields that are unset, and normally those that hold their schema default. Task rules preserve

// explicitly set schema-default values so Airflow config defaults cannot replace them. A field

// is unset when it holds its zero value. A pointer field is unset only when nil, so it can explicitly

// set the zero value of its type, such as false or 0.

// writeSpecFields writes into data each field of spec that rules does not skip. It leaves out only

// the fields that are unset: Airflow fills an unset config-backed field from its own config, so a

// value the author set has to be written even when it equals the schema default. A field is unset

// when it holds its zero value. A pointer field is unset only when nil, so it can explicitly set

// the zero value of its type, such as false or 0.

jason810496 · 2026-10-21T08:20:02Z

 		}
 		encoded, set := encodeSpecValue(value.Field(i), rules.set[name])
-		if !set || (field.schemaDefault != nil && isSameJSON(encoded, field.schemaDefault)) {
+		if !set || (!rules.keepDefaults && field.schemaDefault != nil && isSameJSON(encoded, field.schemaDefault)) {


Suggested change

if !set || (!rules.keepDefaults && field.schemaDefault != nil && isSameJSON(encoded, field.schemaDefault)) {

if !set {

jason810496 · 2026-10-21T08:20:02Z

+  /** Preserve an explicitly set value even when it equals the schema default. */
+  readonly keepDefaults?: boolean;


Same here: drop the flag and the field.default comparison instead of making it conditional.

Suggested change

/** Preserve an explicitly set value even when it equals the schema default. */

readonly keepDefaults?: boolean;

jason810496 · 2026-10-21T08:20:02Z

+        # Native runtimes cannot read Airflow's task defaults. Supply only the config-backed
+        # fields the Python serializer exposes through client_defaults, so explicit values in
+        # the task itself (including values equal to schema defaults) retain precedence.
+        task_default_keys = {
+            "owner",
+            "queue",
+            "retries",
+            "retry_delay",
+            "weight_rule",
+            "execution_timeout",
+            "email_on_failure",
+            "email_on_retry",
+        }
+        task_defaults = {
+            key: value
+            for key, value in OperatorSerialization.generate_client_defaults().items()
+            if key in task_default_keys
+        }


The whitelist is a second copy of knowledge generate_client_defaults() already has: it drops every OPERATOR_DEFAULTS value equal to its schema default, which today leaves exactly these eight keys (pool_slots and priority_weight are hardcoded 1, allow_nested_operators is True on both sides, the rest are None or empty). Keeping the list means a native Dag silently diverges from a Python Dag the day a new config-backed task default lands in OPERATOR_DEFAULTS.

Suggested change

# Native runtimes cannot read Airflow's task defaults. Supply only the config-backed

# fields the Python serializer exposes through client_defaults, so explicit values in

# the task itself (including values equal to schema defaults) retain precedence.

task_default_keys = {

"owner",

"queue",

"retries",

"retry_delay",

"weight_rule",

"execution_timeout",

"email_on_failure",

"email_on_retry",

}

task_defaults = {

key: value

for key, value in OperatorSerialization.generate_client_defaults().items()

if key in task_default_keys

}

# Native runtimes cannot read Airflow's task defaults, so supply the table a Python Dag

# carries. generate_client_defaults() already drops every value equal to its schema

# default, which leaves exactly the config-backed fields.

task_defaults = OperatorSerialization.generate_client_defaults()

jason810496 · 2026-10-21T08:20:02Z

+    A key either side leaves out is fine when the value the other side writes equals the schema
+    default. Python may omit a value matching its ``client_defaults`` table, and native SDKs now preserve
+    explicitly set task values even when they equal the schema default.


compare_dag still passes {} as the Dag-level defaults, so once the Dag path writes explicit defaults too, a fail_fast: false the author set is reported as a key Python does not write. It needs a Dag defaults table built the way get_task_defaults() builds the operator one.

Suggested change

A key either side leaves out is fine when the value the other side writes equals the schema

default. Python may omit a value matching its ``client_defaults`` table, and native SDKs now preserve

explicitly set task values even when they equal the schema default.

A key either side leaves out is fine when the value the other side writes equals the schema

default: Python omits a value matching its ``client_defaults`` table, and an SDK writes every

field the author set, including one that happens to equal the default.

jason810496 · 2026-10-21T08:20:02Z

+**Task config defaults:** A runtime writes every task field the author explicitly set, even when
+its value equals the schema default. A missing task field is filled from the optional
+`client_defaults.tasks` table. Because language runtimes cannot read Airflow's configuration,
+core supplies this table from its own config when the payload does not already include it. Only
+config-backed task fields are supplied (`owner`, `queue`, `retries`, `retry_delay`,
+`weight_rule`, `execution_timeout`, `email_on_failure`, and `email_on_retry`). A task field
+present in the serialized task takes precedence, so an explicit `retries=0` is not replaced by
+a non-zero `[core] default_task_retries`. The Go SDK also preserves explicit schema-default values
+when its spec represents them as set; its non-pointer scalar zero values are treated as unset and
+receive the configured default.


Worth stating as one rule rather than a Go exception.

Suggested change

**Task config defaults:** A runtime writes every task field the author explicitly set, even when

its value equals the schema default. A missing task field is filled from the optional

`client_defaults.tasks` table. Because language runtimes cannot read Airflow's configuration,

core supplies this table from its own config when the payload does not already include it. Only

config-backed task fields are supplied (`owner`, `queue`, `retries`, `retry_delay`,

`weight_rule`, `execution_timeout`, `email_on_failure`, and `email_on_retry`). A task field

present in the serialized task takes precedence, so an explicit `retries=0` is not replaced by

a non-zero `[core] default_task_retries`. The Go SDK also preserves explicit schema-default values

when its spec represents them as set; its non-pointer scalar zero values are treated as unset and

receive the configured default.

**Task config defaults:** A runtime writes every task field the author set and leaves out the

ones they did not; a value equal to the schema default is still written. A missing task field is

filled from the optional `client_defaults.tasks` table, which core supplies from its own config

because a language runtime cannot read it. Only config-backed task fields are supplied (`owner`,

`queue`, `retries`, `retry_delay`, `weight_rule`, `execution_timeout`, `email_on_failure` and

`email_on_retry`). A field present in the serialized task wins over the table, so an explicit

`retries=0` is not replaced by a non-zero `[core] default_task_retries`. In Go a non-pointer

scalar at its zero value counts as unset and takes the configured default; a pointer field

expresses an explicit zero.

jason810496 · 2026-10-21T08:24:50Z

+        if task_defaults:
+            client_defaults = serialized_obj.setdefault("client_defaults", {})
+            if isinstance(client_defaults, dict):
+                existing_task_defaults = client_defaults.setdefault("tasks", {})
+                if isinstance(existing_task_defaults, dict):


No need the if task_defaults: and if isinstance here, we can assert the client_defaults is correct as they're Airflow core module.

jason810496 · 2026-10-21T08:27:03Z

+        DagSerialization.fill_config_defaults(data)
+
+        assert data["client_defaults"] == {"tasks": {"retries": 3, "queue": "celery_q"}}
+


We should also add a new test that define a real Dag with some operators like in #74869 to exercise the set default behavior.

Fix task config defaults for language SDKs

c2440f9

methatp0 requested review from amoghrajesh, ashb, bolkedebruin, bugraoz93, gopidesupavan, jason810496, jscheffl, potiuk and uranusjr as code owners October 21, 2026 05:54

boring-cyborg Bot added area:DAG-processing area:dev-tools area:go-sdk area:java-sdk labels Oct 21, 2026

boring-cyborg Bot added area:ts-sdk backport-to-v3-4-test Backport to v3-4-test labels Oct 21, 2026

jason810496 requested changes Oct 21, 2026

View reviewed changes

Navigation Menu

Repository navigation

Fix task config defaults for language SDKs - #75267

Fix task config defaults for language SDKs#75267
methatp0 wants to merge 1 commit into
apache:mainfrom
methatp0:fix/lang-sdk-task-config-defaults

methatp0 commented Oct 21, 2026

Uh oh!

boring-cyborg Bot commented Oct 21, 2026

Uh oh!

jason810496 left a comment

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

jason810496 Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// keepDefaults preserves an explicitly set task field when it equals the schema default.
		keepDefaults bool

	fields: taskSpecFields,
	keepDefaults: true,
	fields: taskSpecFields,

	if !set \|\| (!rules.keepDefaults && field.schemaDefault != nil && isSameJSON(encoded, field.schemaDefault)) {
	if !set {

		/** Preserve an explicitly set value even when it equals the schema default. */
		readonly keepDefaults?: boolean;

		DagSerialization.fill_config_defaults(data)

		assert data["client_defaults"] == {"tasks": {"retries": 3, "queue": "celery_q"}}

Conversation

methatp0 commented Oct 21, 2026

Uh oh!

boring-cyborg Bot commented Oct 21, 2026

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants