Skip to content

fix(beam): guard jinja_variable_flags collision with pipeline options - #39888

Open
shoemoney wants to merge 1 commit into
apache:masterfrom
shoemoney:fix/beam-jinja-collision
Open

fix(beam): guard jinja_variable_flags collision with pipeline options#39888
shoemoney wants to merge 1 commit into
apache:masterfrom
shoemoney:fix/beam-jinja-collision

Conversation

@shoemoney

@shoemoney shoemoney commented Aug 25, 2026

Copy link
Copy Markdown

Fixes Yaml jinja variable exact-name collision stealing pipeline option at sdks/python/apache_beam/yaml/main.py:64.
#37475

Bug: _preparse_jinja_flags adds every jinja_variable_flags entry as an argparse flag without checking against known pipeline options. A jinja variable named runner, project, temp_location, etc. captures the pipeline flag and converts it into jinja_variables, removing it from pipeline_args.

Fix: Guard the loop by checking flag_name against PipelineOptions.get_all_options(). If the name collides (dash normalized to underscore), skip adding it to the jinja parser and require the variable via --jinja_variables JSON instead.

Evidence: RED->GREEN verified with mocked PipelineOptions. Before fix, --jinja_variable_flags=runner --runner=DirectRunner produces --jinja_variables={"runner": "DirectRunner"} and loses the pipeline flag. After fix, runner stays in pipeline_args and only non-colliding vars are promoted. Dash variant temp-location also guarded. Existing behavior for non-colliding vars unchanged.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Fix verified RED->GREEN. Yaml jinja variable exact-name collision steals pipeline option at main.py:64
@derrickaw

derrickaw commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

The fix seems good, but can you add a test case to cover this fix. Thanks

@derrickaw

Copy link
Copy Markdown
Collaborator

R: @Abacn since you opened the initial issue. Thanks

@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants