ci: add main-branch DRA check for minor version-bump workflow

[ninalee12]

Description

During the recent minor version bump (9.5), each team's version-bump pipeline advanced main to the next dev minor (9.6.0-SNAPSHOT) in addition to cutting the release branch. The version-bump pipelines currently verify only that the release-branch DRA artifacts landed (staging + snapshot for ${BRANCH}.json) — they do not verify that main's DRA snapshot actually published at the new version.

This means a minor bump can report success even if main's DRA pipeline never picked up 9.6.0-SNAPSHOT. The gap was identified during the 9.5 feature freeze. Only ml-cpp and kibana had a main-branch check; this brings the remaining repos up to the same standard.

Tracking: elastic/platform-engineering-productivity#3012

What

Adds a two-step sequence, gated on WORKFLOW=minor:

1. generate-master-dra-check — runs after fetch-dra-artifacts. Computes the next dev minor from NEW_VERSION (e.g. 9.5.0 → 9.6.0) and dynamically uploads a child step via buildkite-agent pipeline upload.
2. fetch-master-dra-artifacts (uploaded at runtime) — polls master.json on GCS with elastic/json-watcher until .version equals ${next_minor}-SNAPSHOT.

Patch and major bumps are unaffected — the if: build.env("WORKFLOW") == "minor" guard means neither step is uploaded.

Notes

• The GCS alias for main's DRA artifacts is master.json (not main.json).
• The next minor is derived from NEW_VERSION, not BRANCH.

[Copilot]

Pull request overview

This PR extends the Buildkite version-bump pipeline to validate that, for minor bumps, the post-bump main (GCS alias master.json) DRA snapshot actually publishes the next dev minor (e.g. 9.5.0 → 9.6.0-SNAPSHOT). This closes a gap where minor bump pipelines could succeed while main never advanced to the new -SNAPSHOT.

Changes:

• Add a minor-workflow-gated step that computes the next dev minor from NEW_VERSION and dynamically uploads a follow-on step.
• Add a dynamically uploaded step that polls master.json on GCS via elastic/json-watcher until .version matches ${next_minor}-SNAPSHOT.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mergify]

This pull request does not have a backport label. Could you fix it @ninalee12? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

.buildkite/pipeline.version-bump.yaml:128

• This heredoc still references next_main; if you rename the version variable (e.g. to next_minor), update the embedded pipeline YAML consistently so the uploaded step uses the same computed version string.

            - echo "Polling $$next_main-SNAPSHOT on master..."
          timeout_in_minutes: 240
          retry:
            automatic:
              - exit_status: "*"

.buildkite/pipeline.version-bump.yaml:112

• The variable name next_main is misleading here: it holds the next minor version derived from NEW_VERSION, not something “main”-specific. Renaming it to something like next_minor (or next_dev_minor) will make the intent clearer and avoid confusion when maintaining this script.

This issue also appears on line 124 of the same file.

      next_main="$$major.$$((minor + 1)).0"
      echo "Waiting for $$next_main-SNAPSHOT on master"

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (4)

.buildkite/pipeline.version-bump.yaml:125

• In the dynamically uploaded step, $$next_main will expand to the shell PID if it reaches the shell unmodified. Since this heredoc is already evaluated by the shell, prefer standard $next_main here so the generated YAML is correct without relying on any additional preprocessing.

          command:
            - echo "Polling $$next_main-SNAPSHOT on master..."
          timeout_in_minutes: 240

.buildkite/pipeline.version-bump.yaml:137

• expected_value in the uploaded YAML should use the shell variable expansion for next_main. Using $$next_main risks producing an incorrect value (PID concatenation) if not rewritten before the shell expands it.

                field: ".version"
                expected_value: "$$next_main-SNAPSHOT"
                polling_interval: "30"

.buildkite/pipeline.version-bump.yaml:112

• The shell script uses $$... for variable expansion (e.g. $$major, $${NEW_VERSION}). In POSIX shells $$ expands to the process ID, so this relies on external preprocessing to rewrite $$ into $ and is easy to misread/break if the command is ever run without that preprocessing. Use normal shell variable expansions ($NEW_VERSION, $major, etc.) to keep this step self-contained and predictable.

This issue also appears in the following locations of the same file:

• line 123
• line 135

      IFS='.' read -r major minor patch <<EOF
      $${NEW_VERSION}
      EOF
      for part in "$$major" "$$minor" "$$patch"; do
        case "$$part" in

.buildkite/pipeline.version-bump.yaml:117

• The dynamically uploaded step has a fixed key. If generate-master-dra-check is retried (manual or via a retry policy added later), it will attempt to upload another step with the same key, which can cause the upload to fail or the pipeline to become non-idempotent. Since nothing references this key, consider omitting it to make retries safer.

        - label: "Fetch master DRA Artifacts"
          key: fetch-master-dra-artifacts
          depends_on: generate-master-dra-check

[github-actions]

TL;DR

The failing :smartbear-testexecute: Run unit tests: MacOS 13 job did not reach unit test execution; it failed during checkout because Buildkite attempted to checkout commit 1b726201e7b8cc176ea7e6d7d1a0fd42ff91412b, which is not present in the fetched PR refs. Immediate action: rerun the build against the current PR head (6b213a5ddd38f0e2ba8fb343c77896f977bacd8d) and ensure checkout fetches the target SHA explicitly when builds are triggered from status events.

Remediation

• Re-trigger the PR build for #7557 so the run uses current head SHA 6b213a5ddd38f0e2ba8fb343c77896f977bacd8d instead of stale SHA 1b726201e7b8cc176ea7e6d7d1a0fd42ff91412b.
• In checkout/bootstrap logic, fetch the exact commit SHA before checkout (or fallback to FETCH_HEAD when the event SHA is absent) to avoid reference is not a tree failures on force-push/rebased PRs.

Investigation details

Root Cause

This is a configuration/CI orchestration failure in source checkout, not a test/code failure from this PR diff.

The job fetches refs/pull/7557/head successfully, but then checks out a different SHA from the event payload:

• fetched head: 6b213a5ddd38f0e2ba8fb343c77896f977bacd8d
• checkout target: 1b726201e7b8cc176ea7e6d7d1a0fd42ff91412b

That target commit is unavailable in the local object graph after fetch, causing git checkout -f <sha> to fail with exit 128. The same failure repeats 3 times and aborts before .buildkite/scripts/unit_test.sh can run tests.

PR diff context: only .buildkite/pipeline.version-bump.yaml changed in this PR; this failed step never reached workflow logic in that file.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16086
• Job/step: :smartbear-testexecute: Run unit tests: MacOS 13
• Key log excerpt (/tmp/gh-aw/buildkite-logs/fleet-server-smartbear-testexecute-run-unit-tests-macos-13.txt):

  # FETCH_HEAD is now `6b213a5ddd38f0e2ba8fb343c77896f977bacd8d`
  $ git checkout -f 1b726201e7b8cc176ea7e6d7d1a0fd42ff91412b
  fatal: reference is not a tree: 1b726201e7b8cc176ea7e6d7d1a0fd42ff91412b
  ...
  🚨 Error: checking out commit "1b726201e7b8cc176ea7e6d7d1a0fd42ff91412b": exit status 128
  

Verification

• Not run in this detective workflow (read-only log analysis only).

Follow-up

• If this recurs, capture the checkout wrapper inputs (event SHA vs fetched SHA) and normalize to PR head when the event SHA is missing.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request