diff --git a/.ci/changed-file-filters.yml b/.ci/changed-file-filters.yml index 8a71a60b7..46f39b325 100644 --- a/.ci/changed-file-filters.yml +++ b/.ci/changed-file-filters.yml @@ -50,3 +50,14 @@ e2e: - *exclude-docs-directory ciFilters: - '.ci/changed-file-filters.yml' +# The macOS CI gate on Bitrise. Narrower than the e2e filter on purpose: these files +# decide which macOS jobs run, so a change to one has to rerun them all, while a Maestro +# flow edit under e2e/ must not. +iosCiConfig: + - 'e2e/bitrise.yml' + - 'e2e/config/ios_ci.yml' + - 'e2e/lib/ios_ci_run_plan.rb' + - 'e2e/lib/ios_ci_reporter.rb' + - 'e2e/scripts/ios_ci_run_plan' + - 'e2e/scripts/report_ios_ci_results' + - 'e2e/scripts/bitrise_ci_helpers' diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2dce65f9e..193a26236 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -116,7 +116,8 @@ jobs: - 'e2e/flows/**' - 'e2e/tests/**' - 'e2e/config.yaml' - - 'e2e/config/matrix.yml' + - 'e2e/config/**' + - 'e2e/bitrise.yml' - '.ci/changed-file-filters.yml' - '.github/workflows/ci.yml' diff --git a/e2e/BITRISE.md b/e2e/BITRISE.md index 55c8372fb..0433db851 100644 --- a/e2e/BITRISE.md +++ b/e2e/BITRISE.md @@ -52,6 +52,31 @@ For example, editing `platforms/react-native/README.md` matches the coarse `chan The GitHub checks are kept non-blocking while the suite stabilizes; they become merge-blocking only once the "Checkout Kit E2E" check is marked required in branch protection. +## The `ci-ios` pipeline + +`ci-ios` is the second pipeline in `e2e/bitrise.yml`. It runs the four macOS jobs that used to run on GitHub Actions: the Swift package tests, the Swift sample build and test, the React Native iOS sample build, and the React Native iOS tests. It is separate from `e2e` rather than a set of extra workflows inside it, because Bitrise reports one status per pipeline: sharing one would tie a merge gate to the BrowserStack device flake that the E2E checks deliberately keep non-blocking. + +### Its trigger carries no `changed_files` + +Unlike `e2e`, the `ci-ios` `trigger_map` entry has no filter at all. `ci-ios` is a merge-blocking check, and a required check that never posts leaves a pull request permanently unmergeable — so the pipeline has to start on every pull request, including a docs-only one. + +Selection happens inside the pipeline instead. The Linux `ci-ios-plan` workflow reads the pull request's changed files, applies the shared filter groups in `.ci/changed-file-filters.yml` through `e2e/config/ios_ci.yml`, and publishes one `CI_IOS_*` variable per job with `share-pipeline-variable`. Each macOS workflow guards on its own variable with `run_if`. A change that needs no macOS job runs the Linux plan and the report, and nothing else. + +This is the same two-layer idea as `e2e` — a cheap first pass, then a precise runtime decision — with the first layer set to "always". + +### The check is self-posted + +`ci-ios-report` runs with `should_always_run: workflow` and posts the `Checkout Kit iOS` Check Run itself, through `e2e/scripts/report_ios_ci_results`. Bitrise's own pipeline status cannot tell the two kinds of not-run apart: + +- a job the plan did not select is a **pass** — there was nothing to build +- a job the plan did select but that never finished is a **failure** + +The reporter also fails when `ci-ios-plan` itself fails, rather than reporting green off an empty selection. `e2e/test/ios_ci_reporter_test.rb` pins all three cases. + +### Changing which files select which job + +Edit `e2e/config/ios_ci.yml`, not the workflows. `e2e/test/ios_ci_run_plan_test.rb` asserts set equality between the variables the plan emits and the `run_if` expressions parsed out of `e2e/bitrise.yml`, so a job added on one side and not the other fails the Ruby tests. + ## Duplicate PR build cancellation Duplicate in-progress PR pipelines are cancelled by Bitrise native Rolling builds rather than a repo-owned cancellation script. Under **Project settings > Builds > Build strategy**, **Abort builds triggered by pull requests** and **Abort running builds** are enabled, so a newer PR build cancels the older one. @@ -175,4 +200,4 @@ The pipeline uses Bitrise cache steps for key-based pnpm/CocoaPods/Gradle cache Do not add `activate-build-cache-for-xcode` or `activate-build-cache-for-gradle`; the Bitrise Build Cache add-on is disabled for Shopify Bitrise apps. -Ruby and Node versions are pinned in `e2e/bitrise.yml` via the Bitrise `tools:` configuration (`ruby: 3.3.6`, `nodejs: 22.14.0`), which Bitrise installs before each workflow runs. Pin exact versions that the target stacks preinstall so setup stays fast and reproducible; a version the stack does not ship is installed on demand and is slower. pnpm is pinned separately through Corepack via the `packageManager` field in `platforms/react-native/package.json`. +Ruby and Node versions are pinned in `e2e/bitrise.yml` via the Bitrise `tools:` configuration (`ruby: "3.4:installed"`, `nodejs: 22.14.0`), which Bitrise installs before each workflow runs. The `:installed` suffix tells each stack to use its own preinstalled 3.4.x rather than compiling one from source. Pin exact versions that the target stacks preinstall so setup stays fast and reproducible; a version the stack does not ship is installed on demand and is slower. pnpm is pinned separately through Corepack via the `packageManager` field in `platforms/react-native/package.json`. diff --git a/e2e/RUNBOOK.md b/e2e/RUNBOOK.md index a6fb280de..1f48f770f 100644 --- a/e2e/RUNBOOK.md +++ b/e2e/RUNBOOK.md @@ -115,3 +115,35 @@ The report keeps a single sticky PR comment, identified by a hidden marker, and in place on every build, so green runs never add a second comment. Because that comment always carries the Tophat install links, it is posted even on a fully green run — a passing build stays installable from the PR. + +## The iOS check failed or never posted + +`Checkout Kit iOS` comes from the `ci-ios` pipeline, described in `BITRISE.md`. Three +layers can break, and the symptom tells you which one. Work down the list in order. + +**The check never appears.** The pipeline did not start. Its `trigger_map` entry has no +file filter, so the usual cause is the branch head: Bitrise reads `trigger_map` from the +pull request's own commit, and a branch older than the entry never triggers. Rebase on +`main` and push. The entry also sets `draft_pull_request_enabled: false`, so a draft posts +nothing until it is marked ready. + +**The check is red but every job says skipped.** `ci-ios-plan` failed, and the reporter +refuses to call an empty selection green. Open that workflow's log. It fetches the changed +file list from GitHub and reads `e2e/config/ios_ci.yml`, so the usual causes are an expired +build token or a malformed config file. + +**The check is red and names a job.** That macOS workflow failed or never finished. The +reporter lists a selected job that produced no result as a failure, so a timeout and a +compile error look different in the summary: a timeout shows as missing, a compile error +shows as failed. Both link back to the Bitrise pipeline. + +**The check is green and every job says skipped.** Expected on a change that touches no +iOS input — documentation, Android, or web. `ci-ios-plan` and `ci-ios-report` still run, +which costs about a minute on Linux. To confirm the selection is right rather than empty by +accident, run the plan locally against the same file list: + +```bash +ruby e2e/scripts/ios_ci_run_plan selected-jobs --changed-file +``` + +It prints a comma-separated job list, or nothing when no macOS job is needed. diff --git a/e2e/bitrise.yml b/e2e/bitrise.yml index 93c45b974..64dfb72f0 100644 --- a/e2e/bitrise.yml +++ b/e2e/bitrise.yml @@ -13,6 +13,10 @@ meta: app: envs: + # One device for every macOS job. GitHub Actions grew two values by accident + # (iPhone 17 Pro for Swift, iPhone 16 Pro for React Native); the port keeps a + # single one so a runtime change cannot break half the jobs. + - CI_IOS_SIMULATOR_DEVICE: iPhone 17 Pro - E2E_BROWSERSTACK_API_RETRIES: "1" - E2E_BROWSERSTACK_TIMEOUT_SECONDS: "1800" - E2E_BROWSERSTACK_POLL_SECONDS: "30" @@ -40,6 +44,14 @@ trigger_map: regex: '^(platforms/(swift|android|react-native)|protocol|e2e|\.ci)/.*|^Package\.(swift|resolved)$' pipeline: e2e + # Deliberately unfiltered. ci-ios is a required check, and a required check that + # never posts leaves a pull request permanently unmergeable. The cheap Linux + # ci-ios-plan workflow decides which macOS jobs run, and ci-ios-report always posts + # — including for a change that needs no macOS job at all. + - pull_request_source_branch: "*" + draft_pull_request_enabled: false + pipeline: ci-ios + pipelines: e2e: # Bitrise's default report name is ci/bitrise//, @@ -86,6 +98,25 @@ pipelines: - e2e-execute-browserstack-run should_always_run: workflow + # The macOS merge gate. Separate from the e2e pipeline on purpose: Bitrise reports one + # status per pipeline, and sharing would tie the merge gate to the BrowserStack device + # flake that e2e/BITRISE.md deliberately keeps non-blocking. + ci-ios: + status_report_name: "ci/bitrise//" + workflows: + ci-ios-plan: {} + ci-ios-swift-package-tests: + run_if: + expression: '{{ enveq "CI_IOS_SWIFT_PACKAGE_TESTS" "true" }}' + depends_on: + - ci-ios-plan + # No run_if: this workflow is the required check, so it posts on every build, + # including one where ci-ios-plan selected nothing and every job was skipped. + ci-ios-report: + depends_on: + - ci-ios-swift-package-tests + should_always_run: workflow + # Deliberately absent from trigger_map: a scheduled build in the Bitrise UI is # the only thing that starts this, alongside a manual start or the API. Do not # add a `triggers:` block either — Bitrise rejects a config that mixes it with @@ -177,6 +208,21 @@ step_bundles: inputs: - key: *rn_cocoapods_cache_key - paths: platforms/react-native/sample/ios/Pods + # xcodebuild spends the first minute of a test run waiting for a cold simulator to + # finish booting. Booting it up front with `bootstatus -b` moves that wait out of the + # measured build step and fails loudly when the runtime is missing from the stack. + create-ios-simulator: + steps: + - script@1: + title: Create and boot the iOS simulator + inputs: + - content: |- + set -euo pipefail + : "${CI_IOS_SIMULATOR_DEVICE:?CI_IOS_SIMULATOR_DEVICE is required}" + xcrun simctl delete all + simulator_uuid="$(xcrun simctl create CIDevice "$CI_IOS_SIMULATOR_DEVICE")" + xcrun simctl bootstatus "$simulator_uuid" -b + envman add --key CURRENT_SIMULATOR_UUID --value "$simulator_uuid" bootstrap-mint: inputs: - mintfile_dir: "" @@ -202,6 +248,80 @@ step_bundles: - paths: ~/.mint workflows: + # Runs on the cheap default Linux stack. Every macOS machine this pipeline boots is + # decided here, so a change that needs no iOS job costs one short Linux build. + ci-ios-plan: + steps: + - git-clone@8: {} + - script@1: + title: Decide which iOS jobs run + timeout: 900 + no_output_timeout: 450 + inputs: + - content: |- + set -euo pipefail + source e2e/scripts/bitrise_ci_helpers + e2e_log "Fetching pull request changed files" + changed_files="$(e2e_changed_files_file)" + ruby e2e/scripts/ios_ci_run_plan validate --changed-files-file "$changed_files" + e2e_log "Asserting the pipeline can run every selected job" + branch_config="$(e2e_branch_bitrise_config)" + ruby e2e/scripts/ios_ci_run_plan assert-pipeline-coverage --pipeline-config "$branch_config" --changed-files-file "$changed_files" + e2e_log "Publishing iOS CI job flags" + bitrise_env_file="$(e2e_deploy_dir)/ios-ci-env.txt" + ruby e2e/scripts/ios_ci_run_plan bitrise-env --changed-files-file "$changed_files" > "$bitrise_env_file" + cat "$bitrise_env_file" + while IFS='=' read -r key value; do + envman add --key "$key" --value "$value" + done < "$bitrise_env_file" + - share-pipeline-variable@1: + inputs: + - variables: |- + CI_IOS_HAS_JOBS + CI_IOS_SELECTED_JOBS + CI_IOS_SWIFT_PACKAGE_TESTS + + ci-ios-swift-package-tests: + meta: + bitrise.io: + # Keep in sync with dev.yml's xcode.version (26.2). Bitrise stable stacks + # only receive Xcode patch updates, so this pins Xcode 26.2 / iOS 26.2 + # (23C54) to match local development. + stack: &macos_stack osx-xcode-26.2.x + machine_type_id: g2.mac.4large + steps: + - git-clone@8: {} + - bundle::create-ios-simulator: {} + - script@1: + title: Run Swift package tests + timeout: 3600 + no_output_timeout: 1800 + inputs: + - content: |- + set -euo pipefail + cd platforms/swift + ./Scripts/xcode_run test ShopifyCheckoutKit-Package + + ci-ios-report: + steps: + - git-clone@8: {} + - script@1: + title: Report iOS CI results to GitHub + timeout: 900 + no_output_timeout: 450 + inputs: + - content: |- + set -euo pipefail + source e2e/scripts/bitrise_ci_helpers + e2e_export_github_token + : "${BITRISE_GIT_COMMIT:?BITRISE_GIT_COMMIT is required}" + # The report names the stages that failed from this roster. Bitrise documents no + # enum for its status field, so log the payload to confirm the literals it uses. + e2e_log "Pipeline stage roster" + printf '%s\n' "${BITRISEIO_FINISHED_WORKFLOWS:-}" + e2e_log "Reporting iOS CI results to GitHub" + ruby e2e/scripts/report_ios_ci_results + e2e-produce-browserstack-run-plan: steps: - git-clone@8: {} @@ -213,28 +333,11 @@ workflows: - content: |- set -euo pipefail source e2e/scripts/bitrise_ci_helpers - changed_files="$(e2e_deploy_dir)/changed-files.txt" e2e_log "Fetching pull request changed files" - # PR builds: the checkout is a shallow merge commit, so the GitHub API - # is the authoritative source for the PR's changed files. Non-PR runs - # (manual, scheduled, local) fall back to diffing the branch against its - # base with a three-dot range (changes since the merge-base, not tip-to-tip). - if [ -n "${BITRISE_PULL_REQUEST:-}" ]; then - ruby e2e/scripts/github_pr_changed_files > "$changed_files" - else - git diff --name-only "${BITRISE_GIT_BRANCH_DEST:-origin/main}...HEAD" > "$changed_files" - fi + changed_files="$(e2e_changed_files_file)" ruby e2e/scripts/e2e_matrix_to_browserstack_run_plan validate --changed-files-file "$changed_files" - # Guard against config-vs-matrix drift: Bitrise resolves the pipeline graph from - # the branch-head bitrise.yml (before the merge checkout), while the matrix here is - # generated from the merged tree. A stale branch can select a target whose build - # workflow does not exist in the graph, which fails cryptically later in - # e2e-execute-browserstack-run. Fail fast against the branch-head config instead. e2e_log "Asserting the pipeline can build every selected target" - branch_config="$(e2e_deploy_dir)/branch-bitrise.yml" - if [ -z "${BITRISE_PULL_REQUEST:-}" ] || [ -z "${BITRISE_GIT_COMMIT:-}" ] || ! git show "${BITRISE_GIT_COMMIT}:e2e/bitrise.yml" > "$branch_config" 2>/dev/null; then - cp e2e/bitrise.yml "$branch_config" - fi + branch_config="$(e2e_branch_bitrise_config)" ruby e2e/scripts/e2e_matrix_to_browserstack_run_plan assert-pipeline-coverage --pipeline-config "$branch_config" --changed-files-file "$changed_files" e2e_log "Producing BrowserStack run plan" ruby e2e/scripts/e2e_matrix_to_browserstack_run_plan expand --changed-files-file "$changed_files" > "$BITRISE_DEPLOY_DIR/browserstack-run-plan.json" @@ -269,10 +372,7 @@ workflows: e2e-build-react-native-ios: meta: bitrise.io: - # Keep in sync with dev.yml's xcode.version (26.2). Bitrise stable stacks - # only receive Xcode patch updates, so this pins Xcode 26.2 / iOS 26.2 - # (23C54) to match local development. - stack: &macos_stack osx-xcode-26.2.x + stack: *macos_stack machine_type_id: g2.mac.4large steps: - git-clone@8: {} @@ -434,11 +534,7 @@ workflows: - content: |- set -euo pipefail source e2e/scripts/bitrise_ci_helpers - e2e_log "Checking GitHub reporting configuration" - if [ -z "${GITHUB_TOKEN:-}" ]; then - : "${GIT_HTTP_PASSWORD:?GITHUB_TOKEN or Bitrise GitHub App GIT_HTTP_PASSWORD is required. Enable Project settings > Repository > Extend GitHub App permissions to builds.}" - export GITHUB_TOKEN="$GIT_HTTP_PASSWORD" - fi + e2e_export_github_token : "${BITRISE_GIT_COMMIT:?BITRISE_GIT_COMMIT is required}" results_root="${E2E_BROWSERSTACK_RESULTS_DIR:-$BITRISE_DEPLOY_DIR/e2e/results}" # The report names the stages that failed from this roster. Bitrise documents no diff --git a/e2e/config/ios_ci.yml b/e2e/config/ios_ci.yml new file mode 100644 index 000000000..de7a8495a --- /dev/null +++ b/e2e/config/ios_ci.yml @@ -0,0 +1,18 @@ +version: 1 +changed_file_filters: .ci/changed-file-filters.yml +# One entry per macOS CI job the Bitrise ci-ios pipeline runs. Each id maps to a +# ci-ios- workflow in e2e/bitrise.yml and to a CI_IOS_ run_if flag, and +# e2e/test/ios_ci_run_plan_test.rb fails when the two sides drift apart. +# +# These filters mirror the GitHub Actions `changes` job in .github/workflows/ci.yml. +# The iOS jobs deliberately subscribe to iosCiConfig rather than the broad e2e filter, +# so editing a Maestro flow does not boot a macOS machine. +jobs: + - id: swift-package-tests + changed_files_filters: + - swift + - protocolSwift + - protocolShared + - packageSwift + - ciFilters + - iosCiConfig diff --git a/e2e/lib/ios_ci_reporter.rb b/e2e/lib/ios_ci_reporter.rb new file mode 100644 index 000000000..dd869b26a --- /dev/null +++ b/e2e/lib/ios_ci_reporter.rb @@ -0,0 +1,125 @@ +# frozen_string_literal: true + +require_relative "../../scripts/lib/json_http_client" + +# Publishes the macOS CI pipeline outcome to GitHub as one check run. +# +# Bitrise's own commit status covers the whole pipeline, which is the wrong shape for a +# merge gate: a job the change does not need is skipped, and a skipped Bitrise workflow +# is indistinguishable from one the pipeline never reached. Only the gate's own selection +# separates the two, so the pipeline reports its own check instead. +class IOSCIReporter + CHECK_NAME = "Checkout Kit iOS" + PLAN_STAGE_NAME = "ci-ios-plan" + WORKFLOW_PREFIX = "ci-ios-" + + def initialize(job_ids:, selected_job_ids:, repository:, sha:, token:, stages:, pipeline_url: nil) + @job_ids = job_ids + @selected_job_ids = selected_job_ids + @repository = repository + @sha = sha + @token = token + @stages = stages + @pipeline_url = pipeline_url + end + + def publish! + client.post_json("/repos/#{@repository}/check-runs", check_run_payload) + end + + def check_run_payload + { + name: CHECK_NAME, + head_sha: @sha, + status: "completed", + conclusion: conclusion, + output: { + title: "#{CHECK_NAME} #{conclusion}", + summary: markdown_summary + } + } + end + + def conclusion + problem_stages.empty? ? "success" : "failure" + end + + def markdown_summary + lines = ["## #{CHECK_NAME}", ""] + lines.concat(plan_failure_lines) + lines.concat(@job_ids.empty? ? [] : job_table) + if @selected_job_ids.empty? + lines << "" + lines << "No iOS job ran for this change." + end + lines.concat(pipeline_link_lines) + lines.join("\n") + end + + private + + # ci-ios-report is still running while it writes this check, so its own stage always + # looks unexecuted. Reasoning only over expected stages leaves it out, along with every + # job the gate deliberately skipped. + def expected_stage_names + [PLAN_STAGE_NAME] + @selected_job_ids.map { |id| "#{WORKFLOW_PREFIX}#{id}" } + end + + def problem_stages + @problem_stages ||= (@stages.failed + @stages.not_executed) + .select { |stage| expected_stage_names.include?(stage.name) } + .uniq(&:name) + end + + def problem_stage_names + @problem_stage_names ||= problem_stages.map(&:name) + end + + def failed_stage_names + @failed_stage_names ||= @stages.failed.map(&:name) + end + + def plan_failure_lines + return [] unless problem_stage_names.include?(PLAN_STAGE_NAME) + + [ + "> [!CAUTION]", + "> `#{PLAN_STAGE_NAME}` #{failed_stage_names.include?(PLAN_STAGE_NAME) ? "failed" : "did not run"}, " \ + "so no job flag was published and every iOS job was skipped.", + "" + ] + end + + def job_table + lines = ["| Status | Job | Outcome |", "|---|---|---|"] + @job_ids.each do |id| + icon, outcome = job_status(id) + lines << "| #{icon} | `#{id}` | #{outcome} |" + end + lines + end + + def job_status(id) + return ["⏭️", "skipped — not needed for this change"] unless @selected_job_ids.include?(id) + + stage_name = "#{WORKFLOW_PREFIX}#{id}" + return ["✅", "passed"] unless problem_stage_names.include?(stage_name) + return ["❌", "failed"] if failed_stage_names.include?(stage_name) + + ["❌", "did not run"] + end + + def pipeline_link_lines + return [] if @pipeline_url.nil? || @pipeline_url.to_s.strip.empty? + + ["", "[Pipeline build](#{@pipeline_url})"] + end + + def client + @client ||= JsonHttpClient.new(host: "api.github.com", error_label: "GitHub", default_headers: {"Accept" => "application/vnd.github+json"}) do |request| + raise "GitHub token is required" unless @token + + request["Authorization"] = "Bearer #{@token}" + end + end +end diff --git a/e2e/lib/ios_ci_run_plan.rb b/e2e/lib/ios_ci_run_plan.rb new file mode 100644 index 000000000..a130c27d7 --- /dev/null +++ b/e2e/lib/ios_ci_run_plan.rb @@ -0,0 +1,193 @@ +# frozen_string_literal: true + +require "pathname" +require "yaml" +require_relative "../../scripts/lib/changed_file_filters" + +# Decides which macOS CI jobs a change needs, and publishes that decision to Bitrise as +# one environment flag per job. +# +# GitHub Actions does the same work with `dorny/paths-filter` plus a per-job `if:`. +# Bitrise has no equivalent, so a cheap Linux workflow evaluates the same shared filters +# from .ci/changed-file-filters.yml and shares the flags with the pipeline. Both gates +# therefore read one filter file, and neither can drift into selecting a different set. +class IOSCIRunPlan + # Drift happens in both directions: a stale branch predating a newly declared job, or a + # workflow deleted while the config still selects it. The hint has to name both remedies. + MISSING_WORKFLOW_HINT = "Rebase on main to pick up the latest workflows, " \ + "or update e2e/config/ios_ci.yml if you're removing this job intentionally." + + WORKFLOW_PREFIX = "ci-ios-" + + attr_reader :config_path, :changed_files + + def self.load(config_path, changed_files: nil) + new(config_path, YAML.safe_load_file(config_path, aliases: true), changed_files: changed_files) + end + + def initialize(config_path, config, changed_files: nil) + @config_path = config_path + @config = config || {} + @changed_files = changed_files&.map(&:to_s) + end + + def jobs + @config.fetch("jobs", []) || [] + end + + def job_ids + jobs.map { |job| job.fetch("id") } + end + + def selected_jobs + return jobs if changed_files.nil? + + jobs.select { |job| job_matches_changed_files?(job) } + end + + def selected_job_ids + selected_jobs.map { |job| job.fetch("id") } + end + + def flag_names + job_ids.map { |id| flag_name(id) } + end + + def workflow_names + job_ids.map { |id| workflow_name(id) } + end + + def bitrise_env + ensure_valid! + selected = selected_job_ids + env = { + "CI_IOS_HAS_JOBS" => (!selected.empty?).to_s, + "CI_IOS_SELECTED_JOBS" => selected.join(",") + } + job_ids.each { |id| env[flag_name(id)] = selected.include?(id).to_s } + env + end + + def missing_workflows(available_workflow_names) + selected_job_ids.map { |id| workflow_name(id) } - available_workflow_names + end + + def missing_workflow_errors(available_workflow_names) + missing_workflows(available_workflow_names).map do |workflow| + job = workflow.delete_prefix(WORKFLOW_PREFIX) + "Run plan selected '#{job}' but this branch's e2e/bitrise.yml has no '#{workflow}' workflow." + end + end + + def validation_errors + errors = [] + errors << "version must be 1" unless @config.fetch("version", nil) == 1 + errors << "jobs must be a non-empty array" unless jobs.is_a?(Array) && !jobs.empty? + validate_changed_file_filters(errors) + validate_jobs(errors) + errors + end + + def ensure_valid! + errors = validation_errors + raise "iOS CI run plan is invalid:\n#{errors.join("\n")}" unless errors.empty? + end + + private + + def flag_name(job_id) + "CI_IOS_#{job_id.upcase.gsub(/[^A-Z0-9]+/, "_")}" + end + + def workflow_name(job_id) + "#{WORKFLOW_PREFIX}#{job_id}" + end + + def job_matches_changed_files?(job) + filter_names = job_filter_names(job) + return true if filter_names.empty? + + changed_file_filters.match?(filter_names, changed_files) + end + + def job_filter_names(job) + job.fetch("changed_files_filters", nil) || [] + end + + def changed_file_filters + @changed_file_filters ||= ChangedFileFilters.load(changed_file_filters_path) + end + + def changed_file_filters_path + path = @config.fetch("changed_file_filters", nil) + return nil if path.to_s.empty? + return path if Pathname.new(path).absolute? + + File.expand_path(path, repo_root) + end + + def repo_root + File.expand_path("..", e2e_root) + end + + def e2e_root + File.expand_path("..", File.dirname(config_path)) + end + + def validate_changed_file_filters(errors) + if changed_file_filters_path.nil? + errors << "changed_file_filters must point to a shared filter file" + return + end + + unless File.exist?(changed_file_filters_path) + errors << "changed_file_filters file does not exist: #{changed_file_filters_path}" + return + end + + errors.concat(changed_file_filters.validation_errors) + rescue Psych::Exception => error + errors << "changed_file_filters could not be parsed: #{error.message}" + end + + def validate_jobs(errors) + return unless jobs.is_a?(Array) + + ids = jobs.map { |job| job.fetch("id", nil) } + errors << "job ids must be unique" unless ids.compact.uniq.length == ids.compact.length + jobs.each { |job| validate_job(errors, job) } + end + + def validate_job(errors, job) + id = job.fetch("id", "") + errors << "job missing id" if id.to_s.empty? || id == "" + + filter_names = job.fetch("changed_files_filters", nil) + if filter_names.nil? + errors << "job #{id} missing changed_files_filters" + return + end + + unless filter_names.is_a?(Array) + errors << "job #{id} changed_files_filters must be an array" + return + end + + return unless changed_file_filters_available? + + filter_names.each do |filter_name| + next if changed_file_filters.filters.key?(filter_name) + + errors << "job #{id} references unknown changed file filter #{filter_name}" + end + end + + def changed_file_filters_available? + return false if changed_file_filters_path.nil? + + changed_file_filters + true + rescue Errno::ENOENT, Psych::Exception + false + end +end diff --git a/e2e/scripts/bitrise_ci_helpers b/e2e/scripts/bitrise_ci_helpers index a05c14be8..7fa896303 100755 --- a/e2e/scripts/bitrise_ci_helpers +++ b/e2e/scripts/bitrise_ci_helpers @@ -17,6 +17,46 @@ e2e_deploy_dir() { printf '%s\n' "$dir" } +# Writes the changed file list and prints its path. Two gates read it — the E2E matrix +# and the iOS CI run plan — and a second copy of this derivation would let them select +# different work from the same commit. +# +# PR builds: the checkout is a shallow merge commit, so the GitHub API is the +# authoritative source for the PR's changed files. Non-PR runs (manual, scheduled, +# local) fall back to diffing the branch against its base with a three-dot range +# (changes since the merge-base, not tip-to-tip). +e2e_changed_files_file() { + local changed_files + changed_files="$(e2e_deploy_dir)/changed-files.txt" + if [ -n "${BITRISE_PULL_REQUEST:-}" ]; then + ruby e2e/scripts/github_pr_changed_files > "$changed_files" + else + git diff --name-only "${BITRISE_GIT_BRANCH_DEST:-origin/main}...HEAD" > "$changed_files" + fi + printf '%s\n' "$changed_files" +} + +# Writes the branch-head bitrise.yml and prints its path. Bitrise resolves the pipeline +# graph from the branch head (before the merge checkout), while the run plan is generated +# from the merged tree. A stale branch can select a job whose workflow does not exist in +# the graph, which fails cryptically later. Comparing against this copy fails fast. +e2e_branch_bitrise_config() { + local branch_config + branch_config="$(e2e_deploy_dir)/branch-bitrise.yml" + if [ -z "${BITRISE_PULL_REQUEST:-}" ] || [ -z "${BITRISE_GIT_COMMIT:-}" ] || ! git show "${BITRISE_GIT_COMMIT}:e2e/bitrise.yml" > "$branch_config" 2>/dev/null; then + cp e2e/bitrise.yml "$branch_config" + fi + printf '%s\n' "$branch_config" +} + +e2e_export_github_token() { + e2e_log "Checking GitHub reporting configuration" + if [ -z "${GITHUB_TOKEN:-}" ]; then + : "${GIT_HTTP_PASSWORD:?GITHUB_TOKEN or Bitrise GitHub App GIT_HTTP_PASSWORD is required. Enable Project settings > Repository > Extend GitHub App permissions to builds.}" + export GITHUB_TOKEN="$GIT_HTTP_PASSWORD" + fi +} + e2e_configure_storefront() { e2e_log "Checking storefront configuration secrets" : "${STOREFRONT_DOMAIN:?STOREFRONT_DOMAIN is required. Check https://app.bitrise.io/app/f51f9054-053e-40f1-81e9-ae727567ae76/workflow_editor#!/secrets and enable Expose for pull requests.}" diff --git a/e2e/scripts/ios_ci_run_plan b/e2e/scripts/ios_ci_run_plan new file mode 100755 index 000000000..4b05aa0a8 --- /dev/null +++ b/e2e/scripts/ios_ci_run_plan @@ -0,0 +1,55 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +require "optparse" +require "yaml" +require_relative "../lib/ios_ci_run_plan" +require_relative "../../scripts/lib/cli_output" + +options = { + config: File.expand_path("../config/ios_ci.yml", __dir__) +} + +parser = OptionParser.new do |opts| + opts.banner = "Usage: e2e/scripts/ios_ci_run_plan validate|bitrise-env|selected-jobs|assert-pipeline-coverage [options]" + opts.on("--config PATH") { |path| options[:config] = path } + opts.on("--changed-files-file PATH") { |path| options[:changed_files_file] = path } + opts.on("--changed-file PATH") { |path| (options[:changed_files] ||= []) << path } + opts.on("--pipeline-config PATH") { |path| options[:pipeline_config] = path } +end + +command = ARGV.shift +parser.parse!(ARGV) + +unless ["validate", "bitrise-env", "selected-jobs", "assert-pipeline-coverage"].include?(command) + warn parser + CliOutput.die("Unknown command #{command.inspect}") +end + +changed_files = [] +changed_files.concat(File.readlines(options.fetch(:changed_files_file), chomp: true)) if options.key?(:changed_files_file) +changed_files.concat(options.fetch(:changed_files, [])) +changed_files = nil unless options.key?(:changed_files_file) || options.key?(:changed_files) + +run_plan = IOSCIRunPlan.load(options.fetch(:config), changed_files: changed_files) + +errors = run_plan.validation_errors +CliOutput.die(errors.join("\n")) unless errors.empty? + +case command +when "validate" + puts "iOS CI run plan is valid" +when "bitrise-env" + run_plan.bitrise_env.each { |key, value| puts "#{key}=#{value}" } +when "selected-jobs" + puts run_plan.selected_job_ids.join(",") +when "assert-pipeline-coverage" + pipeline_config_path = options.fetch(:pipeline_config) { CliOutput.die("--pipeline-config PATH is required") } + pipeline_config = YAML.safe_load_file(pipeline_config_path, aliases: true) || {} + available_workflows = (pipeline_config.dig("pipelines", "ci-ios", "workflows") || {}).keys + coverage_errors = run_plan.missing_workflow_errors(available_workflows) + unless coverage_errors.empty? + CliOutput.die(coverage_errors.join("\n"), hint: IOSCIRunPlan::MISSING_WORKFLOW_HINT) + end + puts "ci-ios pipeline covers the run plan" +end diff --git a/e2e/scripts/report_ios_ci_results b/e2e/scripts/report_ios_ci_results new file mode 100755 index 000000000..113f0bc11 --- /dev/null +++ b/e2e/scripts/report_ios_ci_results @@ -0,0 +1,61 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +require "json" +require "optparse" +require_relative "../lib/bitrise_pipeline_stages" +require_relative "../lib/ios_ci_reporter" +require_relative "../lib/ios_ci_run_plan" + +options = { + config: File.expand_path("../config/ios_ci.yml", __dir__), + repository: ENV["GITHUB_REPOSITORY"] || "Shopify/checkout-kit", + sha: ENV["BITRISE_GIT_COMMIT"], + pr_number: ENV["BITRISE_PULL_REQUEST"], + token: ENV["GITHUB_TOKEN"], + selected_jobs: ENV["CI_IOS_SELECTED_JOBS"], + finished_workflows: ENV["BITRISEIO_FINISHED_WORKFLOWS"], + pipeline_url: ENV["BITRISEIO_PIPELINE_BUILD_URL"] +} + +OptionParser.new do |opts| + opts.on("--config PATH") { |path| options[:config] = path } + opts.on("--repository REPOSITORY") { |repository| options[:repository] = repository } + opts.on("--sha SHA") { |sha| options[:sha] = sha } + opts.on("--pr NUMBER") { |number| options[:pr_number] = number } + opts.on("--selected-jobs LIST") { |list| options[:selected_jobs] = list } + opts.on("--finished-workflows JSON") { |json| options[:finished_workflows] = json } + opts.on("--pipeline-url URL") { |url| options[:pipeline_url] = url } +end.parse! + +pr_number = options[:pr_number].to_s.strip +if pr_number.empty? || pr_number == "false" || pr_number == "0" + warn "No pull request associated with this build; skipping GitHub iOS CI report" + exit 0 +end + +%i[sha token].each do |key| + next unless options[key].nil? || options[key].to_s.strip.empty? + + abort "#{key} is required" +end + +run_plan = IOSCIRunPlan.load(options.fetch(:config)) +tophat_manifest = JSON.parse(File.read(File.expand_path("../../scripts/tophat/targets.json", __dir__))) + +stages = BitrisePipelineStages.from_json( + options[:finished_workflows], + app_slug: tophat_manifest.fetch("app_slug") +) + +reporter = IOSCIReporter.new( + job_ids: run_plan.job_ids, + selected_job_ids: options[:selected_jobs].to_s.split(",").map(&:strip).reject(&:empty?), + repository: options.fetch(:repository), + sha: options.fetch(:sha), + token: options.fetch(:token), + stages: stages, + pipeline_url: options[:pipeline_url] +) +reporter.publish! +puts reporter.markdown_summary diff --git a/e2e/test/ios_ci_reporter_test.rb b/e2e/test/ios_ci_reporter_test.rb new file mode 100644 index 000000000..d106e634f --- /dev/null +++ b/e2e/test/ios_ci_reporter_test.rb @@ -0,0 +1,124 @@ +# frozen_string_literal: true + +require "json" +require "minitest/autorun" +require_relative "../lib/bitrise_pipeline_stages" +require_relative "../lib/ios_ci_reporter" + +class IOSCIReporterTest < Minitest::Test + def workflow(name, status: "succeeded", external_id: "build-slug") + {"name" => name, "status" => status, "external_id" => external_id} + end + + def stage_roster(*workflows) + BitrisePipelineStages.from_json(JSON.generate(workflows), app_slug: "app-slug") + end + + def reporter(selected_job_ids:, stages:, job_ids: nil) + IOSCIReporter.new( + job_ids: job_ids || selected_job_ids, + selected_job_ids: selected_job_ids, + repository: "Shopify/checkout-kit", + sha: "abc123", + token: "token", + stages: stages, + pipeline_url: "https://app.bitrise.io/build/pipeline" + ) + end + + def test_the_check_name_is_the_literal_the_ruleset_pins + assert_equal "Checkout Kit iOS", IOSCIReporter::CHECK_NAME + assert_equal "Checkout Kit iOS", reporter(selected_job_ids: [], stages: stage_roster).check_run_payload.fetch(:name) + end + + def test_every_selected_job_succeeding_passes + stages = stage_roster( + workflow("ci-ios-plan"), + workflow("ci-ios-swift-package-tests") + ) + + assert_equal "success", reporter(selected_job_ids: ["swift-package-tests"], stages: stages).conclusion + end + + def test_a_selected_job_that_failed_fails + stages = stage_roster( + workflow("ci-ios-plan"), + workflow("ci-ios-swift-package-tests", status: "failed") + ) + + assert_equal "failure", reporter(selected_job_ids: ["swift-package-tests"], stages: stages).conclusion + end + + # A skipped workflow carries a blank external_id, which is indistinguishable from a + # stage the pipeline never reached. Only the gate's own selection tells them apart. + def test_a_job_the_gate_did_not_select_is_ignored + stages = stage_roster( + workflow("ci-ios-plan"), + workflow("ci-ios-swift-package-tests"), + workflow("ci-ios-swift-samples", status: "", external_id: "") + ) + report = reporter( + job_ids: ["swift-package-tests", "swift-samples"], + selected_job_ids: ["swift-package-tests"], + stages: stages + ) + + assert_equal "success", report.conclusion + assert_includes report.markdown_summary, "`swift-samples`" + assert_includes report.markdown_summary, "⏭️" + end + + def test_a_selected_job_that_never_ran_fails + stages = stage_roster( + workflow("ci-ios-plan"), + workflow("ci-ios-swift-package-tests", status: "", external_id: "") + ) + report = reporter(selected_job_ids: ["swift-package-tests"], stages: stages) + + assert_equal "failure", report.conclusion + assert_includes report.markdown_summary, "did not run" + end + + # The gate itself failing means no flag was ever published, so every job workflow is + # skipped. Reasoning only over selected jobs would then report a false green. + def test_a_failed_plan_stage_fails_even_though_no_job_was_selected + stages = stage_roster(workflow("ci-ios-plan", status: "failed")) + report = reporter(selected_job_ids: [], stages: stages) + + assert_equal "failure", report.conclusion + assert_includes report.markdown_summary, "ci-ios-plan" + end + + def test_no_selected_job_with_a_green_plan_passes + report = reporter(selected_job_ids: [], stages: stage_roster(workflow("ci-ios-plan"))) + + assert_equal "success", report.conclusion + assert_includes report.markdown_summary, "No iOS job ran for this change" + end + + # ci-ios-report is still running while it writes this check, so its own stage always + # looks unexecuted. It must never count itself as a missing stage. + def test_the_report_stage_never_counts_against_itself + stages = stage_roster( + workflow("ci-ios-plan"), + workflow("ci-ios-swift-package-tests"), + workflow("ci-ios-report", status: "", external_id: "") + ) + + assert_equal "success", reporter(selected_job_ids: ["swift-package-tests"], stages: stages).conclusion + end + + def test_the_summary_links_the_pipeline_build + report = reporter(selected_job_ids: [], stages: stage_roster(workflow("ci-ios-plan"))) + + assert_includes report.markdown_summary, "https://app.bitrise.io/build/pipeline" + end + + def test_the_check_run_payload_reports_a_completed_run_against_the_head_sha + payload = reporter(selected_job_ids: [], stages: stage_roster(workflow("ci-ios-plan"))).check_run_payload + + assert_equal "abc123", payload.fetch(:head_sha) + assert_equal "completed", payload.fetch(:status) + assert_equal "success", payload.fetch(:conclusion) + end +end diff --git a/e2e/test/ios_ci_run_plan_test.rb b/e2e/test/ios_ci_run_plan_test.rb new file mode 100644 index 000000000..e32686bc0 --- /dev/null +++ b/e2e/test/ios_ci_run_plan_test.rb @@ -0,0 +1,240 @@ +# frozen_string_literal: true + +require "minitest/autorun" +require "yaml" +require_relative "../lib/ios_ci_run_plan" + +class IOSCIRunPlanTest < Minitest::Test + CONFIG_PATH = File.expand_path("../config/ios_ci.yml", __dir__) + PIPELINE_PATH = File.expand_path("../bitrise.yml", __dir__) + + SWIFT_FILTERS = ["swift", "protocolSwift", "protocolShared", "packageSwift", "ciFilters", "iosCiConfig"].freeze + REACT_NATIVE_FILTERS = ["reactNative", "protocolTypescript", "protocolShared", "packageSwift", "ciFilters", "iosCiConfig"].freeze + + # The real config carries only the jobs already ported to Bitrise, so the four-job + # selection rules are exercised against a fixture that names all four macOS jobs. + def four_job_config + { + "version" => 1, + "changed_file_filters" => ".ci/changed-file-filters.yml", + "jobs" => [ + {"id" => "swift-package-tests", "changed_files_filters" => SWIFT_FILTERS}, + {"id" => "swift-samples", "changed_files_filters" => SWIFT_FILTERS}, + {"id" => "react-native-build-ios", "changed_files_filters" => REACT_NATIVE_FILTERS}, + {"id" => "react-native-test-ios", "changed_files_filters" => REACT_NATIVE_FILTERS} + ] + } + end + + def plan(changed_files: nil, config: four_job_config) + IOSCIRunPlan.new(CONFIG_PATH, config, changed_files: changed_files) + end + + def selected_ids(changed_files) + plan(changed_files: changed_files).selected_job_ids + end + + def pipeline_config + @pipeline_config ||= YAML.safe_load_file(PIPELINE_PATH, aliases: true) + end + + def ci_ios_workflows + pipeline_config.dig("pipelines", "ci-ios", "workflows") || {} + end + + def test_nil_changed_files_selects_every_job + assert_equal( + ["swift-package-tests", "swift-samples", "react-native-build-ios", "react-native-test-ios"], + selected_ids(nil) + ) + end + + def test_empty_changed_files_selects_no_job + assert_empty selected_ids([]) + end + + def test_swift_change_selects_only_the_swift_jobs + assert_equal( + ["swift-package-tests", "swift-samples"], + selected_ids(["platforms/swift/Sources/ShopifyCheckoutKit/Foo.swift"]) + ) + end + + def test_react_native_change_selects_only_the_react_native_jobs + assert_equal( + ["react-native-build-ios", "react-native-test-ios"], + selected_ids(["platforms/react-native/src/index.ts"]) + ) + end + + # ci.yml lists Package.swift under both the swift and reactNativeIos infra filters, + # so a manifest change has to keep selecting all four macOS jobs after the port. + def test_swift_package_manifest_change_selects_every_job + assert_equal( + ["swift-package-tests", "swift-samples", "react-native-build-ios", "react-native-test-ios"], + selected_ids(["Package.swift"]) + ) + end + + def test_docs_only_change_selects_no_job + assert_empty selected_ids(["platforms/swift/docs/usage.md"]) + end + + # An e2e/** edit that only touches Maestro flows must not boot four macOS machines, + # so the iOS jobs subscribe to the narrower iosCiConfig filter instead. + def test_maestro_flow_change_selects_no_job + assert_empty selected_ids(["e2e/tests/checkout/launch.yaml"]) + end + + def test_pipeline_config_change_selects_every_job + assert_equal( + ["swift-package-tests", "swift-samples", "react-native-build-ios", "react-native-test-ios"], + selected_ids(["e2e/bitrise.yml"]) + ) + end + + def test_flag_names_derive_from_job_ids + assert_equal( + [ + "CI_IOS_SWIFT_PACKAGE_TESTS", + "CI_IOS_SWIFT_SAMPLES", + "CI_IOS_REACT_NATIVE_BUILD_IOS", + "CI_IOS_REACT_NATIVE_TEST_IOS" + ], + plan.flag_names + ) + end + + def test_bitrise_env_publishes_one_flag_per_job + env = plan(changed_files: ["platforms/swift/Sources/ShopifyCheckoutKit/Foo.swift"]).bitrise_env + + assert_equal "true", env.fetch("CI_IOS_HAS_JOBS") + assert_equal "swift-package-tests,swift-samples", env.fetch("CI_IOS_SELECTED_JOBS") + assert_equal "true", env.fetch("CI_IOS_SWIFT_PACKAGE_TESTS") + assert_equal "true", env.fetch("CI_IOS_SWIFT_SAMPLES") + assert_equal "false", env.fetch("CI_IOS_REACT_NATIVE_BUILD_IOS") + assert_equal "false", env.fetch("CI_IOS_REACT_NATIVE_TEST_IOS") + end + + def test_bitrise_env_reports_no_jobs_on_a_docs_only_change + env = plan(changed_files: ["platforms/swift/docs/usage.md"]).bitrise_env + + assert_equal "false", env.fetch("CI_IOS_HAS_JOBS") + assert_equal "", env.fetch("CI_IOS_SELECTED_JOBS") + assert_equal "false", env.fetch("CI_IOS_SWIFT_PACKAGE_TESTS") + end + + def test_missing_workflows_names_a_selected_job_absent_from_the_pipeline + missing = plan(changed_files: ["platforms/swift/Sources/ShopifyCheckoutKit/Foo.swift"]) + .missing_workflows(["ci-ios-swift-package-tests"]) + + assert_equal ["ci-ios-swift-samples"], missing + end + + def test_missing_workflow_errors_name_the_missing_job + messages = plan(changed_files: ["platforms/swift/Sources/ShopifyCheckoutKit/Foo.swift"]) + .missing_workflow_errors(["ci-ios-swift-package-tests"]) + + assert_equal 1, messages.length + assert_includes messages.first, "no 'ci-ios-swift-samples' workflow." + end + + def test_missing_workflow_errors_empty_when_nothing_is_selected + assert_empty plan(changed_files: ["platforms/swift/docs/usage.md"]).missing_workflow_errors([]) + end + + def test_missing_workflow_hint_covers_both_directions_of_drift + hint = IOSCIRunPlan::MISSING_WORKFLOW_HINT + + assert_includes hint, "Rebase on main" + assert_includes hint, "e2e/config/ios_ci.yml" + end + + def test_validation_errors_flags_a_wrong_version + assert_includes plan(config: four_job_config.merge("version" => 2)).validation_errors, "version must be 1" + end + + def test_validation_errors_flags_an_unknown_filter_name + config = four_job_config + config.fetch("jobs").first["changed_files_filters"] = ["swift", "doesNotExist"] + + assert_includes( + plan(config: config).validation_errors, + "job swift-package-tests references unknown changed file filter doesNotExist" + ) + end + + def test_validation_errors_flags_missing_filters + config = four_job_config + config.fetch("jobs").first.delete("changed_files_filters") + + assert_includes plan(config: config).validation_errors, "job swift-package-tests missing changed_files_filters" + end + + def test_validation_errors_flags_non_array_filters + config = four_job_config + config.fetch("jobs").first["changed_files_filters"] = "swift" + + assert_includes plan(config: config).validation_errors, "job swift-package-tests changed_files_filters must be an array" + end + + def test_validation_errors_flags_duplicate_job_ids + config = four_job_config + config.fetch("jobs").last["id"] = "swift-package-tests" + + assert_includes plan(config: config).validation_errors, "job ids must be unique" + end + + def test_validation_errors_flags_a_missing_changed_file_filters_path + config = four_job_config + config.delete("changed_file_filters") + + assert_includes plan(config: config).validation_errors, "changed_file_filters must point to a shared filter file" + end + + def test_bitrise_env_raises_on_invalid_config + assert_raises(RuntimeError) { plan(config: {"version" => 2}).bitrise_env } + end + + def test_load_reads_configuration_from_disk + loaded = IOSCIRunPlan.load(CONFIG_PATH, changed_files: ["platforms/swift/Sources/ShopifyCheckoutKit/Foo.swift"]) + + assert_includes loaded.selected_job_ids, "swift-package-tests" + end + + def test_real_config_has_no_validation_errors + assert_empty IOSCIRunPlan.load(CONFIG_PATH).validation_errors + end + + # Both halves of the gate have to move together: a job added to ios_ci.yml without a + # run_if block runs unconditionally, and a run_if block reading a flag no job emits + # never becomes true, so its workflow is skipped forever. + def test_every_declared_job_owns_exactly_one_pipeline_run_if_flag + consumed = ci_ios_workflows.values + .filter_map { |workflow| workflow.dig("run_if", "expression") } + .flat_map { |expression| expression.scan(/CI_IOS_[A-Z0-9_]+/) } + .uniq + + assert_equal IOSCIRunPlan.load(CONFIG_PATH).flag_names.sort, consumed.sort + end + + def test_every_declared_job_has_a_pipeline_workflow + assert_empty IOSCIRunPlan.load(CONFIG_PATH).missing_workflows(ci_ios_workflows.keys) + end + + # A flag that ci-ios-plan computes but never shares stays unset in every later + # workflow, so its run_if silently evaluates false. + def test_the_plan_workflow_shares_every_flag_it_publishes + steps = pipeline_config.dig("workflows", "ci-ios-plan", "steps") || [] + shared = steps + .filter_map { |step| step["share-pipeline-variable@1"] } + .flat_map { |step| step.fetch("inputs", []).filter_map { |input| input["variables"] } } + .flat_map { |variables| variables.split("\n") } + .map(&:strip) + .reject(&:empty?) + + expected = IOSCIRunPlan.load(CONFIG_PATH).flag_names + ["CI_IOS_HAS_JOBS", "CI_IOS_SELECTED_JOBS"] + + assert_equal expected.sort, shared.sort + end +end