Skip to content

Add fly:bats rake task for running BATs from a local branch #2753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
colins wants to merge 1 commit into
base: main
Choose a base branch
from
+333 −0
Open

Add fly:bats rake task for running BATs from a local branch #2753

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions ci/fly-bats.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
---
# Standalone pipeline for running the BOSH Acceptance Test suite (BATs)
# against a specific branch.
#
# Intended to be driven by `bundle exec rake fly:bats` from src/, which
# pushes the current branch and sets this pipeline automatically.
#
# Manual usage:
# fly -t bosh set-pipeline -p bats-local \
# -c ci/fly-bats.yml \
# --var bosh_repo=https://github.com/cloudfoundry/bosh.git \
# --var bosh_branch=my-feature-branch \
# --var env_name=bats-local \
# --var stemcell_name=bosh-google-kvm-ubuntu-noble \
# --var deploy_args="-o bosh-deployment/external-ip-not-recommended.yml" \
# --var bat_rspec_flags=""
# fly -t bosh unpause-pipeline -p bats-local
# fly -t bosh trigger-job -j bats-local/bats -w

resources:
- name: bosh
type: git
source:
uri: ((bosh_repo))
branch: ((bosh_branch))

# bosh-ci is the same repo as bosh but filtered to ci/ paths so that
# scripts under ci/bats/ are available at bosh-ci/ in the task workspace.
- name: bosh-ci
type: git
source:
uri: ((bosh_repo))
branch: ((bosh_branch))
paths: [ci]

- name: bosh-cli
type: github-release
source:
owner: cloudfoundry
repository: bosh-cli
access_token: ((github_public_repo_token))

- name: stemcell
type: bosh-io-stemcell
source:
name: bosh-google-kvm-ubuntu-noble
Comment on lines +43 to +46

@coderabbitai coderabbitai Bot Jun 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Wire stemcell_name into the resource definition.

Line 46 is still pinned to bosh-google-kvm-ubuntu-noble, so STEMCELL_NAME only changes the task param later in the job, not the stemcell fetched by this pipeline. That means the advertised override is not actually end-to-end.

Suggested fix 
   - name: stemcell
     type: bosh-io-stemcell
     source:
-      name: bosh-google-kvm-ubuntu-noble
+      name: ((stemcell_name))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: stemcell
type: bosh-io-stemcell
source:
name: bosh-google-kvm-ubuntu-noble
- name: stemcell
type: bosh-io-stemcell
source:
name: ((stemcell_name))
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci/fly-bats.yml` around lines 43 - 46, The stemcell resource is still
hardcoded to a fixed name, so the pipeline override is not applied end-to-end.
Update the stemcell resource definition in the fly config to use the existing
stemcell_name variable instead of the literal bosh-google-kvm-ubuntu-noble
value, and keep the task/job param wiring consistent with the same variable so
the fetched stemcell matches the override.


- name: bats
type: git
source:
uri: https://github.com/cloudfoundry/bosh-acceptance-tests.git
branch: master

- name: bosh-deployment
type: git
source:
uri: https://github.com/cloudfoundry/bosh-deployment
branch: master

- name: integration-image
type: registry-image
source:
repository: ghcr.io/cloudfoundry/bosh/integration
tag: main
username: ((github_read_write_packages.username))
password: ((github_read_write_packages.password))

jobs:
- name: bats
serial: true
plan:
- do:
- in_parallel:
- get: bosh
- get: bosh-ci
- get: bosh-cli
params:
globs: [bosh-cli-*-linux-amd64]
- get: stemcell
- get: bats
- get: bosh-deployment
- get: integration-image

- task: make-candidate
image: integration-image
file: bosh-ci/ci/tasks/make-candidate.yml

- task: compile-bosh-release
file: bosh-deployment/ci/tasks/shared/bosh-agent-compile.yml

- task: run-bats
image: integration-image
input_mapping:
bosh-release: compiled-release
config:
platform: linux
inputs:
- name: bosh
- name: bosh-ci
- name: bosh-cli
- name: bosh-deployment
- name: stemcell
- name: bats
- name: bosh-release
caches:
- path: cache-dot-bosh-dir
params:
BAT_INFRASTRUCTURE: gcp
GCP_JSON_KEY: ((gcp_json_key))
GCP_PROJECT_ID: ((gcp_project_id))
STEMCELL_NAME: ((stemcell_name))
ENV_NAME: ((env_name))
DEPLOY_ARGS: ((deploy_args))
BAT_RSPEC_FLAGS: ((bat_rspec_flags))
run:
path: bosh/ci/tasks/run-bats-pipeline.sh
168 changes: 168 additions & 0 deletions ci/tasks/run-bats-pipeline.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
#!/usr/bin/env bash
# run-bats-pipeline.sh
#
# Chains together the full BATs pipeline in a single Concourse task:
# terraform apply → deploy-director → prepare-bats-config → run-bats
# terraform destroy ← destroy-director ← (EXIT trap, always runs)
#
# Required env vars (set via run-bats-pipeline.yml params):
# GCP_JSON_KEY – GCP service account JSON (resolved from Concourse creds)
# GCP_PROJECT_ID – GCP project ID
# ENV_NAME – Unique terraform env name, e.g. "bats-local"
# BAT_INFRASTRUCTURE – "gcp"
# STEMCELL_NAME – Stemcell name for bats-config.yml
# DEPLOY_ARGS – Extra ops-file args for bosh create-env
# BAT_RSPEC_FLAGS – Extra flags appended to the BAT run (optional)

set -eu

@coderabbitai coderabbitai Bot Jun 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the target script around the cited lines.
git ls-files ci/tasks/run-bats-pipeline.sh
wc -l ci/tasks/run-bats-pipeline.sh
cat -n ci/tasks/run-bats-pipeline.sh | sed -n '1,140p'

Repository: cloudfoundry/bosh

Length of output: 7330

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Check whether the script already uses pipefail or guards the pipeline in another way.
rg -n "pipefail|terraform output|jq|environment/metadata|metadata" ci/tasks/run-bats-pipeline.sh

Repository: cloudfoundry/bosh

Length of output: 658

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate any downstream consumers that assume the metadata file exists and is valid.
rg -n "environment/metadata|metadata" ci tasks .github . | sed -n '1,200p'

Repository: cloudfoundry/bosh

Length of output: 23814

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Probe shell semantics relevant to the claim in a read-only way.
python3 - <<'PY'
import subprocess, textwrap, os, tempfile, pathlib, json, sys

scripts = {
    "no_pipefail": "set -eu\nfalse | cat > /tmp/out1\nprintf done\n",
    "with_pipefail": "set -euo pipefail\nfalse | cat > /tmp/out2\nprintf done\n",
}
for name, body in scripts.items():
    p = subprocess.run(["bash", "-lc", body], capture_output=True, text=True)
    print(name, "rc=", p.returncode, "stdout=", p.stdout.strip(), "stderr=", p.stderr.strip())
PY

Repository: cloudfoundry/bosh

Length of output: 429

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Check jq pipeline semantics and downstream consumers of the metadata file.
python3 - <<'PY'
import subprocess, textwrap

tests = {
    "jq_with_entries_on_eof": "printf '' | jq 'with_entries(.value = .value.value)'; echo rc:$?",
    "jq_raw_on_eof": "printf '' | jq -r '.director_public_ip // empty'; echo rc:$?",
}
for name, cmd in tests.items():
    p = subprocess.run(["bash", "-lc", cmd], capture_output=True, text=True)
    print(f"--- {name} ---")
    print("returncode:", p.returncode)
    print("stdout:", repr(p.stdout))
    print("stderr:", repr(p.stderr))
PY

# Inspect the consumer script that reads environment/metadata.
wc -l ci/bats/iaas/gcp/director-vars ci/bats/iaas/gcp/prepare-bats-config.sh ci/bats/tasks/deploy-director.sh
cat -n ci/bats/iaas/gcp/director-vars | sed -n '1,160p'
echo '---'
cat -n ci/bats/iaas/gcp/prepare-bats-config.sh | sed -n '1,120p'
echo '---'
cat -n ci/bats/tasks/deploy-director.sh | sed -n '1,120p'

Repository: cloudfoundry/bosh

Length of output: 8096

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Check jq pipeline semantics and downstream consumers of the metadata file.
python3 - <<'PY'
import subprocess, textwrap

tests = {
    "jq_with_entries_on_eof": "printf '' | jq 'with_entries(.value = .value.value)'; echo rc:$?",
    "jq_raw_on_eof": "printf '' | jq -r '.director_public_ip // empty'; echo rc:$?",
}
for name, cmd in tests.items():
    p = subprocess.run(["bash", "-lc", cmd], capture_output=True, text=True)
    print(f"--- {name} ---")
    print("returncode:", p.returncode)
    print("stdout:", repr(p.stdout))
    print("stderr:", repr(p.stderr))
PY

# Inspect the consumer script that reads environment/metadata.
wc -l ci/bats/iaas/gcp/director-vars ci/bats/iaas/gcp/prepare-bats-config.sh ci/bats/tasks/deploy-director.sh
cat -n ci/bats/iaas/gcp/director-vars | sed -n '1,160p'
echo '---'
cat -n ci/bats/iaas/gcp/prepare-bats-config.sh | sed -n '1,120p'
echo '---'
cat -n ci/bats/tasks/deploy-director.sh | sed -n '1,120p'

Repository: cloudfoundry/bosh

Length of output: 8096

Add pipefail to the script. terraform output -json | jq ... > environment/metadata can still create an empty metadata file if terraform output fails, because jq exits 0 on EOF. set -euo pipefail closes that gap.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci/tasks/run-bats-pipeline.sh` at line 17, The shell script currently uses
set -eu without pipefail, so failures in piped commands can be masked. Update
the script header to use set -euo pipefail in the run-bats-pipeline.sh flow, so
the terraform output -json | jq ... pipeline fails correctly instead of allowing
an empty environment/metadata file. Use the existing pipeline in this script as
the place to apply the change.


ROOT_DIR="$PWD"
TERRAFORM_DIR="${ROOT_DIR}/bosh-ci/ci/bats/iaas/gcp/terraform"
TERRAFORM_VERSION="1.9.8"

# ── Shared working directories ───────────────────────────────────────────────
mkdir -p director-state bats-config environment
# cache-dot-bosh-dir is provided as a Concourse cache volume; create if absent
mkdir -p cache-dot-bosh-dir/.bosh

# prepare-bats-config.sh expects its terraform metadata at terraform/metadata
# but deploy-director.sh expects it at environment/metadata.
# Symlink terraform/ → environment/ so both scripts find what they need.
ln -sf "${ROOT_DIR}/environment" "${ROOT_DIR}/terraform"

# ── Install terraform ────────────────────────────────────────────────────────
if ! command -v terraform &>/dev/null; then
echo "--- Installing terraform ${TERRAFORM_VERSION} ---"
curl -sSL \
"https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip" \
-o /tmp/terraform.zip
unzip -qo /tmp/terraform.zip -d /usr/local/bin terraform
chmod +x /usr/local/bin/terraform
Comment on lines +34 to +40

@coderabbitai coderabbitai Bot Jun 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

git ls-files ci/tasks/run-bats-pipeline.sh
wc -l ci/tasks/run-bats-pipeline.sh
cat -n ci/tasks/run-bats-pipeline.sh | sed -n '1,140p'

Repository: cloudfoundry/bosh

Length of output: 7330

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Look for any existing Terraform version pinning / checksum verification in CI scripts.
rg -n "terraform_|sha256|SHA256|shasum|openssl dgst|gpg|verify.*terraform|terraform.*zip" ci .github .gitlab-ci.yml README.md

Repository: cloudfoundry/bosh

Length of output: 5227

🏁 Script executed:

#!/bin/bash
set -euo pipefail

git ls-files ci/tasks/run-bats-pipeline.sh
wc -l ci/tasks/run-bats-pipeline.sh
cat -n ci/tasks/run-bats-pipeline.sh | sed -n '1,140p'

Repository: cloudfoundry/bosh

Length of output: 7330

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Look for any existing Terraform version pinning / checksum verification in CI scripts.
rg -n "terraform_|sha256|SHA256|shasum|openssl dgst|gpg|verify.*terraform|terraform.*zip" ci .github .gitlab-ci.yml README.md

Repository: cloudfoundry/bosh

Length of output: 5227

Verify the Terraform download before installing it
ci/tasks/run-bats-pipeline.sh:34-40 fetches Terraform over the network and installs it into /usr/local/bin with no checksum or signature check. Bake Terraform into the image, or verify the release checksum/signature before unpacking it.

🧰 Tools
🪛 ast-grep (0.44.0)

[warning] 37-37: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/terraform.zip
Note: [CWE-377] Insecure Temporary File.

(predictable-tmp-file-bash)

[warning] 38-38: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/terraform.zip
Note: [CWE-377] Insecure Temporary File.

(predictable-tmp-file-bash)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci/tasks/run-bats-pipeline.sh` around lines 34 - 40, The Terraform install
path in run-bats-pipeline.sh downloads and unpacks a release directly into
/usr/local/bin without any integrity check. Update the Terraform installation
block to either use a pre-baked image/tooling layer or verify the downloaded
archive against the official checksum/signature before calling unzip and chmod,
using the existing terraform install flow and TERRAFORM_VERSION lookup as the
entry point.

fi

# ── GCP credentials file (used by the GCS backend and the Google provider) ──
GCP_CREDS_FILE="$(mktemp /tmp/gcp-creds-XXXXXX.json)"
echo "${GCP_JSON_KEY}" > "${GCP_CREDS_FILE}"
chmod 600 "${GCP_CREDS_FILE}"

# ── Provision GCP environment via terraform ──────────────────────────────────
echo "--- Provisioning GCP environment (env: ${ENV_NAME}) ---"
pushd "${TERRAFORM_DIR}" >/dev/null

terraform init \
-input=false \
-reconfigure \
-backend-config="bucket=bosh-director-pipeline" \
-backend-config="prefix=bats-terraform/${ENV_NAME}" \
-backend-config="credentials=${GCP_CREDS_FILE}"

terraform apply \
-input=false \
-auto-approve \
-var "project_id=${GCP_PROJECT_ID}" \
-var "gcp_credentials_json=${GCP_JSON_KEY}" \
-var "name=${ENV_NAME}"

# Convert terraform outputs to the flat metadata JSON consumed by director-vars
# and prepare-bats-config.sh.
terraform output -json \
| jq 'with_entries(.value = .value.value)' \
> "${ROOT_DIR}/environment/metadata"

popd >/dev/null
Comment on lines +48 to +72

@coderabbitai coderabbitai Bot Jun 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

sed -n '1,220p' ci/tasks/run-bats-pipeline.sh | cat -n

Repository: cloudfoundry/bosh

Length of output: 8522

🏁 Script executed:

grep -nE 'trap|terraform|cleanup|EXIT' ci/tasks/run-bats-pipeline.sh

Repository: cloudfoundry/bosh

Length of output: 1523

🏁 Script executed:

python3 - <<'PY'
from pathlib import Path
p = Path('ci/tasks/run-bats-pipeline.sh')
text = p.read_text()
for i, line in enumerate(text.splitlines(), 1):
    if 40 <= i <= 170:
        print(f"{i:4}: {line}")
PY

Repository: cloudfoundry/bosh

Length of output: 6362

🏁 Script executed:

python3 - <<'PY'
from pathlib import Path
p = Path('ci/tasks/run-bats-pipeline.sh')
text = p.read_text().splitlines()
for start in (1, 40, 70, 110, 130):
    print(f"\n--- {start} ---")
    for i in range(start, min(start+40, len(text)+1)):
        print(f"{i:4}: {text[i-1]}")
PY

Repository: cloudfoundry/bosh

Length of output: 9672

Move the EXIT trap before Terraform runs. terraform init, apply, and output currently run before trap teardown EXIT, so failures there skip cleanup and can leave partial GCP resources behind.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci/tasks/run-bats-pipeline.sh` around lines 48 - 72, Move the EXIT cleanup
setup so it is registered before any Terraform work starts in
run-bats-pipeline.sh; the current flow around terraform init, terraform apply,
and terraform output can fail before teardown is armed, leaving partial GCP
resources behind. Update the script so the trap that calls teardown is installed
before entering the Terraform provisioning block, while keeping the existing
teardown function and ENV_NAME/TERRAFORM_DIR flow intact.


# ── Teardown trap (always runs on EXIT) ─────────────────────────────────────
function collect_director_diagnostics {
# Only collect when we have a deployed director and bosh-cli is available.
[[ -f director-state/director-creds.yml ]] || return 0
[[ -f "${ROOT_DIR}/environment/metadata" ]] || return 0
command -v bosh-cli &>/dev/null || return 0

local director_ip
director_ip="$(jq -r '.director_public_ip // empty' "${ROOT_DIR}/environment/metadata")"
[[ -n "${director_ip}" ]] || return 0

echo "--- Collecting director diagnostics (IP: ${director_ip}) ---"

# Extract jumpbox SSH private key from the vars-store.
local jumpbox_key_file
jumpbox_key_file="$(mktemp /tmp/jumpbox-key-XXXXXX)"
bosh-cli interpolate director-state/director-creds.yml \
--path /jumpbox_ssh/private_key > "${jumpbox_key_file}" 2>/dev/null || { rm -f "${jumpbox_key_file}"; return 0; }
chmod 600 "${jumpbox_key_file}"

ssh -o StrictHostKeyChecking=no \
-o UserKnownHostsFile=/dev/null \
-o ConnectTimeout=10 \
-i "${jumpbox_key_file}" \
"jumpbox@${director_ip}" \
'echo "=== monit status ===" && sudo /var/vcap/bosh/bin/monit status;
echo "=== bosh_nats_sync log (last 100 lines) ===" && sudo tail -100 /var/vcap/sys/log/nats/bosh-nats-sync.log 2>/dev/null || true;
echo "=== bosh_nats_sync bpm stdout ===" && sudo cat /var/vcap/sys/log/bpm/nats/bosh_nats_sync.stdout.log 2>/dev/null || true;
echo "=== bosh_nats_sync bpm stderr ===" && sudo cat /var/vcap/sys/log/bpm/nats/bosh_nats_sync.stderr.log 2>/dev/null || true;
echo "=== nats log (last 50 lines) ===" && sudo tail -50 /var/vcap/sys/log/nats/nats.log 2>/dev/null || true;
echo "=== nats bpm stdout (last 50 lines) ===" && sudo tail -50 /var/vcap/sys/log/bpm/nats/nats.stdout.log 2>/dev/null || true;
echo "=== health_monitor log (last 100 lines) ===" && sudo tail -100 /var/vcap/sys/log/health_monitor/health_monitor.log 2>/dev/null || true;
echo "=== health_monitor bpm stdout ===" && sudo tail -50 /var/vcap/sys/log/bpm/health_monitor/health_monitor.stdout.log 2>/dev/null || true;
echo "=== health_monitor bpm stderr ===" && sudo tail -50 /var/vcap/sys/log/bpm/health_monitor/health_monitor.stderr.log 2>/dev/null || true' \
2>&1 || echo "(SSH diagnostics failed — VM may not be reachable)"

rm -f "${jumpbox_key_file}"
}

function teardown {
local exit_code=$?
set +e

# Always collect diagnostics – on success this helps correlate logs with
# passing runs; on failure it captures the state at the point of failure.
collect_director_diagnostics

echo "--- Tearing down BOSH director ---"
if [[ -f director-state/director-state.json ]]; then
# destroy-director.sh expects bosh-cli/bosh-cli-* to exist; restore it
# because deploy-director.sh already moved the original binary away.
cp /usr/local/bin/bosh-cli bosh-cli/bosh-cli-restore 2>/dev/null || true
bosh-ci/ci/bats/tasks/destroy-director.sh || true
fi

echo "--- Destroying GCP environment (env: ${ENV_NAME}) ---"
pushd "${TERRAFORM_DIR}" >/dev/null
terraform destroy \
-input=false \
-auto-approve \
-var "project_id=${GCP_PROJECT_ID}" \
-var "gcp_credentials_json=${GCP_JSON_KEY}" \
-var "name=${ENV_NAME}" || true
popd >/dev/null

rm -f "${GCP_CREDS_FILE}"

exit "${exit_code}"
}
trap teardown EXIT

# ── Deploy BOSH director ─────────────────────────────────────────────────────
echo "--- Deploying BOSH director ---"
# deploy-director.sh moves bosh-cli/bosh-cli-* to /usr/local/bin/bosh-cli.
# After this call bosh-cli is installed system-wide as 'bosh-cli'.
bosh-ci/ci/bats/tasks/deploy-director.sh

# ── Prepare BATs config ──────────────────────────────────────────────────────
echo "--- Preparing BATs config ---"
bosh-ci/ci/bats/iaas/gcp/prepare-bats-config.sh

# ── Run BATs ─────────────────────────────────────────────────────────────────
echo "--- Running BATs ---"
# Source the environment file that prepare-bats-config.sh wrote; this exports
# BOSH_ENVIRONMENT, BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_CA_CERT,
# BOSH_ALL_PROXY, and the default BAT_RSPEC_FLAGS.
# shellcheck source=/dev/null
source bats-config/bats.env

# Allow the caller to append extra RSpec flags (e.g. "--tag wip").
if [[ -n "${BAT_RSPEC_FLAGS:-}" ]]; then
export BAT_RSPEC_FLAGS="${BAT_RSPEC_FLAGS}"
fi
Comment on lines +157 to +166

@coderabbitai coderabbitai Bot Jun 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate the relevant files and inspect the surrounding lines with numbers.
git ls-files 'ci/tasks/run-bats-pipeline.sh' 'ci/bats/iaas/gcp/prepare-bats-config.sh'
echo '--- run-bats-pipeline.sh ---'
sed -n '130,190p' ci/tasks/run-bats-pipeline.sh | cat -n
echo '--- prepare-bats-config.sh ---'
sed -n '1,120p' ci/bats/iaas/gcp/prepare-bats-config.sh | cat -n

Repository: cloudfoundry/bosh

Length of output: 5730

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Search for BAT_RSPEC_FLAGS usage to understand precedence and intended behavior.
rg -n "BAT_RSPEC_FLAGS|bats\.env|prepare-bats-config" ci -S

Repository: cloudfoundry/bosh

Length of output: 1932

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Probe shell semantics relevant to the review comment:
# 1) does sourcing a file overwrite a previously set variable?
# 2) what happens with the suggested concatenation logic?
python3 - <<'PY'
import os, tempfile, subprocess, textwrap, json

def run(script):
    p = subprocess.run(["bash", "-lc", script], capture_output=True, text=True)
    return p.returncode, p.stdout.strip(), p.stderr.strip()

cases = [
    ("user flags only", 'BAT_RSPEC_FLAGS="--tag wip"; source <(printf \'export BAT_RSPEC_FLAGS="--default"\'); echo "$BAT_RSPEC_FLAGS"'),
    ("concat if default exists", 'BAT_RSPEC_FLAGS="--default"; extra_bat_rspec_flags="--tag wip"; export BAT_RSPEC_FLAGS="${BAT_RSPEC_FLAGS:+${BAT_RSPEC_FLAGS} }${extra_bat_rspec_flags}"; echo "$BAT_RSPEC_FLAGS"'),
    ("concat if default empty", 'unset BAT_RSPEC_FLAGS; extra_bat_rspec_flags="--tag wip"; export BAT_RSPEC_FLAGS="${BAT_RSPEC_FLAGS:+${BAT_RSPEC_FLAGS} }${extra_bat_rspec_flags}"; echo "${BAT_RSPEC_FLAGS-<unset>}"'),
]
for name, script in cases:
    rc, out, err = run(script)
    print(f"[{name}] rc={rc} out={out!r} err={err!r}")
PY

Repository: cloudfoundry/bosh

Length of output: 670

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the exact lines around the implementation to reason about precedence.
nl -ba ci/tasks/run-bats-pipeline.sh | sed -n '150,175p'
echo '---'
nl -ba ci/bats/iaas/gcp/prepare-bats-config.sh | sed -n '1,60p'

Repository: cloudfoundry/bosh

Length of output: 195

Preserve BAT_RSPEC_FLAGS before sourcing bats.env.
prepare-bats-config.sh writes a default export BAT_RSPEC_FLAGS=... into bats-config/bats.env, so source bats-config/bats.env overwrites any caller-supplied flags. Save the existing value first, then append it after sourcing.

Suggested fix 
+# Preserve user-supplied extra flags before sourcing the generated defaults.
+extra_bat_rspec_flags="${BAT_RSPEC_FLAGS:-}"
+
 # shellcheck source=/dev/null
 source bats-config/bats.env
 
 # Allow the caller to append extra RSpec flags (e.g. "--tag wip").
-if [[ -n "${BAT_RSPEC_FLAGS:-}" ]]; then
-  export BAT_RSPEC_FLAGS="${BAT_RSPEC_FLAGS}"
+if [[ -n "${extra_bat_rspec_flags}" ]]; then
+  export BAT_RSPEC_FLAGS="${BAT_RSPEC_FLAGS:+${BAT_RSPEC_FLAGS} }${extra_bat_rspec_flags}"
 fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Source the environment file that prepare-bats-config.sh wrote; this exports
# BOSH_ENVIRONMENT, BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_CA_CERT,
# BOSH_ALL_PROXY, and the default BAT_RSPEC_FLAGS.
# shellcheck source=/dev/null
source bats-config/bats.env
# Allow the caller to append extra RSpec flags (e.g. "--tag wip").
if [[ -n "${BAT_RSPEC_FLAGS:-}" ]]; then
export BAT_RSPEC_FLAGS="${BAT_RSPEC_FLAGS}"
fi
# Preserve user-supplied extra flags before sourcing the generated defaults.
extra_bat_rspec_flags="${BAT_RSPEC_FLAGS:-}"
# Source the environment file that prepare-bats-config.sh wrote; this exports
# BOSH_ENVIRONMENT, BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_CA_CERT,
# BOSH_ALL_PROXY, and the default BAT_RSPEC_FLAGS.
# shellcheck source=/dev/null
source bats-config/bats.env
# Allow the caller to append extra RSpec flags (e.g. "--tag wip").
if [[ -n "${extra_bat_rspec_flags}" ]]; then
export BAT_RSPEC_FLAGS="${BAT_RSPEC_FLAGS:+${BAT_RSPEC_FLAGS} }${extra_bat_rspec_flags}"
fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci/tasks/run-bats-pipeline.sh` around lines 157 - 166, Preserve
caller-supplied BAT_RSPEC_FLAGS in run-bats-pipeline.sh by saving its current
value before sourcing bats-config/bats.env, since that file sets a default
export which overwrites it. Update the sourcing flow around the bats.env load so
the existing BAT_RSPEC_FLAGS is restored or appended afterward, keeping the
behavior in the BAT_RSPEC_FLAGS handling block and the bats-config/bats.env
source step consistent.


bats/ci/tasks/run-bats.sh
49 changes: 49 additions & 0 deletions src/tasks/fly.rake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
require 'shellwords'

namespace :fly do
desc 'Fly unit specs'
task :unit do
Expand All @@ -9,6 +11,53 @@ namespace :fly do
COVERAGE: ENV.fetch('COVERAGE', false))
end

desc 'Run BATs (BOSH Acceptance Tests) against the current branch via Concourse'
#
# Sets the ci/fly-bats.yml pipeline on Concourse, then triggers and watches
# the bats job. The current branch is pushed to origin automatically so
# Concourse can fetch it.
#
# GCP credentials are resolved from the Concourse credential store
# (((gcp_json_key)) and ((gcp_project_id))).
#
# Useful env vars:
# BATS_ENV_NAME – terraform env name, must be unique per concurrent run
# (default: "bats-local")
# STEMCELL_NAME – GCP stemcell name override
# DEPLOY_ARGS – extra ops-files passed to bosh create-env
# BAT_RSPEC_FLAGS – extra flags appended to the RSpec BATs run
task :bats do
env_name = ENV.fetch('BATS_ENV_NAME', 'bats-local')

branch = `git -C .. rev-parse --abbrev-ref HEAD`.strip
repo = `git -C .. remote get-url origin`.strip
.sub(/\Agit@github\.com:/, 'https://github.com/')
.sub(/\.git\z/, '.git')

# Push the current branch so Concourse can check it out.
sh "git -C .. push origin HEAD"

# ── Set the pipeline ─────────────────────────────────────────────────────
sh [
"fly #{concourse_target}",
'set-pipeline',
'--non-interactive',
'--pipeline bats-local',
'--config ../ci/fly-bats.yml',
"--var bosh_repo=#{Shellwords.escape(repo)}",
"--var bosh_branch=#{Shellwords.escape(branch)}",
"--var env_name=#{Shellwords.escape(env_name)}",
"--var stemcell_name=#{Shellwords.escape(ENV.fetch('STEMCELL_NAME', 'bosh-google-kvm-ubuntu-noble'))}",
"--var deploy_args=#{Shellwords.escape(ENV.fetch('DEPLOY_ARGS', '-o bosh-deployment/external-ip-not-recommended.yml'))}",
"--var bat_rspec_flags=#{Shellwords.escape(ENV.fetch('BAT_RSPEC_FLAGS', ''))}",
].compact.join(' ')

sh "fly #{concourse_target} unpause-pipeline --pipeline bats-local"

# ── Trigger and stream the job output ────────────────────────────────────
sh "fly #{concourse_target} trigger-job --job bats-local/bats --watch"
end

desc 'Fly integration specs'
task :integration, [:cli_dir] do |_, args|
db, db_version = fetch_db_and_version('postgresql')
Expand Down
Loading