Release: Add semi-automated release pipeline#16259
Draft
RussellSpitzer wants to merge 2 commits into
Draft
Conversation
Adds a semi-automated release pipeline that wraps the manual process
documented in site/docs/how-to-release.md (the doc is unchanged in this
PR). The release manager triggers GitHub Actions workflows for
prepare-rc, publish, and cancel-rc; the workflows run scripts under
release/ that mirror the work a PMC member would do by hand.
dev/source-release.sh and dev/stage-binaries.sh are preserved unchanged
as a fallback.
Branching follows Iceberg's existing convention: X.Y.0 RCs are tagged
from main; X.Y.x release branches are created later, after X.Y.0 ships,
from the apache-iceberg-X.Y.0 final tag. The automation enforces this
rather than auto-creating branches.
Scripts (release/)
prepare-rc.sh End-to-end pre-vote: tags from main (X.Y.0) or the
X.Y.x branch (patch); auto-detects RC number;
verifies GitHub CI; aligns the workspace with the
release commit so tarball + binaries match; builds
+ signs source tarball; SVN-stages artifacts;
publishes convenience binaries to Nexus across the
Scala/Spark/Flink/Kafka matrix; closes the Nexus
staging repo; creates a GitHub pre-release; emits
the [VOTE] email. --remote defaults to apache when
that remote exists, falling back to origin.
publish-release.sh Post-vote: refuses to run if HEAD does not match
the RC tag commit; verifies the Nexus staging
repo (profile, state, iceberg-core artifact,
description) before any destructive action;
svn-promotes dist/dev to dist/release; removes
superseded patch releases for the same
MAJOR.MINOR line (e.g. publishing 1.9.2 removes
1.9.0 and 1.9.1; older minor lines are left for
the standard archive.apache.org flow); tags the
final release; promotes Nexus to Maven Central;
creates the GitHub Release; emits the [ANNOUNCE]
email.
cancel-rc.sh Post-failed-vote: same staging-repo verification;
drops the Nexus staging repo; removes RC artifacts
from dist/dev; emits the failure email. The RC tag
is left in place per ASF policy.
Convenience-binary publish matrix
The Scala/Flink/Spark/Kafka matrix is read directly from
gradle.properties (systemProp.knownXxxVersions / defaultScalaVersion)
at module-load time, so the release pipeline cannot drift from the
build's notion of supported versions. Two Gradle passes:
1. Primary-Scala main pass (Scala 2.12 today): full Flink/Kafka
matrix and the Spark versions that still support the primary
Scala -- today Spark 3.x only. Spark 4.x is intentionally excluded
because it dropped Scala 2.12 support and pairing them on the
same `gradlew` invocation is brittle.
2. Secondary-Scala (Spark-only) sweep: one invocation per Spark
version in knownSparkVersions, publishing the spark /
spark-extensions / spark-runtime modules against the secondary
Scala (2.13 today). Covers Spark 3.x (adds the 2.13 variant
alongside 2.12) and Spark 4.x (published only here).
Credential handling
* SVN: every svn invocation goes through release/libs/_svn.sh
helpers (svn_run, svn_run_with_retries, svn_path_exists, svn_list)
that pipe SVN_PASSWORD on stdin via --password-from-stdin so it
never appears in argv (and so is not visible via /proc/<pid>/cmdline).
* Nexus REST: credentials are passed to curl via -K on stdin.
* Gradle publish: credentials are exported as
ORG_GRADLE_PROJECT_mavenUser / mavenPassword env vars rather than
-PmavenUser=/-PmavenPassword= so they never enter argv.
* _redact_secrets covers every secret env var the workflows inject.
Staging-repository verification
In the manual process the release manager visually verifies the right
staging repo in the Nexus UI before clicking Release/Drop. Both
publish-release.sh and cancel-rc.sh perform that check explicitly:
profile == org.apache.iceberg (hard fail)
state == closed (hard fail)
iceberg-core-<version>.pom is present (hard fail)
description contains the expected RC label (warn; bypass with
--allow-description-mismatch)
Shared libraries (release/libs/)
_constants.sh Tag prefix, SVN/Nexus URLs, GitHub repo, version
regexes, and the convenience-binary matrix sourced
from gradle.properties via _parse_gradle_property /
_require_gradle_property. Derives SCALA_PRIMARY,
SCALA_SECONDARY, SPARK_VERSIONS_PRIMARY_SCALA (Spark
major < 4) and SPARK_VERSIONS_SECONDARY_SCALA (all
known Spark) from the parsed values.
_log.sh Color-aware print_*; step_summary writes to
GITHUB_STEP_SUMMARY when present.
_exec.sh exec_process / exec_process_with_retries with secret
redaction; calculate_sha512.
_version.sh validate_and_extract_version,
validate_and_extract_git_tag_version,
validate_and_extract_branch_version,
find_next_rc_number, find_latest_rc_number,
expected_branch_for_version.
_github.sh check_github_checks_passed (gh CLI based).
_nexus.sh Nexus 2 staging API close/promote/drop with
credentials passed via curl -K stdin;
nexus_find_open_staging_repo;
nexus_verify_staging_repo guards publish/cancel
against wrong-repo mistakes.
_svn.sh Credential-on-stdin svn helpers (see above) plus
filter_superseded_patch_releases for the
same-MAJOR.MINOR cleanup pass.
_gradle.sh verify_jdk_17, build_source_tarball (mirrors
dev/source-release.sh), and stage_convenience_binaries
(two-pass policy described above).
Workflows (.github/workflows/)
release-prepare-rc.yml workflow_dispatch with version, rc_number,
dry_run inputs.
release-publish.yml Promotes a passed RC. Exposes
allow_description_mismatch input.
release-cancel-rc.yml Drops a failed RC. Same flag.
ci-release-scripts.yml Runs bats + shellcheck on every PR touching
release/.
All three release workflows share a single concurrency group
(iceberg-release, cancel-in-progress: false) so two dispatches cannot
overlap on git tags, the Nexus staging repo, or SVN paths. Workflows
run on ubuntu-24.04 with the maven-publish environment, pinned action
SHAs, and apt-installed subversion + jq.
Bats test suite (release/tests/)
186 tests across constants, log, exec, version, github, nexus,
gradle, svn, end-to-end branch management, publish_release, and
cancel_rc. Highlights:
* verify_jdk_17 across OpenJDK 11/17/21 and the legacy 1.8 string.
* Table-driven _redact_secrets coverage for every secret env var
consulted by the workflows (including
ORG_GRADLE_PROJECT_mavenUser/Password).
* exec_process_with_retries final-failure log line is redacted.
* _parse_gradle_property: trims systemProp.* and bare keys, fails
cleanly on missing key / missing file.
* Constants drift guard: independently re-parses gradle.properties
and asserts every constant exposed by _constants.sh matches.
SPARK_VERSIONS_PRIMARY_SCALA equals KNOWN_SPARK_VERSIONS filtered
to major < 4; SPARK_VERSIONS_SECONDARY_SCALA equals all of
KNOWN_SPARK_VERSIONS.
* stage_convenience_binaries argv drift guard: main pass carries
the primary-Scala Spark subset, secondary sweep covers every
Spark version (3.x + 4.x); a per-line check asserts no `gradlew`
invocation ever combines -DscalaVersion=2.12 with Spark 4.x.
* nexus_find_open_staging_repo: zero / one / many open repos plus
curl-failure path.
* nexus_verify_staging_repo: profile mismatch, repo open, repo
released, missing artifact, description mismatch,
--allow-description-mismatch override, dry-run no-op.
* svn_run: real-mode confirms password is on stdin and not in argv;
dry-run prints the redacted command without invoking svn.
* svn_run_with_retries: success/retry/give-up paths with
cleanup_path removal.
* filter_superseded_patch_releases: 1.9.2 removes 1.9.0/1.9.1 only;
1.10.x and 2.0.x survive; KEYS and other non-release entries
ignored; boundary anchoring rejects 1.90.x cross-matches; new
version itself never included.
* Hermetic bare-upstream + clone fixture exercising prepare-rc.sh
branch-selection rules end-to-end in dry-run, including the
X.Y.0-from-main path, the X.Y.Z-requires-existing-branch path,
the apache-vs-origin remote default, and the workspace
alignment when HEAD lags origin/main.
* publish_release.bats: arg parsing, missing/duplicate tag guards,
HEAD-vs-RC-commit guard, RC-not-latest guard, --keep-old-releases
messaging, MAJOR.MINOR.x-scoped dry-run wording, dry-run happy
path, and a secret-leak canary.
* cancel_rc.bats: arg parsing, validation, dry-run happy path,
--allow-description-mismatch handling, and a secret-leak canary.
Iceberg-specific notes
* No version-bump commit on the release branch: gitVersion plugin
derives the development version from tags.
* Tag prefix is apache-iceberg-; SVN paths are /dev/iceberg and
/release/iceberg.
Local validation: 186 bats tests pass, shellcheck clean, dry-run smoke
of all three scripts succeeds end-to-end.
The whole-output glob `[[ "$output" != *"-DscalaVersion=2.12"*"4.0"* ]]` matched across the legitimate Scala 2.13 sweep (which does include Spark 4.0/4.1), producing a false failure even though the script's behavior is correct. Filter to lines carrying `-DscalaVersion=2.12` before checking, matching the technique used by the existing argv drift guard.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Contributor
|
Not stale |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Contributor
|
Not stale, not unless to be superseded by the ASF release automation infrastructure |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Includes 186 bats unit tests covering all shared libraries and end-to-end dry-runs of every script.
Rationale for this change
The current Iceberg release process is documented in
How to Release and driven by hand
plus the existing
dev/source-release.sh,dev/stage-binaries.sh, anddev/check-licensescripts. Most of it (Nexus close/release/drop, SVNpromotion, old-release cleanup, dist/dev → dist/release move, final tag,
GitHub Release, vote/announce emails) is still manual.
This PR adds a semi-automated release pipeline modelled after the
Apache Polaris release tooling
and the recent Apache Parquet PR #3548,
adapted to Iceberg's Gradle build and existing release scripts. It does
not change
dev/source-release.sh,dev/stage-binaries.sh, orsite/docs/how-to-release.md; the manual flow remains a fallback.What changes are included in this PR?
Scripts (
release/):prepare-rc.sh— full pre-vote flow: branch validation, RC tag, GitHub CIcheck, source tarball + GPG signing, SVN dist/dev staging, multi-axis
Gradle convenience-binary publish, Nexus staging-repo close, GitHub
pre-release,
[VOTE]email template.publish-release.sh— full post-vote flow: HEAD-vs-RC-tag guard, Nexusstaging-repo verification,
svn mvto dist/release, same-MAJOR.MINORpatch-release cleanup, final tag push, Nexus release to Maven Central,
GitHub Release,
[ANNOUNCE]email template.cancel-rc.sh— failed-vote rollback: same staging-repo verification,Nexus drop, dist/dev SVN cleanup,
[RESULT][VOTE]failure emailtemplate. RC git tag is left in place per ASF policy.
Shared libraries (
release/libs/):_constants.sh— config + version regexes. The Scala/Flink/Spark/Kafkamatrix is read directly from
gradle.properties(systemProp.knownXxxVersions/
defaultScalaVersion) so the pipeline cannot drift from the build._log.sh,_exec.sh,_version.sh,_github.sh._nexus.sh— Nexus 2 staging API helpers;nexus_verify_staging_repochecks profile / state /
iceberg-core-<version>.pompresence /description.
_svn.sh—svnhelpers that pipeSVN_PASSWORDon stdin via--password-from-stdinso it never appears in argv (and is not visiblevia
/proc/<pid>/cmdline); plusfilter_superseded_patch_releasesforthe same-
MAJOR.MINORcleanup._gradle.sh—verify_jdk_17,build_source_tarball(mirrorsdev/source-release.sh), andstage_convenience_binaries(two-passpublish, see "Safety" below).
GitHub Actions workflows:
release-prepare-rc.yml,release-publish.yml,release-cancel-rc.yml— dispatch-only entry points; default to dry-run.
ci-release-scripts.yml— runs bats + shellcheck on every PR/push thattouches
release/.All three release workflows share a single
concurrencygroup(
iceberg-release,cancel-in-progress: false) so two dispatches cannotoverlap on git tags, the Nexus staging repo, or SVN paths.
Are these changes tested?
Locally only:
bats release/tests→ 186 tests pass (constants, log, exec, version,github, nexus, gradle, svn, end-to-end branch management,
publish-release,cancel-rc).shellcheck release/*.sh release/libs/*.sh→ clean.hermetic fixture.
To actually exercise this in production we need an Infra ticket to land
the appropriate secrets on
apache/iceberg:NEXUS_USER/NEXUS_PW(Apache Nexus, already used bypublish-snapshot.yml).ICEBERG_SVN_DEV_USERNAME/ICEBERG_SVN_DEV_PASSWORD(dist.apache.orgcredentials).maven-publishGitHub environment guarding all three release workflows.Are there any user-facing changes?
No. The release pipeline is invisible to end users; this only affects
release managers and contributors.
site/docs/how-to-release.mdis intentionally not updated in thisPR — we want the workflow to land first, then update the docs in a
follow-up PR once the secrets are in place and a real RC has been driven
through it.
Release Workflow
This PR layers three GitHub Actions workflows + locally-runnable Bash
scripts on top of the existing manual flow. All workflows default to
dry-run mode; the release manager has to explicitly uncheck the
Dry runcheckbox before any external service is touched.1. Prepare RC (Pre-Vote)
The release manager launches the "Release - Prepare Release Candidate"
workflow (
release-prepare-rc.yml) viaworkflow_dispatchwith:1.10.0trueBranching follows Iceberg's existing convention: X.Y.0 RCs are tagged
from
main; X.Y.Z (Z ≥ 1) requires theX.Y.xbranch to alreadyexist on the remote (the script does not auto-create branches — the
release manager creates
X.Y.xafter the X.Y.0 release per currentpractice).
prepare-rc.shgpg,svn, JDK 17, gradlew)apache-iceberg-X.Y.Z-rc*tagsgh api .../check-runs)origin/main; X.Y.Z →origin/X.Y.x); align workspace viagit checkout --detachapache-iceberg-X.Y.Z-rcNtagdev/source-release.sh.sha512dev/source-release.shdist/dev/iceberg/<rc>dev/source-release.shdev/stage-binaries.sh[VOTE]email template into the workflow step summaryThe release manager copies the
[VOTE]email out of the step summary andmanually sends it to
dev@iceberg.apache.org.2. Vote
The community votes for at least 72 hours. The release manager tallies
results manually.
If the vote fails → Cancel RC
The release manager launches the "Release - Cancel Release Candidate"
workflow (
release-cancel-rc.yml) with:1.10.0orgapacheiceberg-1234truecancel-rc.shdist/dev/iceberg/<rc>from SVNsvn rm[RESULT][VOTE]failure email templateThe RC git tag is preserved per ASF release policy.
If the vote passes → Publish Release
The release manager launches the "Release - Publish After Vote"
workflow (
release-publish.yml) with:1.10.0orgapacheiceberg-1234truepublish-release.shsvn mvfromdist/dev/iceberg/<rc>todist/release/iceberg/apache-iceberg-X.Y.Zsvn mvMAJOR.MINORline (e.g. publishing 1.9.2 removes 1.9.0 / 1.9.1; older minor lines are left for the standardarchive.apache.orgflow).--keep-old-releasesskips this.apache-iceberg-X.Y.Zfinal taggit tag[ANNOUNCE]email template + reminders forrevapi, issue templates,doap.rdf, versioned docs/Javadoc, site updateIceberg has no version-bump commit on the release branch — the
gitVersionplugin derives the development version from the latest tag —so unlike Parquet there's no
release:update-versionsstep.Safety: Staging Repository Verification
Before any destructive Nexus action (promote in publish, drop in cancel),
both scripts verify the staging repo against the requested
(version, rc_number)pair:org.apache.iceberg(hard fail)closed(hard fail)iceberg-core-<version>.pomexists in the repo (hard fail)Apache Iceberg <version> RC<rc>(warn; bypass with--allow-description-mismatch/ the workflow input)This closes a gap in the current manual flow: typing a staging-repo ID
into a form field replaces the human "look at the artifact tree in
Nexus" check, so the script does that look-up explicitly.
Safety: Convenience-binary publish matrix
The Scala/Flink/Spark/Kafka matrix is read at module-load time from
gradle.properties(systemProp.knownXxxVersions/defaultScalaVersion).Two Gradle passes:
versions that still support Scala 2.12 (i.e. Spark 3.x). Spark 4+ is
intentionally excluded because it dropped Scala 2.12 support.
./gradlewinvocationper Spark version in
knownSparkVersions, publishing theiceberg-spark-<v>_2.13,iceberg-spark-extensions-<v>_2.13, andiceberg-spark-runtime-<v>_2.13modules. This covers Spark 3.x(adding the 2.13 variant) and Spark 4.x (which is published only
here).
Net artifact set:
iceberg-spark-{3.4,3.5}_{2.12,2.13}plusiceberg-spark-{4.0,4.1}_2.13. A bats drift guard re-parsesgradle.propertiesand asserts every Gradle invocation in the dry-runmatches; another guard asserts no
gradlewline ever combines-DscalaVersion=2.12with-DsparkVersions=...4.x.Safety: Credentials
svncall goes throughrelease/libs/_svn.shhelpersthat pipe
SVN_PASSWORDon stdin via--password-from-stdin. Neveron argv.
curlvia-Kon stdin.ORG_GRADLE_PROJECT_mavenUser/mavenPasswordenv vars rather than-PmavenUser=/-PmavenPassword=. Never on argv._redact_secretscovers every secret env var the workflows inject andis exercised by table-driven bats tests.
What this PR is not
dev/source-release.sh,dev/stage-binaries.sh, ordev/check-license. The manual flow remains a supported fallback.site/docs/how-to-release.md. Documentation will beupdated in a follow-up PR once the workflows have driven a real RC end
to end.
bash,bats,shellcheck,gh,svn,curl,jq,gpg,gradle/./gradlew,Java 17 — all already available on
ubuntu-24.04runners or installedvia
apt.Follow-ups
NEXUS_USER/NEXUS_PW/ICEBERG_SVN_DEV_USERNAME/ICEBERG_SVN_DEV_PASSWORDand themaven-publishenvironment onapache/iceberg.dry_rununchecked) on the nextrelease cycle.
site/docs/how-to-release.mdto describethe semi-automated flow alongside the manual one.