Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 21 additions & 17 deletions .github/workflows/attest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,9 @@

# Reusable workflow that signs and attests a release artifact.
#
# This is intended to become the only place in the repository that invokes
# `cosign sign`, `cosign attest`, or `cosign attest-blob`. It is NOT yet: at the
# time this workflow landed, publish.yml still signs the container image inline.
# #266, #267 and #268 route the image, chart and binaries through here; #274
# adds the test that keeps it true afterwards. Concentrating signing does two
# things (ADR-074 decision 4):
# This is the only place in the repository that invokes `cosign sign`,
# `cosign attest`, or `cosign attest-blob`. `TestAttestIsSoleSigner` keeps that
# true. Concentrating signing does two things (ADR-074 decision 4):
#
# 1. It gives every released artifact ONE certificate identity to pin. cosign
# uses the OIDC `job_workflow_ref` as the Fulcio SAN, so signing from a
Expand All @@ -17,22 +14,29 @@
# https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/<TAG>
#
# Signing inline in each caller would instead produce one identity per
# calling workflow and per ref, which is what makes the current
# SECURITY.md verification command accept a main-branch build as a release.
# calling workflow and per ref, which is what made the pre-epic SECURITY.md
# verification command accept a main-branch build as a release.
#
# 2. It isolates the signing token from caller-defined build steps.
#
# This is SLSA Build **L2**, not L3, and nothing here may claim otherwise. L3
# requires the BUILD to run inside the protected reusable workflow; the builds
# stay in the callers (`docker buildx` in publish.yml, the Go cross-compile and
# `helm package` in release.yml) and this workflow signs a digest it is handed.
# A caller that produced the wrong artifact would get a faithful signature over
# the wrong digest. Reaching L3 means moving artifact generation in here, which
# is a separate decision.
# This is SLSA Build **L2**, not L3, and nothing here may claim otherwise.
# Provenance **origin** fields are unforgeable by the build process today
# (minted here from trusted context: GITHUB_REPOSITORY / REF / SHA / SERVER_URL /
# RUN_ID and github.workflow_ref). `externalParameters.subjectKind` is an
# enum-validated pass-through of inputs.subject_kind and is out of that origin
# claim. Builder isolation is absent: the builds stay in the callers —
# `docker buildx` inside the reusable `build-image.yml` (invoked by
# publish.yml / release.yml), and the Go cross-compile / `helm package` in
# release.yml — and this workflow signs a digest it is handed. A caller that
# produced the wrong artifact would get a faithful signature over the wrong
# digest. Reaching L3 means moving artifact generation behind a protected
# build+attest boundary (see ADR-074 decision 4 deferred path), which is a
# separate decision.
#
# Because the build happens in the caller, `runDetails.builder.id` in the
# provenance names the CALLER's workflow, not this one. Naming the attestor as
# the builder would make the predicate false on its face.
# provenance names the CALLER's top-level workflow (release.yml / publish.yml),
# not this one and not build-image.yml where image builds actually run. Naming
# the attestor as the builder would make the predicate false on its face.

name: Attest (reusable)

Expand Down
11 changes: 6 additions & 5 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -440,9 +440,10 @@ jobs:
path: release-assets/
retention-days: 7

# Nine published assets each need a verifying bundle: the four binaries, the
# installer, and the four SBOMs. attest.yml signs one subject per call, so
# this is a matrix over subjects rather than nine near-identical jobs.
# Ten published assets each need a verifying bundle: the four binaries, the
# installer, THIRD_PARTY_NOTICES.md, and the four SBOMs. attest.yml signs one
# subject per call, so this is a matrix over subjects rather than ten
# near-identical jobs.
#
# The binaries carry their SBOM as a predicate, which binds the SBOM to the
# binary. The SBOM files are ALSO signed in their own right, because a user
Expand All @@ -458,8 +459,8 @@ jobs:
id-token: write
strategy:
fail-fast: false
# Nine legs, and the four binary legs sign twice, so an unbounded fan-out
# is ~13 near-simultaneous calls to public-good Fulcio and Rekor from one
# Ten legs, and the four binary legs sign twice, so an unbounded fan-out
# is ~14 near-simultaneous calls to public-good Fulcio and Rekor from one
# caller. attest.yml retries on a fixed 5s/10s backoff with no jitter, so
# legs that fail together retry together. The gate requires every leg, so
# one correlated blip fails the whole release. Three at a time keeps the
Expand Down
4 changes: 4 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@ We credit reporters of confirmed vulnerabilities in the release notes of the fix

Use `--certificate-identity`, not `--certificate-identity-regexp`. An identity that names no workflow and no ref also accepts images built from branches, which are not releases and are labelled non-production when they are signed.

**Build Level (per artifact).** Artifacts this repository publishes with Build L2 provenance today include the `manager` image (index provenance), the Helm chart, the `nvcrectl` binaries, `installer`, `THIRD_PARTY_NOTICES.md`, and the standalone `nvcrectl-*.cyclonedx.json` SBOM release assets. The claim is per-artifact, not project-wide: nothing here states a single level for "the release". Provenance origin fields are unforgeable by the build process because they are minted inside the reusable `attest.yml` workflow (Fulcio names that workflow; origin fields come from `GITHUB_*` context; a guard refuses to name `attest.yml` as the builder). What is absent is builder isolation — the build still runs in the caller (for images, inside `build-image.yml` invoked by the top-level orchestrator) — so we do not claim L3. A same-repo `uses: ./…` boundary additionally rests on branch protection over `.github/workflows/attest.yml` for `main` pushes, and on tag protection / repository rulesets over `v*` tags for the release identity everyone pins.

Pinning the workflow identity above is what makes that level checkable. Verification that omits `--certificate-identity` (or loosens it to a regexp) can still succeed while proving a weaker claim: it no longer distinguishes an attestation minted inside `attest.yml` from one produced elsewhere. `TestVerificationUsesExactIdentity` and `TestPublishedVerifyCommandsAreExact` in `test/releasepolicy` reject the `--certificate-identity-regexp` form (and bare `gh attestation verify` in release-path workflows); the published verification page's exact NVIDIA identity pin is gated by `TestVerificationPagePinsAnExactIdentity` in `test/docspolicy`. The reusable-workflow boundary itself is gated by `TestAttestIsSoleSigner`, `TestAttestIsInvokedAsReusableWorkflow`, `TestAttestPredicateUsesOnlyTrustedContext`, and `TestAttestBuilderIdGuardRejectsAttestorAsBuilder`. No level is claimed here without a test behind it.

Retrieve the provenance with `cosign verify-attestation --type slsaprovenance1` against the index digest, and a platform's SBOM with `--type cyclonedx` against that platform's manifest digest (`crane digest --platform linux/amd64 "${IMAGE}:${TAG}"`).

- CLI binaries, the installer and the SBOMs are each signed, and every release asset ships with a detached Sigstore bundle (`<asset>.sigstore.json`) verified under the same identity as the image:
Expand Down
25 changes: 19 additions & 6 deletions docs/designs/074-supply-chain-attestation.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,24 @@ This contract is published to users and enforced by us. Both the release-time ga

The reason is decision 3. cosign uses the OIDC `job_workflow_ref` as the certificate SAN, so signing from one reusable workflow collapses every artifact onto one identity path where only the ref varies. Signing inline in each caller would give the chart, the image, and the binaries three different identities, and a `main` build a fourth that looks just as legitimate. It also isolates the signing step: caller-defined build steps run in a different job from the one holding the signing token.

**This design targets SLSA Build L2, not L3.** The distinction matters and is easy to overclaim. L3 requires the *build* to be isolated from user-defined steps, and GitHub's mechanism for that is moving the build itself into the reusable workflow. Here the builds stay in the callers — `docker buildx` in `publish.yml`, the Go cross-compile and `helm package` in `release.yml` — and `attest.yml` receives a digest and signs it. A caller that produced the wrong artifact would get a faithful signature over the wrong digest. Isolating the signer is worth having, but it is not the isolation L3 asks for.
**This design targets SLSA Build L2 for every artifact we publish today — the `manager` image, the Helm chart, the `nvcrectl` binaries, and `installer` alike** (plus the blob subjects that also carry provenance, such as `THIRD_PARTY_NOTICES.md` and the standalone SBOM files). Claims are per-artifact, not project-wide: nothing here may state a single Build Level for "the release" as a whole.

Two consequences follow. First, no artifact or document may claim L3 — not the ADR, not the release notes, not `SECURITY.md`. Second, the provenance predicate's `runDetails.builder.id` must name the workflow that actually performed the build, not `attest.yml`. Naming the attestor as the builder would make the predicate false on its face, which is worse than claiming the wrong level.
The original L2 framing was imprecise in a way that made L3 look further away than it is. SLSA v1.0 Build L3 requires that **provenance is unforgeable by the build process**. That is not the same thing as builder isolation. On the unforgeability requirement, `attest.yml` already does most of the work:

Reaching L3 later means moving image, chart, and binary generation into the protected reusable workflow. That is a larger restructure than this record covers — it rewrites the build path rather than adding to it — and it should be its own decision once this contract is in place and stable. Recorded as deferred, not rejected.
- It is a genuine reusable workflow (`on: workflow_call`), invoked as `uses: ./.github/workflows/attest.yml` from the callers. It is not an inlined job, so the Fulcio certificate names **`attest.yml`**, not the caller.
- Provenance **origin** fields — repository, ref, commit, server, run id, and `builder.id` — come from trusted context **inside** that workflow (`GITHUB_REPOSITORY`, `GITHUB_REF`, `GITHUB_SHA`, `GITHUB_SERVER_URL`, `GITHUB_RUN_ID`, and the caller workflow ref GitHub itself sets) and not from a caller-supplied `inputs.*` field. A caller cannot dictate what the predicate says about origin. (`externalParameters.subjectKind` is an enum-validated pass-through of `inputs.subject_kind` and is out of that origin claim.)
- A guard refuses to proceed if `builder_id` resolves to `attest.yml`, on the grounds that the attestor cannot be the builder. Naming the attestor as the builder would make the predicate false on its face, which is worse than claiming the wrong level.

So the build steps in the callers cannot forge or tamper with provenance origin fields today. What is genuinely absent is **builder isolation**: the builds stay in the callers — `docker buildx` inside the reusable `build-image.yml` (invoked by `publish.yml` / `release.yml`), and the Go cross-compile and `helm package` in `release.yml` — and `attest.yml` receives a digest and signs it. `runDetails.builder.id` truthfully names the top-level caller (`release.yml` / `publish.yml`), which for images orchestrates rather than runs the build steps themselves. A compromised caller would produce a bad artifact that `attest.yml` would then honestly attest. That honesty is deliberate, and any change here must not trade a true predicate for a higher number.

Two consequences follow. First, no artifact or document may claim L3 — not the ADR, not the release notes, not `SECURITY.md` — until a recorded decision moves a specific artifact across that line. Second, the same-repo reusable-workflow caveat must be stated whenever this boundary is discussed: `uses: ./…` isolates attestation from the caller's *build steps*, not from *write access*. Caller and attestor live in one repository. For the **dev-image** identity (`publish.yml` on `main`), branch protection over `.github/workflows/attest.yml` is the relevant control. For the **release** identity (`release.yml` on `v*` tags), the SAN everyone pins (`attest.yml@refs/tags/${TAG}`) is minted from whatever commit the pushed tag points at, so the control that gates release identity is **tag protection / repository rulesets over `v*` tags**, not branch protection alone. Claiming L3 without saying so would imply stronger isolation than exists.

A concrete path to per-artifact L3 for images exists and is recorded as deferred, not rejected: `build-image.yml` is already a reusable workflow that builds the image; if provenance were minted there, build and attestation would share one protected boundary. Two costs of that path must be recorded now so a follow-on does not discover them after building it:

1. **Identity mechanics.** If provenance used the same derivation `attest.yml` uses today (`github.workflow_ref`), `builder.id` would still name the top-level `release.yml` / `publish.yml`, not `build-image.yml`. The honest, L3-qualifying identity comes from `job_workflow_ref` — the pattern in GitHub's SLSA Build L3 guide cited in References.
2. **Sole-signer split.** Signing inside `build-image.yml` moves the Fulcio SAN off `attest.yml`, splitting the single pinned identity that decision 3 and Alternative 2 exist to protect: image attestations would verify only under a `build-image.yml@…` identity while the chart and binaries stayed under `attest.yml@…`, every published image verify command would break, and `TestAttestIsSoleSigner` would fail.

That would still be an image-only change — chart, `installer`, and `nvcrectl` binaries would stay at L2 — which is why every published claim must stay per-artifact. Until that decision is taken, the image, chart, binaries, and installer remain Build L2 under the corrected reasoning above.

`attest.yml` validates every input before use: digests must match `^sha256:[0-9a-f]{64}$`, no input may contain a newline or carriage return, and the caller's authoritative `expected_digest` is compared against an independently resolved digest with a mismatch failing the job. It refuses to run on a non-tag ref unless an explicit `allow_untagged` input is set, so a test run cannot quietly produce something that looks like a release attestation.

Expand Down Expand Up @@ -103,7 +116,7 @@ A **daily re-verification job** re-runs the same suite against the latest releas
### Workflow topology

```
publish.yml (tag) ──┐
publish.yml (main) ──┐
├──> attest.yml (workflow_call, isolated signer)
release.yml (tag) ──┘ │
├─ image index -> provenance
Expand Down Expand Up @@ -147,7 +160,7 @@ The contract lives in YAML, and the failure mode is silent: a signing step delet
## Rationale

- **Exact identity over regexp** is the single highest-value decision here. Every other gap is a missing artifact, which is visibly missing. A too-permissive verification command is an artifact that appears present and correct while asserting less than the reader believes.
- **Reusable workflow** improves the security property (signing isolated from caller-defined build steps) and simplifies the consumer contract at the same time. Those usually trade against each other. It does not by itself reach Build L3 — see decision 4.
- **Reusable workflow** makes provenance **origin** fields unforgeable by the build process (Fulcio names `attest.yml`; origin fields are minted from trusted context) and collapses every artifact onto one pinnable identity. That is the corrected L2 claim in decision 4. It does **not** by itself provide builder isolation, and a same-repo `uses: ./…` boundary additionally depends on branch protection over `attest.yml` for `main` and on tag rulesets over `v*` for the release identity — see decision 4.
- **Per-platform SBOM subjects** follow from what an SBOM is. Getting this wrong is not a policy choice, it is a category error, and it is already shipping.
- **Verify what we produce** costs one job and converts a class of silent failure into a red release. Attestations nobody checks are decoration.
- **Signing the SBOMs** closes the gap that remains after everything else is signed, at the cost of a few more bundles.
Expand All @@ -158,7 +171,7 @@ The contract lives in YAML, and the failure mode is silent: a signing step delet

- Every released artifact answers "who built this, from what source, containing what," with one command and one pinned identity.
- The multi-platform SBOM defect is fixed, and the fail-closed digest checks prevent it from recurring silently.
- Provenance is SLSA Build L2 with a single pinnable builder identity, which is a real improvement over no provenance at all. L3 remains available as a follow-on and is not foreclosed by anything here.
- Provenance is SLSA Build L2 **per artifact** (image, Helm chart, `nvcrectl` binaries, and `installer` today), with a single pinnable attestor identity (`attest.yml`). The level is only checkable when verification pins that workflow. L3 remains available as a follow-on — per artifact, with the same-repo caveat and the deferred-path costs in decision 4 stated — and is not foreclosed by anything here.
- Admission controllers can enforce the same contract the documentation publishes, so install-time and runtime checks cannot drift.
- Post-publication tampering has a bounded detection window instead of depending on a user noticing.

Expand Down
27 changes: 27 additions & 0 deletions docs/operations/verifying-artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,33 @@ built on it reports success for an artifact that was never released. The exact f
the workflow **and** the tag, so a signature from `v0.1.0` cannot pass as `v0.2.0`, and a
branch build cannot pass as either.

## Build Level, and why the pin matters

Provenance is **SLSA Build L2 per artifact** today — the `manager` image index, the Helm
chart, each `nvcrectl` binary, `installer`, `THIRD_PARTY_NOTICES.md`, and the standalone
`nvcrectl-*.cyclonedx.json` SBOM release assets alike. The project does not publish a
single project-wide level, and it does not claim L3: builder isolation is still missing
(the build runs in the caller — for images, inside `build-image.yml` invoked by the
top-level orchestrator; `attest.yml` attests a digest it is handed). What L2 here does
claim is that provenance **origin** fields are unforgeable by that build process — Fulcio
names `attest.yml`, origin fields come from trusted `GITHUB_*` context inside that
workflow, and a guard refuses to list `attest.yml` as the builder.

The same-repo reusable-workflow form (`uses: ./.github/workflows/attest.yml`) isolates
attestation from the caller's build steps, not from write access to the repository. For
`main` (dev-image) pushes the practical control is branch protection over `attest.yml`;
for the release identity pinned above, it is tag protection / repository rulesets over
`v*` tags.

**Pinning the identity above is the check that makes the level observable.** Drop the
`--certificate-identity` flag (or replace it with a loose regexp) and verification can
still go green while proving less than Build L2: you no longer know the attestation was
minted inside the reusable workflow. The published commands keep the exact pin. The
gate-test enumeration that backs this claim lives in
[SECURITY.md](https://github.com/NVIDIA/cluster-readiness-engine/blob/main/SECURITY.md#supply-chain) (kept in one place so renaming a test
cannot leave two pages asserting enforcement by a name that no longer exists); this page
links rather than duplicates it.

## Verifying the container image

The image is a multi-platform index. Two different things are attested, to two different
Expand Down
Loading
Loading