Skip to content

fix(sbom, vex, build): keep artifacts with the image in every repository - #282

Draft
reyreavman wants to merge 6 commits into
mainfrom
test/cleanup/final-repo-sbom-lifecycle
Draft

fix(sbom, vex, build): keep artifacts with the image in every repository#282
reyreavman wants to merge 6 commits into
mainfrom
test/cleanup/final-repo-sbom-lifecycle

Conversation

@reyreavman

@reyreavman reyreavman commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

With --final-repo, the SBOM and VEX of an image existed in exactly one copy, in the final repo: reading them from the stages repo returned nothing, a project where one final image is built from another (fromImage/import) failed with advice to enable SBOM generation that was already enabled, and for a multi-platform image the artifacts were unreachable in the final repo altogether. This PR adds the artifact locality contract (specs/020-sbom-artifact-locality), e2e tests that falsify each breakage, and the fix: artifacts live alongside the image in every registry-backed repository that holds the image.

Repro of the dependent-build failure from a clean checkout: a two-image werf.yaml (base from scratch, app with fromImage: base), build.sbom.enable: true, then werf build --repo <r> --final-repo <rf> against a clean registry fails with the image ... must have an SBOM artifact attached; to generate an SBOM for the image, rebuild it with SBOM generation enabled.

What

Convergence and lookups

  • SBOM and VEX converge into the repository the image was built in (the stages repo), regardless of --final-repo; publishing elsewhere can no longer move where artifacts are written.
  • A build of dependent final images (fromImage and import) with --final-repo on a clean registry succeeds, and the dependent image's SBOM contains the base image's components.
  • The build report still references the final repo when --final-repo is used; werf stage image still prints the stages repo reference.

Placement and propagation

  • After a build with --final-repo, werf sbom get and werf attest ls/get/verify return the artifacts against both the stages repo and the final repo.
  • For a multi-platform image, each platform manifest digest carries its own SBOM in both repositories; image-level artifacts (VEX) sit on the index digest; werf sbom get --repo <final-repo> --digest <index-digest> --platform <p> returns that platform's SBOM. Previously both platform SBOMs were copied onto the index digest, where the second displaced the first.
  • Propagation runs on every build and is idempotent: a repository holding the image without its artifacts is repaired by the next successful build, so the registry state no longer depends on build history.

Copy paths

  • Stage copies between repositories (final repo, cache repos) carry the artifacts of every manifest an image index references, and repair a destination that already holds the manifest without its artifacts.
  • werf stages copy and bundle copy carry the attached artifacts of the images they transfer.
  • The backend-mediated secondary-to-primary copy carries artifacts only when the image digest survived the copy; otherwise nothing is attached whose payload names a different digest, a warning names what was left behind, and werf-generated artifacts are regenerated by convergence in the same run.
  • Meta repo migration moves only metadata records, which carry no artifacts; unchanged.

Cleanup

  • Cleanup and purge are not changed: they were verified sound (live images protected, only orphaned artifacts collected). The retention e2e pins this.

Verification status

  • VERIFIED locally against a live registry (macOS, Docker): dependent images (fromImage+import, both docker backends) and multi-platform placement — red on main, green with the fix.
  • UNVERIFIED locally: the cleanup retention e2e needs the amd64-only builder base and runs only in CI.

Why

Two unintended regressions inside PR #225 produced the single-copy state. 55dc5ea4b (content-based tag) stopped storing the final repo descriptor where SBOM propagation read it, silently killing the copy into the final repo. a891ed411 (nil-panic fix) then routed convergence through a shared accessor that prefers the published descriptor, moving the single remaining copy from the stages repo into the final repo. The root defect is one descriptor answering two questions: "the image as published" (what the report needs) and "the image being described" (what convergence and the base/import SBOM lookups need). The fix splits them: contentTagDesc always describes the image in its build repository, the new finalContentTagDesc carries the published final repo descriptor.

The alternative — teaching cleanup to protect the single final-repo copy, as the original review card suggested — was rejected: cleanup was never the problem (it already protects live images), and a single copy stays fragile no matter how careful the cleanup is. The contract and its rationale live in specs/020-sbom-artifact-locality/spec.md.

werf cleanup deletes stages from the final repo and then collects orphaned
sha256-* artifact indexes there, but no e2e run exercised that path against a
live SBOM. The existing final-repo test stops right after the build, and the
cleanup case in the VEX suite asserts deletion in the stages repo, so a
regression that dropped an in-use SBOM from the final repo would go unnoticed.

Build a project with --final-repo, read the SBOM back from the final repo and
from the stages repo, run cleanup twice with the built commit reachable from
origin so retention policies hold the stage, and read the SBOM again after
each run.

Address the stages repo by its stage tag rather than by the digest from the
build report, because copying a stage into the final repo may change the
manifest digest. Keep the bare remote outside the work tree: the fixture adds
the whole project directory to the image, so an in-tree remote breaks
giterminism.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
@reyreavman

Copy link
Copy Markdown
Collaborator Author

Verification

  • Ran the new entries against a local registry:2 with REGISTRY_STORAGE_DELETE_ENABLED=true: task test:e2e paths="./test/e2e/sbom/..." labelFilter="final-repo && cleanup" parallel=1. Two defects in the test itself surfaced this way and are fixed in the diff: the in-tree bare remote failing giterminism, and stage image output needing --log-quiet plus last-line parsing because the test helper returns combined stdout+stderr.
  • Not run: a green pass. The host is arm64 and the fixture's builder base (registry.deckhouse.io/container-factory) is amd64-only, so the build stops at image ... does not match the specified platform (linux/arm64/v8). The pre-existing final-repo entries fail identically on this host, so the blocker is the machine, not the new test.
  • Mutation: not run, for the same reason. The mutation to try once CI is green: make cleanupFinalStages delete unconditionally instead of protecting stages found in the stages repo, and confirm the first sbom get from the final repo after cleanup fails.

Review focus

  • The retention precondition. The test asserts the SBOM survives, so it only means something while the image is genuinely in use — that rests on the commit being reachable from origin/main. If werf's git-history retention treats this setup as unreachable, every entry would still pass for the wrong reason: nothing would be there to delete. Worth confirming against the cleanup log that the stage was evaluated and kept, not skipped.

Follow-up

  • File a bug: with --final-repo, the VEX artifact is published only into the stages repo. vexStep.Converge targets the repository from the stages stage descriptor, and convergeVexByImagesSets runs after the SBOM propagation step, so nothing carries VEX into the final repo.
  • File a bug: for multi-platform builds the SBOM is not propagated into the final repo. PropagateArtifacts reads the final stage descriptor from the per-platform stage image, which is only set on the single-platform path, so the copy is skipped.
  • Extend this suite with the equivalent VEX retention case once the first item is fixed; asserting it today would only lock in the missing artifact.

…final repo

Two gaps around --final-repo had no coverage.

A project where one image is the base of another (fromImage) or imports
files from another exercises the base/import SBOM lookup during
convergence of the dependent image. If that lookup resolves against a
repository that does not hold the base SBOM, the build fails with advice
to rebuild with SBOM generation enabled, which cannot help. Cover both
dependency kinds: build with --final-repo against a clean registry, read
the merged SBOM from the final repo, and check a rebuild serves both
SBOMs from cache instead of regenerating them.

For a multi-platform image the SBOMs belong on the platform manifest
digests, never on the index digest, and the registry-level index copy
into the final repo preserves the platform manifest digests. Assert each
platform manifest carries its SBOM in both the stages repo and the final
repo, the index digest carries none in either, and sbom get resolves a
platform SBOM from the final repo through the index digest reported to
the user. The stapel fixture builds for both platforms on the Docker
backend with WERF_EXPERIMENTAL_STAPEL_ARM, following the multi-platform
signing suite.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
…ookup

Rebuild the dependent-images fixtures from scratch instead of the trusted
builder base. An image derived from the builder base inherits the
io.deckhouse.internal.builder label, and with
WERF_E2E_ALLOW_LOCAL_BUILDER_IMAGES a failed base SBOM lookup silently
degrades into ErrSbomNotRequired while the asserted base packages surface
through the filesystem scan — the previous entries passed without
exercising the lookup at all. The scratch fixtures leave no escape: the
fromImage entries now fail on current main, reproducing the broken base
SBOM lookup under --final-repo, and need no builder base or external
environment beyond the registry.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
Add specs/020-sbom-artifact-locality: SBOM, VEX and other attached
artifacts live alongside the image in every registry-backed repository
that holds the image. The spec states the contract over pairs of a
repository and a subject digest, fixes the convergence target to the
repository the image was built in, requires every stage copy to carry
attached artifacts with repair on repeat, defines the digest-preservation
rule for backend-mediated copies, and records the root cause of the
current breakage: two unintended regressions inside PR #225 that first
killed the propagation step and then moved the single remaining copy
into the final repo.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
With --final-repo the SBOM and VEX of an image existed in exactly one
copy, in the final repo: publishFinalImage overwrote the image's content
tag descriptor with the final repo one, artifact convergence resolved its
target through that descriptor, and the propagation step into the final
repo had been dead since the descriptor stopped being stored where the
step read it. Reading the SBOM from the stages repo returned nothing, a
project where one final image is built from another failed with advice
to enable SBOM generation that was already enabled, and for a
multi-platform image the artifacts were copied onto the index digest —
where no retrieval path looks and where the second platform's copy
displaced the first.

Split the two roles of the content tag descriptor: contentTagDesc keeps
describing the image in the repository it was built in and is the single
target for artifact convergence and for the base and import SBOM
lookups, while the new finalContentTagDesc carries the published final
repo descriptor for the build report. Replace the per-platform SBOM
propagation with one step that runs after both SBOM and VEX convergence
and carries every attached artifact kind: for a single-platform image
onto the published final descriptor, for a multi-platform image onto the
platform manifest digests preserved by the registry-level index copy,
with image-level artifacts following the index digest. The step runs on
every build and the copies are idempotent, so a repository holding the
image without its artifacts is repaired by the next run.

Extend the same contract to the remaining copy paths: stage copies
between repositories carry the artifacts of every manifest an index
references and repair a destination that already holds the manifest,
werf stages copy and bundle copy carry the artifacts of the images they
transfer, and the backend-mediated secondary-to-primary copy carries
artifacts only when the digest survived the copy — a statement about the
source digest is not a statement about the destination digest — leaving
a warning about what was left behind otherwise.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
@reyreavman reyreavman changed the title test(sbom, cleanup): cover SBOM retention across cleanup fix(sbom, vex, build): keep artifacts with the image in every repository Sep 3, 2026
@reyreavman

Copy link
Copy Markdown
Collaborator Author

Verification

  • Local e2e against a live registry:2 (macOS, Docker backends): task test:e2e paths="./test/e2e/sbom/..." labelFilter="final-repo && (dependent || multiplatform)" parallel=1 — 6/6 specs, red on main before the fix, green after. This is the mutation evidence for the new tests: each suite demonstrably distinguishes the broken state from the fixed one.
  • The original dependent-images entries were green on main for the wrong reason: the fixture's base image inherited the io.deckhouse.internal.builder label from the trusted builder base, and WERF_E2E_ALLOW_LOCAL_BUILDER_IMAGES degraded the failed base-SBOM lookup into a silent skip, while the asserted base packages surfaced via the filesystem scan rather than the merge. The fixtures were rebuilt from scratch images, which leaves no escape.
  • Multi-platform placement was pinned by registry inspection of a local reproduction: on main, the final repo held one artifact on the index digest (linux/arm64 — the second platform's copy displaced the first via the type+image-name replace key), platform manifest digests held none in the final repo, and the stages repo was correct.
  • Full unit suite green (72 suites). Scoped task lint:golangci-lint on every touched package: 0 issues.
  • Not run: full-repo task lint:golangci-lint — fails on this host with a pre-existing pkg/deno/embed_darwin_arm64.go: no matching files found typecheck (missing local embed asset, reproduced on a clean tree without these changes).
  • Not run locally: the cleanup retention e2e (final-repo && cleanup) — its fixture needs the amd64-only builder base; CI settles it. It read the SBOM from the final repo and failed against the stages repo on main; with the fix both reads must pass.

Review focus

  • The multi-platform propagation assumes the registry-level index copy into the final repo preserves platform manifest digests, and attaches per-platform artifacts digest-to-digest on that basis. Verified empirically on registry:2; worth confirming there is no in-scope path that rewrites an index's children.
  • CopyFromStorage now copies attached artifacts also when the destination already holds the manifest (repair semantics). That adds a fallback-index HEAD/GET per stage copy on the already-present path — mostly cache refills. If that shows up in build timings, the guard belongs in CopyAllAttachedArtifacts, not in the callers.
  • The secondary→primary digest-compare rule leaves user-signed attestations behind when the digest changed, with a warning. The wording of that warning is user-facing; check it reads right in a real log.

Follow-up

  • Re-run the full e2e matrix in CI (this PR): retention/cleanup suite is the one local runs cannot cover.
  • Extend the VEX e2e suite with a final-repo read-back case, mirroring test/e2e/sbom/final_repo_test.go — VEX now travels with propagation, but no e2e asserts it from the final repo yet.
  • Enable the pending multi-platform SBOM entries (native-chroot/native-rootless) once SBOM generation works on Buildah.
  • Consider a werf.io-facing note: with --final-repo, artifacts are now present in both repositories; registry storage grows accordingly.

Final repo cleanup had its protection inverted: a final stage whose stage
ID was absent from the stages repo was marked protected, while nothing
protected the ones still present, so every final stage whose counterpart
survived cleanup landed in the deletion set. One werf cleanup run against
--final-repo emptied the final repo of every live image while keeping
whatever the stages repo no longer knew about. Nothing caught it because
no test ran cleanup against a final repo.

Restore the mirror semantics: a final stage whose stage ID is still
present in the stages repo after cleanup is protected (reason "found in
repo"), the rest are deleted. Rewrite the unit test that pinned the
inverted behavior and add the deletion case.

Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant