fix(ci): stop masking release.yml's safety-evidence pipeline with || true - #66
Merged
Merged
Conversation
…true release.yml's entire safety-evidence regeneration pipeline was masked with `|| true` on every step — gofusa check/cyber/tara/verify/ coverage/boundary/sci, plain `go test`, and govulncheck — so none of them could fail the workflow before their output was committed to main at every tagged release. A tool crash, malformed evidence file, or genuine coverage/compliance regression would silently commit stale or broken evidence under a new version tag while the release still looked green. Root cause: only gofusa iec61508/iso26262/iec62443 have a legitimate, previously-documented reason to tolerate a non-zero exit (they exit 1 when compliance gaps are found, which is expected for a SEOOC — commit 23e2407 / PR #23 masked exactly these three, deliberately). A later commit (9af80ce, the v1.10 conformance PR) broadened the masking to every other step in the file with no corresponding justification — that overreach is what this fixes. Verified locally with the pinned v0.47.0 gofusa binary that every now-unmasked command exits 0 cleanly against the current tree: fmea, check, cyber, tara, verify, go test, coverage, boundary, sci, safety-case, release, govulncheck. Left `|| true` in place only on the three compliance gap-report commands, with a comment explaining why (iec62443 currently has 1 real, unrelated gap and correctly exits 1 — the job must not fail on that). Closes #53 Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
`release.yml`'s entire safety-evidence regeneration pipeline was masked with `|| true` on every step — `gofusa check/cyber/tara/verify/coverage/boundary/sci`, plain `go test`, and `govulncheck` — so none of them could fail the workflow before their output was committed to `main` at every tagged release.
Root cause
Only `gofusa iec61508/iso26262/iec62443` have a legitimate, previously-documented reason to tolerate a non-zero exit: they exit 1 when compliance gaps are found, which is expected for a SEOOC (Safety Element Out Of Context) that doesn't claim full standalone conformance. Commit `23e2407` (PR #23) masked exactly these three, deliberately, with a clear commit message explaining why.
A later commit (`9af80ce`, the RELAY v1.10 conformance PR) broadened the masking to every other step in the file with no corresponding justification. That overreach — not a genuinely flaky check — is what this fixes.
Fix
Remove `|| true` from every step except the three compliance gap-report commands, which keep it with a comment explaining why (`iec62443` currently has 1 real, pre-existing, unrelated gap and correctly exits 1 — that job step must not fail the workflow on it).
Verification
Ran the pinned v0.47.0 `gofusa` binary locally, simulating the exact release.yml step sequence with `set -e` (so any failure would abort immediately): `fmea`, `check`, `cyber`, `tara`, `verify`, `go test`, `coverage`, `boundary`, `sci`, `safety-case`, `release`, `govulncheck` — all exit 0 cleanly against the current tree. The three gap-report commands were run separately: `iec61508`/`iso26262` exit 0 (0 gaps), `iec62443` exits 1 (1 gap, as expected — this is why it keeps `|| true`).
`release.yml` only runs on tag push so this PR's own CI won't exercise it directly; verification was done by replicating its steps locally against the pinned tool version.
Closes #53