feat(changelog): commit an auditable evidence ledger for filtered sources - #116
Merged
Conversation
…rces The generator already built per-group exclusion detail, but only inside `--preview`, into a tmpdir. The weekly cron run discarded it and left the PR body with reason counts alone, so a reviewer could not tell whether a real change had been dropped. Commits skipped by author never reached a group at all, so they were missing from every count. Every run now builds an evidence ledger and commits it to `scripts/changelog/audit/<n>.json`, where it outlives the Actions artifacts. Each exclusion carries a confidence: heuristic reasons (regex, commit type, missing changed files) go in an open "Needs review" block in the PR body, structural ones fold away. The ledger reconciles collected commits against logical changes, eligible and excluded groups, records author skips separately, and reports a non-zero `unaccounted` when a group vanishes between grouping and classification. Repository visibility is now explicit in the config and decides redaction: private sources keep their links, SHAs, and file counts but not their titles, subjects, or paths, and an unconfigured repository is treated as private. The same rule gates the model's own discard list.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#112 reported
non_material_docs: 32and nothing else, so there was no way to tell whether a real change had been dropped. The detail a reviewer needs was already being computed, then thrown away:generate-changelog-draft.ts:1341-1350createPreviewWorkspace(gated onoptions.preview)model-output.jsondiscardedItemswith reasonsCommits dropped by
shouldSkipCommitAuthornever reached a change group either, so they were absent from every reason bucket with nothing reconciling the totals.What changed
The ledger is built on every run and committed to
scripts/changelog/audit/<n>.json. Actions artifacts expire, so the committed file is what makes an exclusion answerable months later. It is also uploaded as a run artifact and rendered into the run log on a quiet week, when no PR exists to carry it.Exclusions are split by how much a reviewer has to trust them. Heuristic reasons (
non_material_docs,routine_maintenance,internal_only,benchmark_maintenance,non_recipe_change, plusmissing_changed_filesbecause it means lost evidence) come from regexes and commit types, so a real change can hide behind them. They get an open Needs review block in the PR body, capped at 10 with an explicit overflow line rather than a silent truncation. Structural reasons fold into a<details>.The reconciliation is stated, so a silent drop is visible:
Redaction is driven by explicit repository visibility. Five of six monitored repos are public;
0xnenlabs/steelandsteel-dev/infraare not, and public-repo artifacts and job logs are publicly downloadable, so there is no "keep it in CI" option. Private sources keep their links, SHAs, and file counts but not their pull request titles, commit subjects, or file paths. An unconfigured repository is treated as private, so a newly monitored source cannot leak before someone declares it. The same rule gates the model's discard prose: text survives only when every reference behind it is public.A redacted heuristic row still tells a reviewer where to look:
Testing
15 new cases in
tests/changelog-v2.test.tscovering confidence classification, redaction in both directions, the unknown-repo default, reconciliation including the injected-drift case, the needs-review cap, and the rendered section layout. The preview CLI test now asserts the ledger records an author-skipped bot commit that never reaches a group, which is the hole that had no coverage at all.bun test tests/changelog-v2.test.ts67 pass,bun run typecheckandbun run checkclean.Unrelated and pre-existing on
main:tests/e2e/llm-endpoints.test.tsfails on the homepage logo class assertion (class="shrink-0"vsshrink-0 self-center sm:self-auto), from 9258122. Left alone.