From 6212f1fe60995dd5ee5011c228e17f3c88952dbe Mon Sep 17 00:00:00 2001 From: Arnel Robles Date: Wed, 2 Sep 2026 16:58:41 +0800 Subject: [PATCH] DELIVERY: a green check proves something about where it ran Two silent passes from the umbraco-pwa work on 2 September, same shape both times. A 4KB request body cap that nothing enforced in the suite, because RequestSizeLimit only sets IHttpMaxRequestBodySizeFeature and TestServer does not implement it, so the attribute was inert and an oversized body came back 202 while three documents stated the cap. Then a mutation check that could not see its own mutation, because gitleaks detect scans commits and the password under test was uncommitted. The scan was clean and the allowlist looked proven. Both exited zero with a real runner. The thing under test was outside the scope the runner was looking at. --- DELIVERY.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/DELIVERY.md b/DELIVERY.md index c79023e..72fb05f 100644 --- a/DELIVERY.md +++ b/DELIVERY.md @@ -342,6 +342,41 @@ The world moved; the code did not. leaving them to be noticed on the next pull request. If a job can fail without anyone pushing, it needs a route to a human that does not depend on anyone pushing. +### A green check proves something about where it ran + +A check runs somewhere. That somewhere has to be the place the property actually holds, and the +default is usually not it. Both halves of this were found in one afternoon, on the same repository, +and neither announced itself. + +**Caught:** a 4KB request body cap that nothing enforced in the suite. `[RequestSizeLimit]` does not +reject anything itself, it sets `IHttpMaxRequestBodySizeFeature`, and Kestrel is what reads it. The +integration suite runs on `WebApplicationFactory`, which serves over TestServer, and TestServer does +not implement that feature. So the attribute was inert there and an 8KB body came back `202`. Three +documents stated the cap. Any test written for it in that suite would have passed while asserting +nothing. The fix was not a better assertion, it was moving the test to the suite that starts the +real host in a child process. + +**Caught, an hour later:** a mutation check that could not see the mutation. A secret-scanning +allowlist had been widened, and widening one far enough to disable the rule looks identical to +widening it correctly, so the rule was tested by putting a real-looking password in the file. The +scan came back clean and the allowlist looked safe. `gitleaks detect` scans commits. The password +was in the working tree, uncommitted, and was never examined. `--no-git` is the flag that scans +files, and with it the rule fired immediately. + +The shape both times: the runner was real, the command exited zero, and the thing under test was +outside the scope the runner was looking at. Worth asking before trusting any green: + +- **Does this host implement the thing being asserted?** Test hosts, in-memory servers and fakes + routinely omit features the production host enforces. Limits, timeouts, TLS redirection and + middleware ordering are the usual casualties. +- **What is this tool's unit of work?** Commits, staged changes, the filesystem, one package, the + whole solution. A scanner pointed at history says nothing about your disk. +- **Did the check see my change at all?** If deliberately breaking the thing leaves the check green, + the check was never reading it, whatever else it was doing. + +The third question is the cheap one and it subsumes the others. It is also the one skipped most +often, because by then the check is already green and the work feels done. + ### Publishing is not releasing Pushing a package is one step. If the repository does not also record what shipped, the project