ssegura · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026 · Oct 19, 2026
diff --git a/docs/adr/0006-event-stream-and-store.md b/docs/adr/0006-event-stream-and-store.md
@@ -1,7 +1,7 @@
 # ADR-0006: One event stream, and every interaction persisted
 
-**Status:** Accepted, amended at M1.4, M1.6, M1.7, M11.1, M9.1 and M11.3, and in #325, #327 and #334, and on 10 October 2026
-**Date:** 2026-09-11 (amended 2026-09-12, 2026-09-13, 2026-09-15, 2026-09-22, 2026-09-23, 2026-09-24, 2026-09-27, 2026-09-30, 2026-10-10)
+**Status:** Accepted, amended at M1.4, M1.6, M1.7, M11.1, M9.1 and M11.3, and in #325, #327 and #334, and on 10 and 19 October 2026
+**Date:** 2026-09-11 (amended 2026-09-12, 2026-09-13, 2026-09-15, 2026-09-22, 2026-09-23, 2026-09-24, 2026-09-27, 2026-09-30, 2026-10-10, 2026-10-19)
 
 ## Context
 
@@ -619,3 +619,21 @@ Three things follow:
 **The schema fault is `F200`.** Where this record speaks of F101 faults for replies off their schema,
 it uses the numbering of Faults 0.7.0; since Faults 0.8.0 that fault is `F200` (ADR-0014's and
 ADR-0016's amendments).
+
+## Amendment (19 October 2026): replies a check could not judge
+
+**`RunEvent` gains `ReplyNotJudged`: a reply some checks could not judge, and which, each with the
+reason it gave.** The listener that runs the checks announces it after all of them have judged the
+reply, once for the reply however many checks could not judge it, and not at all for a reply every
+check could judge. Both reports count it, in one place for both, and say how much of the run could
+not be checked ([ADR-0014](0014-response-conformance.md)'s amendment of the same day). The stored
+run does not keep it: judging the stored replies again gives it again.
+
+It is announced on the one thread that delivers everything, and on some APIs nearly every reply
+brings one: 287,676 in a minute against restcountries, whose document gives no shape for any reply.
+Each is small, and every listener but the two reports passes it over in one comparison. Measured on
+19 October 2026 on swagger-petstore, restcountries and rest-scs, three runs of a minute each before
+and after, the same seed for each pair, on battery power: the run sent on average 2% to 4% fewer
+requests, the nine pairs differing by between −21% and +16%, four up and five down, and the time
+with no request in flight stayed between 0.5% and 0.8% of the run. Three runs do not tell that
+apart from what two runs of the same build differ by.
diff --git a/docs/adr/0014-response-conformance.md b/docs/adr/0014-response-conformance.md
@@ -1,6 +1,6 @@
 # ADR-0014: A reply is judged against the specification document itself, by an off-the-shelf validator
 
-**Status:** Accepted, amended at M1.8 and on 10 and 17 October 2026
+**Status:** Accepted, amended at M1.8 and on 10, 17 and 19 October 2026
 **Date:** 2026-09-13
 
 ## Context
@@ -193,3 +193,70 @@ through a reference that does not resolve is dropped from the canonical model wi
 would otherwise look undeclared. A reply the parser drops on its own — written as a word rather than
 an object — leaves only a warning about the document, naming no operation, and does look
 undeclared; [the known limitations](../known-limitations.md#what-a-run-says) say so.
+
+## Amendment (19 October 2026): what could not be checked is counted and said
+
+**The gap the M1.8 amendment left is closed: a run counts, check by check, the replies it could not
+judge and why, and says so in its summary and in `report.json`. The exit code does not change.**
+
+### What counts
+
+Every reply is judged by at least one check — whether it is a 500 is never in doubt — so "replies no
+check could judge" would always be none and say nothing. What a reader needs is, for each check,
+how many of the replies it applies to it could not judge, and why. Three checks can be unable to;
+the others decide from the status code or from what was sent, and always can.
+
+- **The check of a reply's shape (`F200`)** counts a reply it would have compared with a shape and
+  could not, for one reason each, the first that applies, in this order: the document describes the
+  reply only in words, or names its media type and gives no shape (`noShapeDeclared`); the
+  document could not be kept, or the shape cannot be reached in it — another file, nothing, a
+  circle (`shapeUnusable`); the reply is not JSON though the document gives a shape for it
+  (`notJson`); the body was cut short (`cutShort`); the reply names a character set other than UTF-8
+  (`notUtf8`); the checker cannot be built for the shape, or cannot finish with it
+  (`shapeUnusable`); a hidden key made the body something that is not JSON (`keyHidden`). A reply
+  that got no answer, has no body, or answers with a status code or content type the document does
+  not declare is not counted: there is nothing to judge, or another check reports it.
+- **The check of a status code (`F920`)** counts a reply to an operation that declares no reply at
+  all (`noReplyDeclared`), and, on an operation read only in part, a status code not among those
+  it read (`readInPart`). A code it did read is held to as on any other operation: only what looks
+  undeclared may be declared in the part it could not read.
+- **The check of a content type (`F921`)** counts a status code declared with no content type
+  (`noContentTypeDeclared`) and, on an operation read only in part, a type not among those it read
+  (`readInPart`).
+
+Nothing a check reports changes: only what it says about a reply it reports nothing about.
+
+**The shape of a reply whose status code or content type the document does not declare is not
+counted.** It was weighed and set aside by RESTest's maintainer: `F920` and `F921` report such a
+reply, and counting it again would say the same thing twice. The cost is stated rather than hidden:
+with either of those checks switched off, nothing says those replies went unchecked, and the known
+limitations say so.
+
+### Where it is said
+
+- **The summary**, after the verdict and before the operations that could not be tested, one line
+  for each check and reason, most replies first: `the shape of 287676 replies, on 22 operations: the
+  document gives no shape`. Nothing is printed when every reply was checked.
+- **`report.json`**, under `notChecked`: one row for each check and reason, with its code, the
+  reason in one word, the phrase the screen prints, the count, and every operation the replies came
+  from. An empty list when every reply was checked. The words for the reasons are a closed list in
+  RESTest's code, and a test holds the documentation to it.
+
+How it gets there: a check that can be unable to judge gives a verdict — what it found wrong, or why
+it could not judge — inside the module of checks, and the listener that runs the checks announces one
+event for a reply that some checks could not judge, naming each. The interface every check
+implements is unchanged, so a check written elsewhere stays one method to write. A check that throws
+is still RESTest's own failure, counted as before and ending the run with `4`, never a reply it could
+not judge.
+
+### Why the exit code does not change
+
+A run's `0` still means that no fault was found in what the run could check. Making it mean "and
+everything was checked" would change what `0` means, which this project's command line has promised
+not to do within a major version ([ADR-0015](0015-command-line-contract.md)); and it would end
+almost every run with something else, since most documents leave some reply without a shape: of
+fifteen APIs run for a minute each on 19 October 2026, eleven had replies some check could not
+judge, and on two of them, restcountries and rest-scs, not one reply's shape could be checked.
+Whether a run may be told to fail when something could not be checked is not decided here.
+Schemathesis, the tool closest to this, prints what it could not validate as a warning that fails a
+run only when asked.
diff --git a/docs/adr/0015-command-line-contract.md b/docs/adr/0015-command-line-contract.md
@@ -1,7 +1,7 @@
 # ADR-0015: One command, a time budget spent in full, and an exit code that means something
 
-**Status:** Accepted, amended at M1.7, M1.8, M2.7a, M2.10a, M11.1, M9.1, M11.3, M12.1a, M12.1c and M12.1b, in #314 and #344, and on 1, 11 and 12 October 2026
-**Date:** 2026-09-14 (amended 2026-09-15, 2026-09-18, 2026-09-22, 2026-09-23, 2026-09-30, 2026-10-01, 2026-10-11, 2026-10-12)
+**Status:** Accepted, amended at M1.7, M1.8, M2.7a, M2.10a, M11.1, M9.1, M11.3, M12.1a, M12.1c and M12.1b, in #314 and #344, and on 1, 11, 12 and 19 October 2026
+**Date:** 2026-09-14 (amended 2026-09-15, 2026-09-18, 2026-09-22, 2026-09-23, 2026-09-30, 2026-10-01, 2026-10-11, 2026-10-12, 2026-10-19)
 
 ## Context
 
@@ -1286,3 +1286,17 @@ describes the machine given, which still answers first.
   directory than that address names. That is not the disagreement the earlier amendment ruled out:
   the run was never going to use the first address without `--url`, and it uses it as soon as it
   can.
+
+## Amendment (19 October 2026): replies a run could not judge
+
+A run now counts the replies each check could not judge, and why, and says so in its summary and in
+`report.json` ([ADR-0014](0014-response-conformance.md)'s amendment of the same day). **The exit codes
+do not change**: a reply a check could not judge is not a fault, so a run whose only such replies
+were of that kind still ends with `0`, and `0` still means that nothing was found wrong in what could
+be checked.
+
+What this does not fix is the case the amendment fixing what `--url` throws away left open: a run
+in which every request is refused still reads much like one pointed at the wrong address. A refusal
+is a reply the checks judge — a `404` the document declares is fine, one it does not declare is
+`F920` — so counting what could not be judged does not tell an API that refuses everything from an
+address where the API is not. That case is still open.
diff --git a/docs/command-line.md b/docs/command-line.md
@@ -145,7 +145,7 @@ The number the command ends with, which is what a build server or a script acts
 
 | Code | Meaning |
 |---|---|
-| `0` | The run finished and found nothing wrong. `--print-campaign`, `--print-settings`, `version` and `--help` answer `0` too |
+| `0` | The run finished and found nothing wrong, in what it could check: the summary and `notChecked` in report.json say what it could not. `--print-campaign`, `--print-settings`, `version` and `--help` answer `0` too |
 | `1` | The run finished and found at least one fault in the API |
 | `2` | The command line was wrong, or a plan, a settings file or a key it named could not be used. Nothing was sent. A file of arguments that cannot be read is one of these |
 | `3` | Nothing was tested: the document describes no operation that can be tried, there is no address to send requests to or nowhere to write the results, or not one request was answered - none could be built, the budget ran out before the first, or nothing at the address replied. The run says which. A script that starts the API and RESTest together meets the last one when RESTest is quicker |
@@ -155,7 +155,10 @@ The number the command ends with, which is what a build server or a script acts
 
 A fault found is `1` rather than `0` because the usual use is a gate that should go red when the API
 is broken. A check switched off reports no fault, so a run's `0` says that nothing was found among
-the kinds of fault whose check was on ([the faults](faults.md#switching-a-check-off)). An operation
+the kinds of fault whose check was on ([the faults](faults.md#switching-a-check-off)), in the replies
+those checks could judge: a reply a check could not judge, such as one whose document gives no
+shape for its body, is no fault, and the summary and `notChecked` in report.json say how many there
+were ([what a run leaves behind](report.md#notchecked)). An operation
 RESTest had to skip is not an error: a run says what it skipped and carries on, and only a run in
 which nothing at all was tested answers `3`. A run stopped from outside never answers `0` or `1`, so
 that one stopped half-way never passes for one that finished.

diff --git a/docs/faults.md b/docs/faults.md
@@ -97,16 +97,30 @@ handed over, so everything the document can say counts — shapes built from oth
 between shapes, references from one to another — and the details lines under the fault say which
 part of the body disagreed and how.
 
-Nothing uncertain is reported. The check ends quietly, and reports nothing, when:
-
-- the document says nothing about what that operation returns with that status code or that media
-  type;
-- the reply does not say it is JSON — `application/json`, or a media type ending in `+json` — or
-  has no body, or says it is written in a character set other than UTF-8;
-- the body was too large to keep whole, so there is no whole body to judge;
-- the disagreement could have been caused by a key the run handed over with `--auth` being hidden in
-  the reply rather than by the API;
-- the shape the document gives is one the checker cannot finish judging.
+Nothing uncertain is reported. The check reports nothing when it has nothing to judge — a reply with
+no body, or one whose status code or content type the document does not declare, which `F920` and
+`F921` report — and when it cannot judge what it has. In the second case it says so: the run counts
+the reply under `notChecked` in `report.json`, with the reason in brackets below, and its summary
+says how many there were ([what a run leaves behind](report.md#notchecked)). It cannot judge a reply
+when, taking the first of these that applies:
+
+1. the document describes the reply only in words (`noShapeDeclared`);
+2. the document could not be kept to check against (`shapeUnusable`);
+3. the document names the reply's media type and gives no shape for it (`noShapeDeclared`);
+4. the shape the document gives cannot be reached — it points at another file, at something the
+   document does not define, or round in a circle (`shapeUnusable`);
+5. the reply is not JSON — `application/json`, or a media type ending in `+json` — though the
+   document gives a shape for it (`notJson`);
+6. the body was too large to keep whole, so there is no whole body to judge (`cutShort`);
+7. the reply says it is written in a character set other than UTF-8 (`notUtf8`);
+8. the checker cannot be built for the shape the document gives, a pattern it cannot read, say
+   (`shapeUnusable`);
+9. a key the run handed over with `--auth` was hidden in the body and it could then not be read as
+   JSON (`keyHidden`);
+10. the checker cannot finish judging the body against the shape (`shapeUnusable`).
+
+Where a key was hidden and the body is still JSON, the reply is judged, and only what the hiding may
+have caused is let pass.
 
 A property the document marks `writeOnly` is one the API is sent and never hands back, so a reply
 without it is not wrong for that, even where the document calls it required.
@@ -254,12 +268,17 @@ and `default` included.
 - **What is left alone.**
   - A server error, 500 and up: `F100`, `F205` and `F121` report those, so no reply is counted
     twice over its status code.
-  - An operation whose document declares no reply at all, since it has promised none.
-  - An operation RESTest could not read in full: a reply the document declares, and RESTest could
-    not read — one that points at a shape the document never defines, say — would otherwise look
-    undeclared. The run lists what it could not read before it starts. A reply the library RESTest
-    reads documents with drops on its own, such as one written as a word rather than an object, is
-    not known to be missing, and does look undeclared ([known
+  - An operation whose document declares no reply at all, since it has promised none. The run
+    counts each such reply as one this check could not judge (`noReplyDeclared`, under
+    `notChecked` in `report.json`), unless the operation is one of those below, read in part,
+    which comes first (`readInPart`).
+  - On an operation RESTest could not read in full, a status code it could not find among what it
+    read: a reply the document declares, and RESTest could not read — one that points at a shape the
+    document never defines, say — would otherwise look undeclared. The run lists what it could not
+    read before it starts, and counts each such reply as one this check could not judge
+    (`readInPart`). A status code it did read is held to as on any other operation. A reply the
+    library RESTest reads documents with drops on its own, such as one written as a word rather than
+    an object, is not known to be missing, and does look undeclared ([known
     limitations](known-limitations.md#what-a-run-says)). An operation RESTest only had to call by
     its method and path, because the document gives its name to an operation RESTest reads first,
     is not one of these: the new name costs it nothing else.
@@ -294,12 +313,16 @@ F921  A Content Type The Document Does Not Declare
   in the document where the API sends it.
 - **What is left alone.**
   - A reply with no body, whatever type it names: an answer to `HEAD`, a `204`.
-  - A status code the document does not declare, which is `F920`'s, and one it declares with no
-    content type at all, even when a body comes: the document described that reply only in words.
+  - A status code the document does not declare, which is `F920`'s.
+  - A status code it declares with no content type at all, even when a body comes: the document
+    described that reply only in words. The run counts each such reply as one this check could not
+    judge (`noContentTypeDeclared`), unless the operation was read in part, below, which comes first
+    (`readInPart`).
   - A document that declares `*/*`, which promises no type in particular — and is what a Swagger 2.0
-    document that says nothing about types is read as.
-  - A server error, an operation RESTest could not read in full, and a reply to a body sent under a
-    media type the operation does not accept, as for `F920`.
+    document that says nothing about types is read as, so such a reply counts as checked.
+  - A server error, and a reply to a body sent under a media type the operation does not accept, as
+    for `F920`; and, on an operation RESTest could not read in full, a type it could not find among
+    what it read, counted as for `F920` (`readInPart`).
 - **What the API wrote is quoted cut short.** The fault quotes the content type as the API sent it,
   cut at 100 characters, with anything that would move a terminal's cursor or change its colours,
   and invisible characters that would reorder or hide what is shown around them, replaced by `?`.
@@ -426,3 +449,9 @@ in which nothing at the address answered ends with `3`, even when it reported re
 reply in time: something that takes connections and never speaks HTTP does exactly that, so the run
 says how many such requests there were beside the address.
 [The exit codes](command-line.md#exit-codes) has every number a run can end with.
+
+A `0` covers what the run could check. Replies a check could not judge — a body the document gives
+no shape for, say — are not faults and do not change the exit code; the summary says how many there
+were, and `notChecked` in `report.json` says which and why ([what a run leaves
+behind](report.md#notchecked)). A build that should know whether everything was checked reads that
+list, which is empty when it was.