[ssegura]

Related: None. ADR-0014's M1.8 amendment named this gap: a run that cannot check a reply says nothing about it.

What you can do now that you could not before

A run that finds nothing wrong now says what it could not check. When a check cannot judge a reply — the document gives no shape for its body, the body is not JSON, the shape points at something the document never defines — the summary counts those replies after the verdict, by check and reason, and report.json lists them under notChecked, with the operations they came from. A minute against restcountries, whose document describes every reply only in words, used to end with its 4,872 faults and nothing else; it now also says that not one of its 287,676 replies had its body compared with a shape, and why.

Try it yourself

With only Java and git. The "API" is a stand-in of a few lines on the HTTP server that comes with Java: it answers every request 200 with a small JSON body. Its document declares two operations: /health describes its reply only in words, and /version names application/json and gives no shape for it.

git fetch origin && git switch feat/m3-6-what-could-not-be-checked
./mvnw -q install -DskipTests
mkdir -p try
cat > try/Stand.java <<'EOF'
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;

/** A stand-in that answers every request with 200 and a small JSON body. */
public class Stand {
    public static void main(String[] args) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 18766), 0);
        server.createContext("/", exchange -> {
            byte[] body = "{\"status\": \"up\"}".getBytes();
            exchange.getResponseHeaders().add("Content-Type", "application/json");
            exchange.sendResponseHeaders(200, body.length);
            exchange.getResponseBody().write(body);
            exchange.close();
        });
        server.start();
    }
}
EOF
cat > try/health.yaml <<'EOF'
openapi: 3.0.3
info: {title: Health, version: "1"}
paths:
  /health:
    get:
      operationId: getHealth
      responses:
        "200": {description: up and running}
  /version:
    get:
      operationId: getVersion
      responses:
        "200":
          description: the version
          content:
            application/json: {}
EOF
java try/Stand.java &
./restest run try/health.yaml --url http://127.0.0.1:18766 --budget 10s --seed 1

Real output, from a fresh clone of this branch; the run ends with 0:

RESTest testing Health at http://127.0.0.1:18766

2 of 2 operations can be tested, seed 1, budget 10s
  what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent

585543 requests to 2 operations in 10.0s, 3% of it idle
  opening lap: 2 requests in 93ms, 2 of 2 operations answered 2xx
  585543 2xx
no faults found
some replies could not be checked; report.json names their operations:
  the shape of 585543 replies, on 2 operations: the document gives no shape
  the content type of 292772 replies, on 1 operation: the document declares the status code with no content type
report written to restest-out/report.json (8.6 KiB)
the run itself was not kept; pass --store to keep every request and reply

The three lines after no faults found are new. restest-out/report.json (one line; laid out here for reading) says the same, operation by operation:

"notChecked": [
  {"code": 200, "id": "SCHEMA_INVALID_RESPONSE",
   "label": "F200:Schema Violation: Received A Response From API With A Structure/Data That Is Not Matching Its Schema",
   "what": "the shape", "reason": "noShapeDeclared", "why": "the document gives no shape",
   "replies": 585543,
   "operations": [{"operation": "getHealth", "replies": 292772}, {"operation": "getVersion", "replies": 292771}]},
  {"code": 921, "id": "RESTEST_UNDECLARED_CONTENT_TYPE",
   "label": "F921:A Content Type The Document Does Not Declare",
   "what": "the content type", "reason": "noContentTypeDeclared",
   "why": "the document declares the status code with no content type",
   "replies": 292772,
   "operations": [{"operation": "getHealth", "replies": 292772}]}
]

The number of requests depends on the machine. The same command on master (git switch --detach afa51e5d, then ./mvnw -q install -DskipTests again) also ends with 0, and says nothing about what it could not check:

605781 requests to 2 operations in 10.0s, 4% of it idle
  opening lap: 2 requests in 89ms, 2 of 2 operations answered 2xx
  605781 2xx
no faults found
report written to restest-out/report.json (7.9 KiB)

How it works

Every reply goes through eight checks (the faults). Five of them always reach a verdict: whether a reply is a 500, a 501 or another 5xx, whether a request broken on purpose was accepted, whether a request got no reply. Three sometimes cannot. The check of a reply's shape (F200) has nothing to compare a body with when the document gives no shape. The checks of a status code (F920) and a content type (F921) have nothing to hold a reply to when the document declares no reply, or no type, or could only be read in part. Until now each of the three said nothing in that case, so a run that could check nothing looked exactly like one that checked everything and found it fine.

Now each of the three says why, from a closed list of nine reasons: noShapeDeclared, notJson, cutShort, notUtf8, shapeUnusable and keyHidden for the shape; noReplyDeclared for the status code; noContentTypeDeclared for the content type; and readInPart for either of those two. Each check gives a reply at most one reason, the first that applies in its own order (the faults give each order). What a check reports as a fault does not change for any reply. A reply with nothing to judge, such as one with no body, is not counted, and neither is one whose status code or content type the document leaves out: F920 and F921 report those.

reply ──> the eight checks ──> faults             ──> the screen, report.json: faultsBy…, findings
                          └──> "could not judge,  ──> the screen, report.json: notChecked
                                because …"

The part of RESTest that runs the checks announces, once for each reply some check could not judge, which checks and why. Both reports count those announcements in one shared place: the screen prints one line per check and reason, most replies first, and report.json lists every operation. The exit code does not change: 0 means no fault was found in what could be checked, and the line above says what could not.

What changed in the code

• restest-core
  • oracle/WhyNotJudged, a closed enum of the nine reasons, each with the word report.json writes and the phrase the screen prints.
  • oracle/CouldNotJudge: the kind of fault a check reports, what it looks at (the shape, the status code, the content type), and why it could not.
  • RunEvent.ReplyNotJudged, carrying the reply and a non-empty list of CouldNotJudge.
  • The public Oracle interface is unchanged.
• restest-oracles
  • A package-private SaysWhyNot, which declares the rule's kind() and what() once and gives a Verdict: the faults found, or the reason it could not judge. Not-judged verdicts are made once per reason.
  • ResponseSchemaOracle, UndeclaredStatusOracle and UndeclaredContentTypeOracle implement it. Their judge is verdict(...).findings(), so the two cannot disagree. Each follows the order its Javadoc and docs/faults.md give.
  • SpecificationDocument.pointerToSchema returns a Shape: the trail to it, NONE_GIVEN or OUT_OF_REACH. ResponseSchemaOracle keeps each declared reply's Shape beside the checkers it already kept, since it now looks it up for replies that are not JSON too.
  • HeldToTheDocument no longer leaves out an operation read only in part. Each rule decides at its own step: what it read is judged, and only what it could not find is readInPart.
  • OracleListener asks the rules that can say why, announces their faults as before, and sends one ReplyNotJudged after all rules. It allocates its list only when a rule could not judge. A rule that throws is still RESTest's own failure (exit 4), never "could not judge".
• restest-report
  • NotChecked counts, by check and reason, replies and the operations they came from, in one place for both reports, as ServerErrors and Phases already do.
  • ConsoleReport prints its lines after the verdict, before the operations that could not be tested.
  • JsonReport writes notChecked[] after skippedOperations[].
  • Phases passes the event over.
• Documentation
  • docs/report.md: the new summary line; notChecked[] with a real example; its operations; and the nine reasons.
  • docs/faults.md: each check's order and what it counts; a 0 covers what could be checked.
  • docs/command-line.md and manual chapter 9: the 0 row.
  • Manual chapter 4, the glossary and docs/known-limitations.md.
  • Amendments:
    • ADR-0014 closes the M1.8 gap;
    • ADR-0015: the exit codes do not change, and a run in which everything is refused still reads much like a wrong address;
    • ADR-0006: the new event and what it costs.
• Tests
  • Every reason of every rule, and every case that is not counted, pinned in the rules' own tests.
  • The listener's single announcement.
  • Both reports' order and singulars.
  • A command-line run that ends 0 and says what it could not check.
  • DocumentedReportTest holds the reasons listed in docs/report.md to the enum, and requires a real run to write notChecked.
• Left for later
  • Whether a run may be told to fail when something could not be checked.
  • Keeping the reasons in run.sqlite: judging a stored run again would give them again.
  • Showing them in an HTML report.
  • Telling a run that everything refused from one pointed at a wrong address.

Evidence

• Tests: 27 new test methods, and the existing tests of the three rules' replies left alone now also say whether, and why, each was not judged. ./mvnw verify is green on all nine modules, 4,594 tests. CI: added below once it has run.

• Architecture tests: one added (DocumentedReportTest: the reasons on the page are exactly RESTest's), none weakened.

• Mutation score, restest-oracles: 97% (294 of 304), 98% line coverage; master 98% (249 of 255), 96%. The four new survivors are equivalent: a null returned inside Optional.map becomes the same empty result. The other two were there before.

• Per-request overhead: measured on swagger-petstore, restcountries and rest-scs, three one-minute runs of master and of this branch, alternated, with the same seed for each pair, on battery power.

  • On average this branch sent 2% to 4% fewer requests.
  • The nine pairs differed by −21% to +16%: four up, five down.
  • Three runs do not tell that apart from how much two runs of one build differ.

  API	Requests, master	Requests, this branch	Idle, both
  swagger-petstore	583,522; 489,710; 454,092	458,686; 515,364; 525,628	0.7–0.8%
  restcountries	261,543; 228,749; 236,474	233,181; 237,934; 233,624	0.5%
  rest-scs	322,347; 295,924; 318,250	291,569; 299,829; 304,109	0.5–0.8%

• Smoke run: the two pinned images, run as the smoke job runs them (10 s, seed 20260914), three runs of master and of this branch, alternated.

  • Pet shop. Both: 19 operations tested, 14–16 answering 500, 2–3 kinds of fault, 15–17 operations with faults. master 36,996–56,162 faults; this branch 35,795–57,323. This branch adds the shape of 46,529–72,080 replies, on 19 operations: the document gives no shape and the content type of 12,238–18,925, on 18.
  • Activity catalogue. Both: 49 operations, 3 answering 500, 3 kinds, 30 operations with faults. master 54,464–63,084 faults; this branch 61,852–65,395. This branch adds the shape of 92,790–98,140 replies and the content type of 89,322–94,476, both on 29 operations.
  • Every run ended with 1. One round of the activity catalogue was run again, on both builds, after its container stopped answering mid-campaign.

• Idle time: 4.4–5.4% in the smoke runs on both builds; 0.5–0.8% in the one-minute runs.

• Artefact: the fifteen Web Fuzzing Dataset APIs measured for the earlier rules of fault, run again locally, a minute each, seed 42.

  • Eleven had replies some check could not judge. Four did not (bibliothek, kafka-rest, petclinic-rest, spring-rest-example); spring-rest-example answers only status codes its document leaves out, which F920 reports, or no body at all.
  • Before writing the code, a replay of these APIs' runs of 17 October, written apart from RESTest, predicted which would have replies whose shape could not be checked and why. This run found them on the same APIs, for the same reasons, in counts of the same order (they are different runs): noShapeDeclared on nine, and notJson on rest-scs's text replies.
  • The replay missed features-service, which it predicted at 0. Its Swagger 2.0 document declares replies with no schema, which conversion writes as a media type with nothing under it; the run counted 19,153.
  • Every run ended with 1, as on master.

  API	Shape not checked	Content type not checked	Status code not checked
  swagger-petstore	391,947 on 19 ops (no shape)	155,774 on 18	—
  restcountries	287,676 on 22 (no shape)	286,159 on 22	—
  rest-scs	260,617 on 11 (not JSON), 4,848 on 11 (no shape)	4,848 on 11	—
  scout-api	106,647 on 29 (no shape)	90,358 on 29	—
  features-service	19,153 on 9 (no shape)	100 on 1	—
  user-management	7,633 on 11 (no shape)	7,591 on 11	—
  rest-ncs	3,063 on 6 (no shape)	3,063 on 6	—
  rest-news	1,365 on 2 (no shape)	1,365 on 2	—
  gestaohospital	969 on 16 (no shape), 120 on 2 (not JSON)	969 on 16	—
  market	258 on 2 (no shape)	258 on 2	—
  catwatch	124 on 4 (no shape)	124 on 4	21 on 4 (read in part)

Decisions taken

• The exit code does not change (decided with the maintainer). ADR-0015 freezes what each code means within a major version, and a new code would fail every build that waits for 0. Nearly every document leaves some reply without a shape, so a code for "not everything checked" would almost never be 0. Whether a run may be asked to fail on it is left open. Schemathesis, for comparison, shows what it could not validate as a warning that fails a run only when asked.
• The shape of a reply whose status code or content type the document does not declare is not counted (decided with the maintainer). F920 and F921 report those replies, and counting them again would say the same thing twice. The cost is that with either check switched off nothing says those replies went unchecked, and the known limitations say so.
• Counted by check, not by reply. Every reply is judged by at least the check of a 500, so "replies no check could judge" would always be zero.
• No switch. This counts and reports; it changes nothing a run sends or reports as a fault, so it is not a lever with something to turn off. The overhead above is within what two runs of one build differ by.
• The interface every check implements is unchanged. What a rule can say about a reply it could not judge is a capability inside restest-oracles. A check written elsewhere stays one method to write.
  • A reviewer pointed out that the three public rules now carry public kind(), what() and verdict(). The last returns a type only that module can read. The Javadoc says so.
• Where a rule's identity lives. The kind of fault and what a rule looks at are declared once on the rule, not repeated in every verdict. That came from the simplification pass, and so did the following:
  • not-judged verdicts are made once per reason;
  • the shape of a declared reply is looked up once per document;
  • the listener builds its list only when a rule could not judge.
• What F200 counts. It judges any reply whose status code the document declares, 5xx included, so a 500 with no shape declared counts under noShapeDeclared. All 1,517 of restcountries' 500s are in that row. F920 and F921 leave server errors to the server-error checks.
• Read in part, decided by each rule. An operation read only in part used to be left alone entirely. Now what was read is judged, and found fine, and only a status code or type not found among what was read is readInPart. Which faults are reported does not change.
• Wording. The screen's phrase is "the document gives no shape", and what is "the shape". Both read the same for one reply as for thousands.
• One reviewer finding not acted on. Amending ADR-0036: it is about how RESTest numbers its own kinds of fault, and holds no list of what its checks leave alone. ADR-0014's amendment and docs/faults.md hold those.
• Test fixtures changed.
  • Specifications.pets() now models /noreply as declaring no content, as a model read from that document would.
  • An existing test said it tried "an operation the model does not know" with one the model does know. It now uses one it does not.
• Sample output not yet updated. The summaries quoted in the README and in docs/report.md's "The summary" come from runs before this change, and a run of the same pet shop now prints the new lines. They are re-taken, with the rest of the documentation's output, by the pull request that prepares the next release.

Open questions

None.

---

• Targets master
• English throughout, including comments and test names
• The hard rules in AGENTS.md hold
• Nothing from "Out of scope" in docs/DESIGN.md without explicit approval

🤖 Generated with Claude Code