Skip to content

jsontestsuite harness silently ignores files with unrecognized prefixes and passes vacuously on an empty corpus #4

Description

@victornicolet

The per-file dispatch is a chain of if name.starts_with("y_") / else if "n_" / else if "i_" with no final else branch, so any JSON file in tests/data/json_test_suite whose name does not begin with one of those three prefixes is read, parsed, and then its result is discarded without being counted or checked. In addition, the only precondition asserted is dir.is_dir(); if the corpus directory exists but contains no .json files (e.g. test data not checked out / not fetched), entries is empty, failures is empty, and the test reports success while exercising nothing. Both make the suite report a pass for cases it never actually validated.

Impact

Regressions in parse_json can go undetected: a mis-prefixed or newly added test file is silently skipped, and a missing/empty test-data checkout yields a green test run that verifies zero JSON inputs.

Suggestion

Add a final else { panic!("unrecognized test file prefix: {name}") } branch to the prefix dispatch, and assert a minimum corpus size (e.g. assert!(entries.len() > 0) or an exact expected count) before the loop so an empty or truncated data directory fails the test.

Code

if name.starts_with("y_") { ... } else if name.starts_with("n_") { ... } else if name.starts_with("i_") {
    match result { Ok(_) => i_accept += 1, Err(_) => i_reject += 1 }
}
// no else: files with other prefixes are silently ignored

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingImprovements to testing.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions