Skip to content

Avoid needless completeness queries and cap CI database concurrency - #7185

Merged
roed314 merged 3 commits into
LMFDB:mainfrom
roed-math:completeness-precheck-ci-concurrency
Aug 9, 2026
Merged

Avoid needless completeness queries and cap CI database concurrency#7185
roed314 merged 3 commits into
LMFDB:mainfrom
roed-math:completeness-precheck-ci-concurrency

Conversation

@roed-math

@roed-math roed-math commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add an optional database-independent precheck phase to
    CompletenessChecker.
  • Use it to recognize incompatible number-field root-discriminant and Galois
    root-discriminant constraints before querying for null stored values.
  • Add regression coverage proving that the contradictory query does not access
    the database and that compatible intervals are not short-circuited.
  • Cap matrix execution at six concurrent jobs to reduce peak load on the shared
    devmirror database.

Motivation

With the statement timeout introduced in #7181, UtilsTest.test_complete
repeatedly failed while executing this preliminary null-data query
(e.g. https://github.com/LMFDB/lmfdb/actions/runs/31268853079/job/93204410081):

SELECT "id"
FROM "nf_fields"
WHERE "degree" = 5
  AND "rd" >= 40
  AND "rd" <= 60
  AND "grd" IS NULL
LIMIT 1;

The original query is mathematically impossible: its root-discriminant interval
lies strictly above its Galois-root-discriminant interval. The existing
NFBound logic detects that contradiction, but only after the generic null-data
probe.

This PR introduces a narrow precheck contract for conclusions that are valid
independently of database contents. It does not move ordinary completeness
guarantees ahead of null checking: those can depend on all relevant columns
having been computed, so they stay behind the null-data check.

Safety

  • Predicates carrying SQL-null semantics (None anywhere in the rd/grd
    value, e.g. {"$ne": None} for IS NOT NULL) bypass the numeric precheck
    entirely: the real-number model cannot represent them (to_rset reads
    None as the whole real line, so {"$ne": None} would collapse to the
    empty set and misreport a nonempty search as impossible). A sentinel keeps
    an omitted constraint distinguishable from an explicit null predicate, and
    a regression test covers four null-predicate shapes, including the
    {"$or": [{"$ne": None}, {"$lte": 30}]} collapse.
  • Compatible queries still use the existing null-data checks.
  • The existing fallback to "completeness unknown" is unchanged.
  • The test expectation is not weakened.
  • The workflow's 60-minute job timeout is unchanged.
  • No database index or schema change is included.

Verification

  • Focused no-database-access, compatible-input, and null-predicate regression
    tests pass (22 tests total in test_utils.py).
  • Calling results_complete on the contradictory query against the real
    database with psycodict statement logging enabled executes zero SQL
    statements and returns in under a millisecond, with the exact existing
    result triple.
  • tox -e tests -- lmfdb/tests/test_utils.py passes (21 tests, 19.7s), with
    UtilsTest.test_complete at 14.9s.
  • The same targeted test passes with PGOPTIONS='-c statement_timeout=120s'
    (the pre-Raise the CI database statement timeout to 10 minutes #7184 threshold) and with PGOPTIONS='-c statement_timeout=10min'.
  • Workflow YAML parses; jobs.test.strategy retains fail-fast: false and the
    unchanged matrix expression, with max-parallel: 6 added and
    timeout-minutes: 60 untouched.
  • git diff --check, pyflakes, pylint (repo flags) and
    ruff check --preview --select=E722 pass on the edited files.
  • Full CI: all 12 devmirror matrix jobs passed on both the original commits
    (run 31300780554,
    16 minutes total) and the null-predicate fix
    (run 31332813750).
    Details below are from the first run:
    • The formerly failing shard (devmirror belyi higher_genus_w_automorphisms hypergm lattice maass_forms tests users, the one running lmfdb/tests/test_utils.py) passed.
    • UtilsTest.test_complete took 9.22s in CI (it previously stalled on the canceled null probe until the 120s statement timeout); the two new regression tests took 0.00s.
    • No statement timeout or QueryCanceled appears anywhere in that job's log.
    • All matrix entries were still created; max-parallel limited execution without suppressing jobs (observed 6 in progress + 6 queued, with the second wave starting as first-wave jobs finished).

🤖 Generated with Claude Code

roed-math and others added 2 commits August 9, 2026 03:16
Add an optional database-independent precheck hook to CompletenessChecker,
running after $or/$and decomposition and before any table lookup or
null-count query.  Register an NFBound precheck that recognizes a root
discriminant range lying above the Galois root discriminant range, so the
intrinsically impossible query no longer waits on the expensive null-data
probe that CI's statement timeout cancels.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The devmirror shards share one database; cap concurrent matrix jobs at six
so functional CI does not double as an uncontrolled load test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
psycodict predicates involving None carry SQL-null semantics ({"$ne": None}
means IS NOT NULL), which to_rset misreads as empty numeric sets: None maps to
the whole real line, so its complement is empty.  The precheck could therefore
declare a nonempty search (e.g. grd IS NOT NULL, or "non-null or at most 30")
intrinsically impossible without touching the database.

Conservatively bypass the numeric precheck whenever None appears anywhere in
the rd/grd expression tree, using a sentinel so an omitted constraint stays
distinguishable from an explicit null predicate.  The numeric contradiction
fast path is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@roed-math

Copy link
Copy Markdown
Contributor Author

Pushed 2e6a262 fixing a correctness issue in the precheck: psycodict predicates carrying SQL-null semantics ({"$ne": None} = IS NOT NULL) were misread by the numeric model as empty sets (to_rset maps None to the whole real line, so its complement is empty), so e.g. {"grd": {"$ne": None}} — and the nastier collapse {"grd": {"$or": [{"$ne": None}, {"$lte": 30}]}} — could be declared intrinsically impossible without touching the database.

The precheck now conservatively declines any rd/grd constraint containing None anywhere in its expression tree, using a sentinel so an omitted key (no constraint) stays distinguishable from an explicit null predicate. The numeric contradiction fast path is unchanged (still zero SQL statements for the motivating query), and a table-driven regression test covers the four null-predicate shapes. NumberSet/to_rset themselves are untouched.

@roed314
roed314 merged commit 92a75f5 into LMFDB:main Aug 9, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants