Add congruence support to IntegerSet, fixing "multiple of" completeness errors

[roed-math]

"Multiple of conductor/level" searches (elliptic curves over Q, classical modular forms, modular
curves, mod-ell Galois representations) produce {"$mod": [r, m]} query constraints, which made
to_rset raise ValueError: Unsupported key $mod in the completeness checker; since LMFDB#6879 the
error is caught but every such search still logs it and flashes "There was an error in the
completeness checking code". This adds congruence support to the checker's set machinery.

Representation. CongruenceSet is a layer on top of IntegerSet: it inherits from it and adds
a modulus and a set of allowed residues. IntegerSet(query) returns one when parsing needs it (the
way pathlib.Path() returns a PosixPath), so ordinary interval and finite sets keep using the
plain IntegerSet. A bounded congruence set with few enough elements collapses into an exact finite
IntegerSet, which makes unions of branches with different congruences exact instead of
over-approximations. Congruence remains only as the value object describing a residue system.

Parsing. parse_set(query, cls) builds set objects and combines them with the sets' own
union, intersection and complement, tracking congruences exactly through $and, $or,
$not and $ne via the CRT. There is a single implementation of the set algebra and of the
exactness invariant. to_rset still returns a Sage RealSet for the callers that want one, and
integer_normalize sharpens interval endpoints onto the congruence.

Soundness. Every set records whether it describes the query exactly; where a set is not exactly
representable the code over-approximates and marks it inexact, and is_subset never certifies
containment in an inexact set while difference never subtracts one. So completeness is never
claimed incorrectly: an interval like [1, 1000] is not considered a subset of the multiples of 7
(the pathology that blocked LMFDB#6853), while bounded searches such as conductor a multiple of 7 and at
most 400000 are correctly certified complete.

Iteration. Iterating a congruence set steps from each element to the next rather than walking
the ambient interval and filtering, so the cost is proportional to the number of values produced.
A user-supplied modulus could otherwise make a completeness check scan billions of integers between
two matches. Bounded, one-sided and two-sided unbounded ranges are all handled without testing
nonmembers, and the checkers that consume every element (PrimeBound, CPrimeBound, Smooth,
MaassBound) decline to certify rather than enumerating an unbounded or enormous query set.

$mod semantics follow psycodict's query language: the modulus must be positive and the residue
is reduced to r % m, so negative database values match by their nonnegative residue.

Unit tests cover the congruence and set behavior, sparse iteration (including a structural check
that no ambient integer is tested for membership), the checker guards, the comparison protocol, an
exhaustive small-domain comparison against a naive evaluation of the query, and the queries from the
issue; a page regression test checks that multiple-of-conductor searches no longer flash the error.

Addresses LMFDB#6822.

🤖 Generated with Claude Code

[roed-math]

Addressed the external review's P1 finding (inexact over-approximations treated as exact): commit 1e4b78c makes NumberSet/IntegerSet store the exactness flag that to_rset_cong already computed, propagates it conservatively through union/intersection/difference/negation/arithmetic, and refuses the two unsound operations — is_subset never certifies containment in an inexact right-hand side, and difference never subtracts an inexact set (bound_under subtracts the exact static interval instead, so covering bounds still work).

The reviewer's counterexample — (even in [0,2]) OR (odd in [10,12]) — is now inexact: {1} is no longer certified as a subset and difference no longer wrongly removes 1. Mixed-range/mixed-congruence regression tests added.

No lost functionality: inexact sets remain usable on the LEFT of is_subset, so every previous certification survives — verified against the original multiple-of-conductor issue queries, the full completeness test suites (test_utils 20, cmdline_search 62), and PR #31's usage patterns simulated directly. The tradeoff is deliberate conservatism: genuine members of inexact unions are no longer certified, which only ever weakens a claim, never falsifies one.

[roed314]

I think this would be better as an extra layer on top of an IntegerSet (so this inherits from IntegerSet).

[roed314]

In particular, a layer on top of IntegerSet (which has an underlying rset) would allow the easy representation of numbers congruent to a mod n in an interval, which is one of the main use cases.

[roed314]

And if the interval is short enough, this can collapse into a finite list (solving the problem that you can't represent unions in some cases).

[roed314]

It also makes your _pair constructions unnecessary, since they can be folded into methods.

[roed-math]

Done in 33d1153. CongruenceSet now inherits from IntegerSet and adds the modulus and residues; IntegerSet(query) returns one when parsing needs it (the way pathlib.Path() returns a PosixPath), so ordinary interval and finite sets keep using the plain IntegerSet. Congruence survives, but only as the value object describing a residue system: it is no longer meant to be a set, and the set-with-a-range is CongruenceSet.

Following the rest of the thread:

Collapse into a finite list. CongruenceSet.collapse() lists the elements when there are at most COLLAPSE_CAP (1000) of them, and union, difference, complement and is_subset use it. So your counterexample, (even in [0,2]) OR (odd in [10,12]), is now the exact set {0, 2, 11} instead of an inexact [0,2] ∪ [10,12]: 1 is no longer in the representation at all, and 2 is certified rather than refused. Beyond that size the union still falls back to one range plus one congruence, which is now marked exact under a sharper condition than before (no integer of the combined range may satisfy one input's congruence while lying only in the other input's range). That condition replaces the old "the two ranges are equal" test, which stopped working once parsing normalized endpoints onto the congruence, and it keeps cases like 0 mod 3 in [0,3000] ∪ 1 mod 3 in [0,3000] exact.

The _pair constructions are gone. Parsing is now parse_set(query, cls), which builds set objects and combines them with the sets' own union, intersection and complement methods; _pair_empty, _pair_intersection, _pair_union, _pair_complement and to_rset_cong are all deleted. IntegerSet._build(rset, cong, exact) is the single constructor (it normalizes the range, picks the class and records exactness), so the exactness invariant has one implementation rather than one for triples and one for sets. to_rset is unchanged for the callers that want a Sage RealSet.

The soundness properties from the previous round are preserved (the mixed-branch case is still a regression test, in both its collapsing and non-collapsing forms), and there is now an exhaustive small-domain test that compares the represented set against a naive evaluation of the query on [-20, 20] for union, intersection, difference, complement and subset certification.

[roed314]

This doesn't handle the case that other is not a Congruence.

[roed-math]

Fixed in 33d1153:

def __eq__(self, other):
    if not isinstance(other, Congruence):
        return NotImplemented
    return self.modulus == other.modulus and self.residues == other.residues

Returning NotImplemented lets Python try the reflected operation and fall back on identity, instead of the class asserting that it is unequal to something it knows nothing about. The custom __ne__ is removed (Python derives it from __eq__), and __hash__ is kept so that congruences stay usable in sets and dict keys.

test_congruence_comparison covers the NotImplemented return itself, equality and inequality against an unrelated object in both operand orders, and hashing.

[roed-math]

Second review round addressed in 33d1153 (plus a merge of main, which the branch was 96 commits behind).

Merge blocker: iteration is now proportional to matches, not interval width

CongruenceSet.__iter__ steps from one element to the next through Congruence.iter_up / iter_down instead of walking every integer of the ambient interval and filtering. For each interval the least element of each residue class is computed once; those all lie in one period, so yielding them in order and then adding the modulus enumerates the whole progression in sorted order without a heap and with O(1) work per value.

All three interval shapes are handled without touching nonmembers: bounded (increasing), bounded above only (decreasing, as before), and unbounded in both directions (a fair alternation between nonnegative and negative values, matching what iterating ZZ used to give). The plain conductor_type=multiple search, which is unbounded, is the case that mattered most.

S = IntegerSet({"$mod": [0, 10**12], "$gte": 0, "$lte": 10**12})
it = iter(S)
next(it)   # 0
next(it)   # 10**12, previously ~10**12 candidates later
next(it)   # StopIteration

The regression tests are structural rather than timing based: as well as checking the values, test_congruence_iteration temporarily instruments Congruence.__contains__ and asserts that producing the elements of a set spanning 10^10 integers performs zero membership tests. Also covered: nonzero residues with a large modulus, negative bounded intervals, multiple residues (sorted, no duplicates), and one sided and two sided unbounded iteration.

Defensive checker guards

IntegerSet.cardinality(cap) counts elements arithmetically (per residue class for a CongruenceSet) and short circuits at cap; enumerable() is the predicate built on it, bounded by a documented ENUMERATION_CAP of 10000.

PrimeBound, CPrimeBound, Smooth and MaassBound now decline to certify rather than enumerating an unbounded or enormous query set, replacing the weaker is_finite() guard (finite is not the same as small: [1, 10**12] is finite). A completeness checker is always allowed to answer "not certified", and in each of these cases the old code would have reached the same answer, only slowly. IntegerSet.is_subset uses the same bound for its enumeration fallback, which is strictly more permissive than the old span based test: a sparse set with a huge span but few elements can now be checked element by element.

The other consumers were audited and left alone because their inputs are already small by construction: HMFBound lists degrees only after certifying they lie in [2,6] and discriminants only after intersecting with the finite set of fields we have, and the number field checker's r2opts and nopts are confined to [0, n/2] and [1, 47].

$mod semantics aligned with psycodict

parse_mod follows psycodict's query language: the modulus must be a positive integer, and the residue is reduced to r % m, matching psycodict's own value[0] % value[1] and therefore matching negative database values by their nonnegative residue (-7 in IntegerSet({"$mod": [0, 7]}) is True). A modulus of 0 or a negative modulus is now rejected with a ValueError instead of being silently reinterpreted: psycodict has no meaning for either (postgres' MOD would divide by zero or follow the sign of the column), so the checker must not invent one. Since LMFDB#7098 a raising check is downgraded to "completeness unknown" rather than an error, so this is not a user visible failure.

Validation

sage -python -m pytest on lmfdb/tests/test_utils.py (25 passed), lmfdb/elliptic_curves/test_ell_curves.py (31 passed) and lmfdb/tests/test_cmdline_search.py (62 passed), plus full CI on this push.

Checked directly:

1. /EllipticCurve/Q/?conductor=389&conductor_type=multiple contains 389.a1 with no completeness error (test_Cond_multiple_search).
2. {"conductor": {"$mod": [0, 7], "$lte": 400000}} is certified by the conductor at most 500000 bound.
3. {"conductor": {"$mod": [0, 7]}} returns not certified in 0.15s.
4. [1, 1000] is still not a subset of the multiples of 7.
5. Large modulus queries return promptly: bounded 0.16s, unbounded 0.15s, {"$mod": [3, 2**40], "$gte": 1, "$lte": 10**15} 0.15s.
6. Mixed range and mixed congruence unions that stay inexact never regain exact=True through intersection, union, negation, difference or complement.
7. The pre-existing completeness tests still pass, including the newly added expectations.

One thing found in passing, not fixed here

PrimeBound.__call__ never checks its own bound (unlike CPrimeBound, which calls super().__call__), so {"conductor": 1000000007} is certified as "elliptic curves with prime conductor at most 300 million" even though that prime is past the bound. This predates the PR and is unrelated to congruences, so I have left it alone rather than widening this branch; happy to fix it here or separately, whichever you prefer.

[roed-math]

Follow-up in 47e923a: one more non-terminating iteration, found while stress testing congruence queries against the other tables.

NFBound._one_n walks the discriminants in a range via IntegerSet.stickelberger and relies on clear_S rejecting one to break out of the loop. For a congruence-free range that happens within a few candidates, but a congruence set can skip arbitrarily many elements in a row without producing a candidate at all: every multiple of 10^6 is a multiple of 4 with (m//4) % 4 == 0, which Stickelberger's condition rejects inside the generator, so nothing is ever yielded. Before:

results_complete("nf_fields", {"degree": 2, "disc_abs": {"$mod": [0, 10**6], "$gte": 1}}, db)

never returned (with direct iteration it spins in the generator; with the old filtering iterator it would have been just as stuck, one modulus further down). It now returns (False, None, None) in 0.15s. The loop only runs for an enumerable range, which leaves congruence-free queries exactly as they were (they broke out immediately) and still falls through to the Minkowski bound. Regression tests cover both the unbounded case and a bounded congruence range that is still certified.

Extra local validation on top of what I listed above, all against devmirror:

suite	result
lmfdb/tests/test_utils.py	25 passed
lmfdb/elliptic_curves/test_ell_curves.py	31 passed
lmfdb/tests/test_cmdline_search.py	62 passed
lmfdb/number_fields	34 passed
maass_forms, hilbert_modular_forms, bianchi_modular_forms, abvar/fq and groups/abstract browse pages, sato_tate_groups	155 passed, 1 failed

The one failure is test_bmf.py::BMFTest::test_download_magma, which fails identically on main in a clean worktree, so it is not from this branch.

CI on this repo is currently backed up (every branch's run has been queued for the last hour), so the CI verdict will land later than this comment.

[roed314]

GPT signed off.

[roed-math]

Superseded by LMFDB#7139, opened upstream from this same branch (with main merged in, so it carries the PrimeBound bound check from LMFDB#7128). Closing here; review continues upstream.