Add incremental gps_images generation pipeline (LMFDB#7028)#43
Open
roed-math wants to merge 2 commits into
Open
Add incremental gps_images generation pipeline (LMFDB#7028)#43roed-math wants to merge 2 commits into
roed-math wants to merge 2 commits into
Conversation
Subgroup diagrams show '?' whenever a subgroup's tex name has no row in gps_images (fallback in lmfdb/groups/abstract/main.py diagram_js). Scoping against devmirror found 505,257 referenced tex names with no image (7.25M references from gps_groups.tex_name / gps_subgroup_search.subgroup_tex and quotient_tex), including \PSL(2,7) from the issue's example 336.208. Add an incremental path to the Make-images pipeline: - dump-missing-group-names.py: like dump-group-names.py, but computes reference counts server-side (gps_subgroup_search has 275M rows), emits only names missing from gps_images, most-referenced first, and skips names whose macros latex cannot expand (e.g. the broken \PSOPlusPlus names, which the site's KaTeX cannot render either). - load-new-imgs.py: builds an 'imageadder' file with only genuinely-new rows and appends via db.gps_images.copy_from instead of rewriting the whole 733K-row table. - images/eq.tex: define 18 macros used by missing names, matching the KaTeX definitions in lmfdb/templates/base.html (\POPlus, \Unitary, \GOrth*, \GSO*, \POmega*, etc.). - Readme: document the incremental path and that dvipng needs -D 120 to match existing images (verified pixel-identical against db rows). Verified: rendered names reproduce existing db images pixel-for-pixel at -D 120 (\PGL(2,7), A_4, C_2\times S_4, ?); full 504,306-name render + upload file produced separately (data upload, no PR needed for that part). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…check (LMFDB#7028) The incremental dump classified names as unrenderable using eq.tex's macro set, which lacked six group-name macros present in the site-wide KaTeX list (base.html): \ASp, \PO, \OPlus, \OMinus, \PSigmaL, \Dic. Names using them would have been silently dropped from image generation. Add the six macros to eq.tex (bodies matching KaTeX) and a preflight, macro_check.py, that compares the two macro sets in both directions (with a documented exclusion list for base.html's non-group macros) and aborts dump-missing-group-names.py before any query when they drift. Verified: preflight passes on the synced files and fails correctly on doctored copies; re-scoping the 2026-07-19 snapshot under the synced set shows no referenced tex name uses any of the six macros, so the 504,306-row upload file was already complete and the 951 excluded names (all using the undefined \PSOPlusPlus) are truly broken; the full-file validator still exits 0 with the new preamble, 60/60 sampled rows re-rendering pixel-identically; the six macros spot-rendered and visually checked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Addressed the external review's P2 finding (macro drift between the renderer and site KaTeX): commit 5e242c5 adds the six missing group macros (\ASp, \PO, \OPlus, \OMinus, \PSigmaL, \Dic) to eq.tex — all 77 site group macros now covered — and adds Impact check on the actual data: re-scoping the 2026-07-19 snapshot found zero wrongly-excluded names — none of the six macros occurs in any of the 1,015,994 referenced tex names — so the existing 504,306-row upload file is complete for this snapshot and unchanged; the 951 excluded names are all the truly-broken \PSOPlusPlus family. The validator passes under the new preamble (60/60 pixel-identical) and the six macros were spot-rendered. The fix matters for future names using those macros rather than for the current upload. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Subgroup diagrams show
?when a subgroup's tex name has no row in gps_images; scoping againstdevmirror found 505,257 referenced tex names (7.25M referencing rows) with no image, including the
\PSL(2,7)from 336.208 reported in the issue. This PR adds an incremental path to the Make-imagespipeline:
dump-missing-group-names.py(server-side reference counts, missing names only, most-referencedfirst),
load-new-imgs.py(appends only new rows viacopy_frominstead of rewriting the table), 18 neweq.tex macros matching the site's KaTeX definitions, and Readme notes including the
-D 120dvipng flagneeded to match existing images (verified pixel-identical). A full 504,306-row upload file has been
generated and validated with this pipeline and is ready to load. Addresses LMFDB#7028.
🤖 Generated with Claude Code