Problem
The graph extension set is written out three times, and a fourth place holds a deliberate subset. Nothing enforces that any of them agree.
| where |
contents |
kind |
rules/unified_rules.yaml:218-225 — extension_map |
8 keys mapping to pangenome |
source of truth for "is this a graph file" |
rules/unified_rules.yaml:492 — pangenome_graph.when.extensions |
the same 8, byte-identical |
rule guard |
rules/unified_rules.yaml:626 — pangenome_reference_mc.when.extensions |
the same 8, byte-identical |
rule guard |
src/meta_disco/header_classifier.py — GRAPH_TEXT_EXTENSIONS |
4 of the 8 |
deliberate subset: the text formats the fetcher can parse (GFA_CONFIG.extensions reuses it) |
Add a ninth graph format (.gaf, odgi .og) and you must edit three lists. Miss extension_map and the file has no category; miss pangenome_graph and it produces no base claim (not_classified); miss pangenome_reference_mc and a new-format -mc- graph silently stays pangenome instead of pangenome.reference. Nothing fails loudly in any of those cases.
Fix, in two parts — and the obvious half doesn't work
1. The two rule lists can be collapsed with a YAML anchor. Verified: yaml.safe_load resolves anchors, and both rules live in the same document (document #1; the file is loaded with safe_load_all and the first --- is at line 238). So this works with zero code change:
- id: pangenome_graph
when:
extensions: &graph_exts [".gfa", ".gfa.gz", ".rgfa", ".rgfa.gz", ".gbz", ".vg", ".gbwt", ".xg"]
...
- id: pangenome_reference_mc
when:
extensions: *graph_exts
2. An anchor CANNOT reach extension_map. It is in document #0, and YAML aliases do not cross document boundaries — confirmed:
yaml.safe_load_all('a: &x [1]\n---\nb: *x\n') -> ComposerError
So the extension_map ↔ rules relationship needs a test, not an anchor. That is the more valuable half anyway, because it is the edge where a miss produces a silently unclassified file.
Proposed guard
A drift test in the spirit of test_orchestration.py and test_stub_payloads_cover_all_file_types:
def test_graph_extension_lists_agree():
graph_keys = {k for k, v in rules.extension_map.items() if v == "pangenome"}
for rule_id in ("pangenome_graph", "pangenome_reference_mc"):
assert set(rule_extensions(rule_id)) == graph_keys
def test_gfa_config_extensions_are_graph_extensions():
"""A deliberate subset — the text formats the fetcher can parse — but it must
not contain an extension the graph rules do not know."""
assert set(GFA_CONFIG.extensions) <= graph_keys
Both pass today (verified): extension_map pangenome keys are exactly the 8, and GFA_CONFIG.extensions is the 4-element text subset.
The GFA_CONFIG subset relationship matters in a way that is easy to miss: a text extension added to GFA_CONFIG but not to the graph rules would be fetched and parsed, produce no base pangenome claim, and then — if the content carries rGFA tags — receive a data_type: pangenome.reference claim on a record whose data_modality is not_classified. That is precisely the incoherent record #151's review found by another route.
Generalization (optional, larger)
when.file_format matches the extension string, and UnifiedRules.get_file_type() (which returns the extension_map category) has zero callers repo-wide. If _rule_matches resolved extension → category and rules could say when: {file_type: pangenome}, both rule lists would disappear and extension_map would become the single source of truth it already looks like. That is a rule-engine change touching every file type; the drift test is the cheap 90%.
Tasks
Refs: #148, #151 (added GRAPH_TEXT_EXTENSIONS and deduplicated the Python side), #154 (fact model — extension becomes a fact, not a filename artifact).
Problem
The graph extension set is written out three times, and a fourth place holds a deliberate subset. Nothing enforces that any of them agree.
rules/unified_rules.yaml:218-225—extension_mappangenomerules/unified_rules.yaml:492—pangenome_graph.when.extensionsrules/unified_rules.yaml:626—pangenome_reference_mc.when.extensionssrc/meta_disco/header_classifier.py—GRAPH_TEXT_EXTENSIONSGFA_CONFIG.extensionsreuses it)Add a ninth graph format (
.gaf, odgi.og) and you must edit three lists. Missextension_mapand the file has no category; misspangenome_graphand it produces no base claim (not_classified); misspangenome_reference_mcand a new-format-mc-graph silently stayspangenomeinstead ofpangenome.reference. Nothing fails loudly in any of those cases.Fix, in two parts — and the obvious half doesn't work
1. The two rule lists can be collapsed with a YAML anchor. Verified:
yaml.safe_loadresolves anchors, and both rules live in the same document (document #1; the file is loaded withsafe_load_alland the first---is at line 238). So this works with zero code change:2. An anchor CANNOT reach
extension_map. It is in document #0, and YAML aliases do not cross document boundaries — confirmed:So the
extension_map↔ rules relationship needs a test, not an anchor. That is the more valuable half anyway, because it is the edge where a miss produces a silently unclassified file.Proposed guard
A drift test in the spirit of
test_orchestration.pyandtest_stub_payloads_cover_all_file_types:Both pass today (verified):
extension_mappangenome keys are exactly the 8, andGFA_CONFIG.extensionsis the 4-element text subset.The
GFA_CONFIGsubset relationship matters in a way that is easy to miss: a text extension added toGFA_CONFIGbut not to the graph rules would be fetched and parsed, produce no basepangenomeclaim, and then — if the content carries rGFA tags — receive adata_type: pangenome.referenceclaim on a record whosedata_modalityisnot_classified. That is precisely the incoherent record #151's review found by another route.Generalization (optional, larger)
when.file_formatmatches the extension string, andUnifiedRules.get_file_type()(which returns theextension_mapcategory) has zero callers repo-wide. If_rule_matchesresolved extension → category and rules could saywhen: {file_type: pangenome}, both rule lists would disappear andextension_mapwould become the single source of truth it already looks like. That is a rule-engine change touching every file type; the drift test is the cheap 90%.Tasks
pangenome_graph/pangenome_reference_mcextension lists with a YAML anchor.extension_map↔ rules drift test (anchors cannot express it).GFA_CONFIG.extensions ⊆ graph extensionssubset test.when: {file_type: <category>}and givingget_file_type()its first caller.Refs: #148, #151 (added
GRAPH_TEXT_EXTENSIONSand deduplicated the Python side), #154 (fact model — extension becomes a fact, not a filename artifact).