Skip to content

Graph extension list is written three times with no drift guard (anchor fixes two; extension_map needs a test) #159

Description

@NoopDog

Problem

The graph extension set is written out three times, and a fourth place holds a deliberate subset. Nothing enforces that any of them agree.

where contents kind
rules/unified_rules.yaml:218-225extension_map 8 keys mapping to pangenome source of truth for "is this a graph file"
rules/unified_rules.yaml:492pangenome_graph.when.extensions the same 8, byte-identical rule guard
rules/unified_rules.yaml:626pangenome_reference_mc.when.extensions the same 8, byte-identical rule guard
src/meta_disco/header_classifier.pyGRAPH_TEXT_EXTENSIONS 4 of the 8 deliberate subset: the text formats the fetcher can parse (GFA_CONFIG.extensions reuses it)

Add a ninth graph format (.gaf, odgi .og) and you must edit three lists. Miss extension_map and the file has no category; miss pangenome_graph and it produces no base claim (not_classified); miss pangenome_reference_mc and a new-format -mc- graph silently stays pangenome instead of pangenome.reference. Nothing fails loudly in any of those cases.

Fix, in two parts — and the obvious half doesn't work

1. The two rule lists can be collapsed with a YAML anchor. Verified: yaml.safe_load resolves anchors, and both rules live in the same document (document #1; the file is loaded with safe_load_all and the first --- is at line 238). So this works with zero code change:

- id: pangenome_graph
  when:
    extensions: &graph_exts [".gfa", ".gfa.gz", ".rgfa", ".rgfa.gz", ".gbz", ".vg", ".gbwt", ".xg"]
...
- id: pangenome_reference_mc
  when:
    extensions: *graph_exts

2. An anchor CANNOT reach extension_map. It is in document #0, and YAML aliases do not cross document boundaries — confirmed:

yaml.safe_load_all('a: &x [1]\n---\nb: *x\n')  ->  ComposerError

So the extension_map ↔ rules relationship needs a test, not an anchor. That is the more valuable half anyway, because it is the edge where a miss produces a silently unclassified file.

Proposed guard

A drift test in the spirit of test_orchestration.py and test_stub_payloads_cover_all_file_types:

def test_graph_extension_lists_agree():
    graph_keys = {k for k, v in rules.extension_map.items() if v == "pangenome"}
    for rule_id in ("pangenome_graph", "pangenome_reference_mc"):
        assert set(rule_extensions(rule_id)) == graph_keys

def test_gfa_config_extensions_are_graph_extensions():
    """A deliberate subset — the text formats the fetcher can parse — but it must
    not contain an extension the graph rules do not know."""
    assert set(GFA_CONFIG.extensions) <= graph_keys

Both pass today (verified): extension_map pangenome keys are exactly the 8, and GFA_CONFIG.extensions is the 4-element text subset.

The GFA_CONFIG subset relationship matters in a way that is easy to miss: a text extension added to GFA_CONFIG but not to the graph rules would be fetched and parsed, produce no base pangenome claim, and then — if the content carries rGFA tags — receive a data_type: pangenome.reference claim on a record whose data_modality is not_classified. That is precisely the incoherent record #151's review found by another route.

Generalization (optional, larger)

when.file_format matches the extension string, and UnifiedRules.get_file_type() (which returns the extension_map category) has zero callers repo-wide. If _rule_matches resolved extension → category and rules could say when: {file_type: pangenome}, both rule lists would disappear and extension_map would become the single source of truth it already looks like. That is a rule-engine change touching every file type; the drift test is the cheap 90%.

Tasks

  • Collapse pangenome_graph / pangenome_reference_mc extension lists with a YAML anchor.
  • Add the extension_map ↔ rules drift test (anchors cannot express it).
  • Add the GFA_CONFIG.extensions ⊆ graph extensions subset test.
  • Consider when: {file_type: <category>} and giving get_file_type() its first caller.

Refs: #148, #151 (added GRAPH_TEXT_EXTENSIONS and deduplicated the Python side), #154 (fact model — extension becomes a fact, not a filename artifact).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions