List AnVIL's donor and biosample ids on the records they tie to (#669)

[NoopDog]

Closes #669. Slice 2 of epic #692.

What changed

• Records. DonorRecord and SampleRecord gain repository_ids: every AnVIL donor_id / biosample_id that ties to the record, each with its repository, dataset and table. The schema class is RepositoryIdentifier. The field is absent where no ID ties.
• Entity map (sources/anvil_entity_map.yaml, entity_map.py).
  • published now names one table per kind: donor → anvil_donor, sample → anvil_biosample. Each gives its source_rows and repository_id column.
  • A table may name only an identifier and read no donor slot. Six such tables are added, the ones AnVIL builds biosamples from: 1000G sample, PRIMED sample, HPRC sample, IGVF sample, MAGE sequencing_library and NIA_CARD sample.
• Check (make check-entity-map, anvil_entities.check).
  • Each mapped table is held to every kind it serves. It serves donors if it reads a slot or AnVIL donors are made from its rows, and samples if AnVIL biosamples are. For each kind, at least half its identifier values must be recognised as that kind.
  • A table that serves no kind is refused.
  • Each published table's source_rows and repository_id columns must hold a value on some row.
• Import (make import-anvil-entities).
  • Each tied AnVIL record writes a donor_repository_id or sample_repository_id line carrying its ID, verbatim.
  • anvil_biosample gets its own file under anvil_published/.
  • Tables read for identifiers only are listed in the output.
  • The check and the import both list the datasets the map leaves out, with their AnVIL record counts (ENCORE: 3 donors, 634 biosamples).
• Reconcile (reconcile_entities).
  • ID lines are recognised by their field's kind and listed on the record, never translated. The value map's seeder and review queue never see them: entity_evidence.entries takes fields as iter_evidence does.
  • A sample that only a biosample names now gets a SampleRecord.
  • An AnVIL ID tying to two of our records is refused, and so is an ID line in a file that isn't a published table's.
  • The report's entities block gains repository_ids: per kind, the records holding IDs, the distinct IDs, how many IDs each record holds, and per-dataset counts. samples gains named_by_files.
• Identifier rule igvf_sample_accession (IGVFSM plus 8 characters, names a sample only), so IGVF's 1,935 biosamples tie.
• ADR-0002 decision 2 corrected. It said AnVIL's donor UUID is minted again in every snapshot. Epic: donor and sample records, keyed by identifier, so files can be found by who they came from #692 measured the same IDs in prod anvil15 and dev's 1000G snapshot.

Why

The explorer's donor and biosample pages are keyed by AnVIL's IDs, and our records are keyed by our identifier (coriell:HG00183). Nothing joined the two. AnVIL's IDs were the same in prod anvil15 and dev's 1000G snapshot, and map many to one onto our records, so each record lists the ones that tie to it (#692's measurements).

Assumptions I made

• Records for samples no file names. A biosample's sample gets a SampleRecord even when no file names it, as donors already did (decided at the read-back).
• No fallback identifier. An AnVIL record whose source row names no recognised identifier ties to nothing and is counted, so there is no dataset_local fallback. ENCORE's HEK293 lines stay out of the map; minting them belongs with Capture the donors and biosamples AnVIL leaves out of T2T_CHRY, MAGE and nhp_dGTEx, and link them across datasets #559 (decided at the read-back).
• Uniqueness. An AnVIL ID is one record's by its kind, repository and ID. The same UUID in two datasets tying to one record is listed once per dataset, because PRIMED and T2T share donor IDs. The same UUID used for a donor and a biosample is fine too, because ties are keyed by kind. That happens: PRIMED's HG00183 donor and biosample share 775b7b7a….
• Ambiguity is judged on raw text. At import, a record whose source rows name two different identifier strings is still counted as ambiguous and skipped, even if reconcile would read both as one sample (GM19908 / NA19908). Identifiers stay uninterpreted at import (contract 1.1). It measured zero on anvil15 (decided at review).
• Where the kind lives. The ID line's field (donor_repository_id vs sample_repository_id) says which kind it names. It is not on the envelope; moving it there belongs with the envelope split (Entity evidence envelope: its own schema class instead of a runtime refusal of entity_identifier #694). Reconcile does not check that the field's kind matches the file's table, because it has no map to check against without hard-coding AnVIL's table names (decided at review).
• What the entity map refuses. A dataset in the map must have at least one table reading a donor slot. Otherwise the import would write no submitter generation, while it writes a dataset's two generations together.
• The shared data has not been updated yet. data/entity_evidence/ and run 20261014_004231 are shared across clones, and main's reader refuses the new line fields. So the verification below ran in a scratch folder. After merge: make import-anvil-entities, then make reconcile RUN_DIR=output/anvil/20261014_004231.

How to verify

Run from the repo root on this branch, with the anvil15 manifests on disk (make download). The import and reconcile below write to a scratch folder and a copy of the run, never to the shared data/ or output/.

1. Check the map against the manifests:
   uv run python scripts/import_anvil_entities.py --check
   Expect 17 tables across 9 dataset(s), no problems, and two lines listing AnVIL_ENCORE_293T and AnVIL_ENCORE_RS293 as left out.
2. Import into scratch:
   uv run python scripts/import_anvil_entities.py --entity-root /tmp/ev669
   Expect the six read for identifiers only lines and Wrote 57,373 entity lines.
3. Reconcile a clone of the run:

   cp -cR output/anvil/20261014_004231 /tmp/run669
   rm -rf /tmp/run669/reconciled
   uv run python scripts/reconcile.py --run /tmp/run669 --entity-root /tmp/ev669
   

   It takes about 5 minutes.
4. Inspect: gzip -dc /tmp/run669/reconciled/donors.ndjson.gz | grep '"HG00183"' and the same for samples.ndjson.gz. The report's block is jq .entities.repository_ids /tmp/run669/reconciled/reconcile_report.json.

The issue's definition of done, mapped to these steps:

• The 1000G and T2T donors made for HG00183 both appear in its repository_ids. Step 4: the donor lists b5f9eb7a… (1000G high coverage), and 775b7b7a… listed for both PRIMED and T2T.
• Every AnVIL donor and biosample in a mapped dataset ties to exactly one record, or is counted with its reason; left-out datasets are listed. Steps 1, 2 and 4, measured:
  • Donors: 10,071 of the mapped datasets' 10,083 tie, as 6,596 distinct IDs. The other 12 are counted: 10 HPRC donors with no mapped source row (step 2's printout), and 2 HPRC_R2 rows named CHM13 / GRCh38 (entities.unrecognised).
  • Biosamples: 16,544 of 16,547 tie, as 16,264 distinct IDs. The other 3 (mHomSap3, CHM13, GRCh38) are in entities.unrecognised.
  • ENCORE is listed in step 1.
• An ID tying to two records is refused. Covered by test_a_repository_identifier_tying_to_two_records_is_refused_not_chosen. On anvil15 none does: step 3 completes. A UUID that is both a donor's and a sample's (6,418 on anvil15) is not a conflict, because ties are per kind (test_one_uuid_as_a_donor_id_and_a_sample_id_ties_once_per_kind).
• The biosample side's cardinality is measured. Step 4's ids_per_record for sample: 5,148 records hold IDs, of which 1,946 hold 1, 2,471 hold 4, 707 hold 6 and 24 hold 8. One person's sample collects IDs from PRIMED, 1000G high coverage (two: its participant and sample rows), T2T, and MAGE (up to four: kgp_sample and the sequencing libraries). On the donor side, 3,202 records hold 2 and 192 hold 1. The 8 donors with none are the same 8 Epic: donor and sample records, keyed by identifier, so files can be found by who they came from #692 found.

Also checked: against the shared run's current reconciled output, every file record and sample list is identical. donors is identical apart from repository_ids, and samples differs only by repository_ids and IGVF's 1,935 new sample records. Gate at dccb6ad: make test (3,011 passed, 9 skipped), make test-schema, make lint-all, make format-check and make validate-metadata all pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
🐦 Built with cc-claude-tools

[Copilot]

🟡 Changes recommended

Identifier-only imports can fail after publishing evidence, and malformed repository IDs can produce schema-invalid records.

2 open issues

• Handle identifier-only imports and clear stale submitter evidence · New
• Reject malformed imported repository IDs before recording evidence · New

What changed in this PR

Connects meta-disco’s donor and sample records to AnVIL’s stable identifiers, enabling explorer joins for #669.

Changes:

• Adds repository identifiers to entity records and reconciliation reports.
• Extends entity imports to biosamples and identifier-only tables.
• Adds IGVF sample recognition, tests, and documentation.

File	Description
tests/​test_sample_identity.py	Tests IGVF sample recognition.
tests/​test_entities.py	Tests imports, identifier ties, and reporting.
src/​meta_disco/​sources/​anvil_entity_map.yaml	Maps biosamples and identifier-only tables.
src/​meta_disco/​schema/​classification.yaml	Defines repository identifiers.
src/​meta_disco/​schema/​classification_model.py	Updates generated schema models.
src/​meta_disco/​sample_identity.py	Maps entity kinds to ID fields.
src/​meta_disco/​rules/​sample_namespaces.yaml	Recognises IGVF sample accessions.
src/​meta_disco/​reconcile_entities.py	Attaches IDs and reports cardinality.
src/​meta_disco/​models.py	Defines repository-ID constants.
src/​meta_disco/​entity_map.py	Supports published tables by kind.
src/​meta_disco/​entity_evidence.py	Filters evidence by requested fields.
src/​meta_disco/​anvil_entities.py	Imports IDs and checks expanded mappings.
scripts/​import_anvil_entities.py	Reports datasets omitted from mappings.
Makefile	Updates entity-import comments.
docs/​adr/​0002-derivation-graph.md	Corrects UUID stability documentation.
CLAUDE.md	Documents expanded entity handling.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

Static review identified no unresolved blocking issues, and regression coverage addresses the previously reported failures.

No open issues 🎉

2 resolved since last review

• Handle identifier-only imports and clear stale submitter evidence
• Reject malformed imported repository IDs before recording evidence

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No unresolved blocking issues remain, and regression coverage addresses identifier linking, uniqueness, malformed values, and translation separation.

No open issues 🎉

_{Review effort: Balanced}