Skip to content

Probe reference resolvability through the backend writer - #867

Open
ehennestad wants to merge 2 commits into
mainfrom
zarr-support/6-writer-reference-probe
Open

ehennestad wants to merge 2 commits into
mainfrom
zarr-support/6-writer-reference-probe

Conversation

@ehennestad

@ehennestad ehennestad commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Background — This PR is part of a series that makes the write path storage-agnostic (stacked on #866), so that a Zarr v3 writer can be added later. Today MatNWB can only write HDF5 files.

Problem — Before writing a typed object, MetaClass.export checks that every ObjectView and RegionView property points to a target that already exists in the file. If a target does not exist yet, the object is skipped and exported again later by NwbFile.resolveReferences. This check used a raw HDF5 file id (writer.FileId), so it only works with the HDF5 backend. A writer for another storage format cannot provide an HDF5 file id.

Solution — Add the check to the writer interface as validateReferenceResolvable(referenceValue), and call that from MetaClass.export instead of using the file id. The HDF5 implementation does the same thing as before, so nothing changes in how files are written.

What changed

  • New method validateReferenceResolvable(referenceValue) on io.backend.base.Writer, implemented in io.backend.hdf5.HDF5Writer.
  • MetaClass.export no longer uses writer.FileId. Behavior is the same: objects with missing reference targets are still exported in the second pass.
  • Added the new method to the expected-methods list in the base writer test.
Implementation notes

The HDF5 implementation calls io.getRefData the same way the export code did before. This keeps the check consistent with the actual reference write: for a RegionView, it verifies that the referenced dataset can be opened, not just that its path exists. The error identifiers that captureReferenceErrors checks for (NWB:getRefData:InvalidPath and NWB:ObjectView:MissingPath) are unchanged.

How to test

Export and read back a file containing an object reference:

nwb = NwbFile( ...
    'identifier', 'stack-demo', ...
    'session_description', 'reference probe demo', ...
    'session_start_time', datetime(2026,8,25, 'TimeZone', 'local'));
ts = types.core.TimeSeries('data', (1:10)', 'data_unit', 'n/a', ...
    'starting_time', 0, 'starting_time_rate', 1);
nwb.acquisition.set('ts', ts);
tbl = types.hdmf_common.DynamicTable( ...
    'description', 'table with a reference column', 'colnames', {'ref'}, ...
    'ref', types.hdmf_common.VectorData('description', 'refs', ...
        'data', types.untyped.ObjectView(ts)), ...
    'id', types.hdmf_common.ElementIdentifiers('data', int64(0)));
nwb.analysis.set('tbl', tbl);
nwbExport(nwb, 'ref_demo.nwb');
in = nwbRead('ref_demo.nwb', 'ignorecache');
disp(in.analysis.get('tbl').getRow(1))
                ref             
    ____________________________

    1×1 types.untyped.ObjectView

Checklist

  • Have you ensured the PR description clearly describes the problem and solutions?
  • Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
  • If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Claude Code

@ehennestad
ehennestad changed the base branch from backend-agnostic-links to main August 25, 2026 12:33
@ehennestad
ehennestad force-pushed the zarr-support/6-writer-reference-probe branch from 022bb65 to 6b60aba Compare August 25, 2026 17:45
@ehennestad
ehennestad marked this pull request as ready for review August 25, 2026 17:50
@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.27%. Comparing base (2880afc) to head (7033673).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #867   +/-   ##
=======================================
  Coverage   95.27%   95.27%           
=======================================
  Files         234      234           
  Lines        8311     8313    +2     
=======================================
+ Hits         7918     7920    +2     
  Misses        393      393           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ehennestad
ehennestad force-pushed the zarr-support/6-writer-reference-probe branch from 6b60aba to ee8c44f Compare August 26, 2026 17:43
ehennestad and others added 2 commits August 31, 2026 12:42
MetaClass.export probed whether ObjectView/RegionView targets exist by
calling io.getRefData with writer.FileId, reaching through the abstract
writer to a raw HDF5 file id. The probe's return value was discarded;
only its success or failure matters, feeding the deferred second export
pass in NwbFile.resolveReferences.

Give the base Writer a validateReferenceResolvable method that states
this contract in backend-neutral terms: return silently when every
target resolves, throw 'NWB:getRefData:InvalidPath' when one does not,
and let 'NWB:ObjectView:MissingPath' propagate unchanged. HDF5Writer
implements it with the same io.getRefData call, so a RegionView probe
still verifies the referenced dataset opens. MetaClass.export no longer
touches writer.FileId.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The base-writer smoke test compares the class's method set against an
explicit list, so the interface method added by this branch must be
listed for the suite to pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ehennestad
ehennestad force-pushed the zarr-support/6-writer-reference-probe branch from ee8c44f to 7033673 Compare August 31, 2026 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant