fix: correctly parse headerless _motion.tsv files#420
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #420 +/- ##
==========================================
+ Coverage 87.38% 87.41% +0.03%
==========================================
Files 65 65
Lines 4779 4793 +14
Branches 782 787 +5
==========================================
+ Hits 4176 4190 +14
Misses 510 510
Partials 93 93 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
effigies
left a comment
There was a problem hiding this comment.
Thanks for this! I think the cleaner patch would actually be to the loadTSVGZ immediately above where we make the .pipeThrough(new DecompressionStream('gzip')) call optional.
I would suggest we give it a more general name, e.g., loadHeaderlessTSV, with a compression parameter. You'll also need to extract the headers from the associated channels.tsv file, which will involve adding a name field to:
bids-validator/src/schema/associations.ts
Lines 93 to 104 in f0cf0fc
You can then pass in this.associations.channels.name as the headers.
| if (headerless) { | ||
| this.columns = new ColumnsMap() as Record<string, string[]> | ||
| } |
There was a problem hiding this comment.
Here you undo the load. That's not going to work.
Fixes #394
Description
This PR fixes a bug where the Deno validator incorrectly parsed the first row of data in
_motion.tsvfiles as column headers, resulting in false-positiveCUSTOM_COLUMN_WITHOUT_DESCRIPTIONerrors.According to BEP029,
_motion.tsvfiles are uncompressed and explicitly headerless. However,the standard
_loadTSVfunction was previously hardcoded to always consume the first row asheaders.
Changes Made
src/files/tsv.ts: Introduced aheaderlessboolean parameter (defaultfalse) to_loadTSVand its memoized wrappers. Whenheaderlessis true, the parser safely bypassesduplicate header checks, auto-generates dummy columns to preserve table structure, and retains
the first row of numerical data.
src/schema/context.ts: InsideBIDSContext.loadColumns(), if the file suffix ismotion, it now explicitly calls the TSV parser withheaderless: true. After structuralvalidation, it dynamically clears the parsed
columnsmap for motion files so thatevalColumnsignores them (mirroring the legacy validator's behavior).src/schema/associations.ts& Tests: Updated existing data loaders and test calls toexplicitly pass
falseto maintain standard behavior for all other BIDS TSVs.Testing
_motion.tsvfiles; verified the custom column errors disappeared.
deno task testagainst thebids-examplessubmodule) with 349 passing tests and 0 failures.