[ENH] Add phenotype and rawbids directories to "study" datasets

[nikhil153]

This is a continuation of discussion on BIDS DatasetType started in #1972 and #2185. The proposed DatasetTypes include: bids-study, raw, and derivative.

I think this is a fantastic idea with the capacity to curate modular and nested standardized datasets.

However, the directory layout for bids-study proposed in #2185 is suboptimal from the data visibility perspective. Currently, the bids-study does not have a root-level subdirectory for bids-raw dataset. Based on previous conversations it was suggested that bids-raw dataset can be stowed inside sourcedata dir. This hidden location for a valid and probably the most common DatasetType is probably not ideal and will confuse new users.

So, I would like to make a case for treating sourcedata, bids-raw, and bids-derivative with equal importance by putting them on the same root-level inside the bids-study directory tree. In my experience this is more intuitive and helpful for data management of neuroimaging studies, where potentially different people will handle these three DatasetTypes.

The suggested bids- prefix for these directories is mostly to avoid possible confusion between source vs raw data connotations from past discussions and to indicate each of them can be stand-alone BIDS datasets.

Happy to hear your thoughts @michellewang, @jbpoline, @yarikoptic, @effigies, @mathdugre, @julia-pfarr, @nburgos, @AliceJoubert, @Adam-Ismaili-92, @surchs

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (6764039) to head (4cd73f4).
⚠️ Report is 21 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2191   +/-   ##
=======================================
  Coverage   83.07%   83.07%           
=======================================
  Files          22       22           
  Lines        1696     1696           
=======================================
  Hits         1409     1409           
  Misses        287      287           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jbpoline]

This makes a lot of sense to me, and would help everyone to more straightforwardly differentiate source-data and raw-bids much more easily.

[AliceJoubert]

This solution feels much clearer, I think it's a good idea !

[effigies]

Two notes on the schema. I don't have a strong opinion about bids-raw/, except to note that this will be the first use of a hyphen that doesn't separate key-value entities. If you want to be consistent with older versions of the spec, you could call it rawdata/. Many people seemed to think that was supposed to be valid...

[effigies]

This was removed after this discussion: #2185 (comment)

I didn't see a justification for re-adding it, so I want to make sure this isn't slipped in with little consideration.

[nikhil153]

Yes, should have explained this addition based on offline conversations.
The reason for removing phenotype directory was that it would presumably appear in the raw dataset. But isn't that true for derivatives directory as well? It's a bit confusing having derivatives part of all three DatasetTypes but enforcing phenotype only be part of raw or derivative. In practice, phenotype data curation is likely to be more independent than that of derivatives, which is one of the reasons we prefer it to be at the top level.

[effigies]

Sorry for the long silence.

The reason for removing phenotype directory was that it would presumably appear in the raw dataset. But isn't that true for derivatives directory as well? It's a bit confusing having derivatives part of all three DatasetTypes but enforcing phenotype only be part of raw or derivative. In practice, phenotype data curation is likely to be more independent than that of derivatives, which is one of the reasons we prefer it to be at the top level.

So the logic in derivatives is that any raw dataset is a valid derivative dataset, so you cannot restrict, you can only expand. Furthermore, you can have derivatives of derivatives, so nesting is entirely reasonable. I suppose you could preprocess phenotypic data and include it in a derivative, so it might even make sense.

IMO, if phenotypic data is worth including in the study, it's worth including in the raw data, regardless of the independence of curation. I still don't see the use case for phenotype at the study level.

That said, I won't block this.

[nikhil153]

@effigies - Thanks for the feedback and patiently dealing with these multiple intertwined open issues and hurried PR from my side - a result of mixed-up online and offline conversations. Will respond individually to the inline comments.

[julia-pfarr]

A few thoughts:

1. I think we need to be more clear about what bids-study is. If I was a naive user, I would think bids-study is a layout for organizing my study in "the bids way". Meaning that as soon as I move from unorganized sourcedata to some sort of organized data, it should be BIDS compliant. Thus, I do not need a bids prefix for raw or derivatives since my assumption is "everything BIDS". If we want to expand bids-study to allow it to be consumed by other schemas or standards (e.g., Nipoppy), I think we can find a different way than the folder-naming to make them compatible and validate. E.g., adding expanded validation rules in the sense of „if type bids-study is used, everything that BIDS has a spec on (raw, derivatives, phenotype etc.) needs to be in BIDS standard“ or „„if type bids-study is used, at least X% of data needs to be BIDS“ or add an output-validator file that lists which of the data was validated by BIDS. (idk if any of those suggestions make sense, it’s just wild ideas :-D)

2. It is true that phenotype is supposed to be on the level of the raw data. However, the <measurement_tool_name>.tsv/.json are connected to the (recommended) sessions.tsv/.json file which should be kept at the root level. Thus, it would be more convenient to not have too deep nesting of phenotype. Also, as measurement data can also evolve from source to raw, the <measurement_tool_name>.tsv/.json might contain information for both types.

3. in offline conversation we also had the discussion about making bids-study not a DatasetType but a StudyType. Reasoning for that would be to have a clear specification for StudyType, maybe the validation would be wasier than changing the nesting logic of existing DatasetTypes and it is expandable to e.g. Meta or Mega Analysis etc. Wdyt?

[julia-pfarr]

re-posting my comment from bids-specification/bids-website#688 here again since it was misplaced in that PR:

I kind of don't understand why we need to serve all these different cases of where to store what?

Why can't we just say

• root/sourcedata --> out of scanner data
• root/raw --> raw data in BIDS
• root/derivatives --> anything starting with preprocessing, in BIDS
• If you have only one of those three, adding a second level folder is not necessary.

[yarikoptic]

There is a lot to discuss and unpack. And I do like the clarity of sourcedata (non-BIDS), raw (BIDS), derivatives (BIDS). But, then immediately questions come up, in particular with account that "study" BIDS dataset is just a "base" case of any BIDS dataset, just without direct data (sub-*) in it. Questions:

• where do we place non-BIDS derivatives?
• where, for a BIDS derivative dataset, do we place BIDS input datasets used to produce it
  • when they are "raw BIDS"
  • when they are "derivative BIDS"
  • when they are a mix of both

My point, is that ATM sourcedata/ and derivatives/ separation and lack of further formalization under them is made to foster flexibility. That is where any BIDS dataset can place relevant source data (from scanner or other BIDS datasets, BIDS raw or not) under sourcedata/ and then store produced derivatives (some of which might be BIDS) produced either from sourcedata/ ones (in case of "study") and/or current BIDS dataset itself (if not "study") - under derivatives/.
sourcedata/raw within study is just a suggestion since derivatives/ are for "derived from BIDS datasets" hence "raw BIDS" cannot be there by definition.

Going back to items

• 1: it does not need to be all or nothing: "study" DatasetType just follows what BIDS defines already for "raw" and "derivative" types -- basic folder structure, files and metadata at the root. As soon as we formalize sourcedata/ or derivatives/ more -- it would benefit from that too. Similarly BEP on stimuli tries to further formalize stimuli/ folder.
• 2: sorry -- not following there, may be partially due to weak knowhow on phenotype/
• 3: what is StudyType -- we do not have that in bids yet I think?

[julia-pfarr]

For the sake of moving forward with this PR, let’s only review and approve/disapprove of the changes made by @nikhil153 and move this bigger discussion to a dedicated meeting (maybe at distribits?). Wdyt @yarikoptic and @effigies, could you do a final review on Nikhil's changes?

---

Still, I'll post my answers here, to use this PR as the starting point for following discussions (and that I don't forget what I wanted to say...):

1. I actually did not mean an "all or nothing" approach but my thoughts were very much like yours. At the very base, I assume bids-study means I should follow BIDS wherever is possible. Meaning, that if I have data in the derivatives folder, it should follow a BIDS spec IF there is one. If there is none, then it’s not in BIDS but that’s fine because we don’t have a spec for this (yet). So derivatives can hold both, non-BIDS and BIDS derivatives.
   My point here was more about if we need a bids- prefix for the folders or not. So "it is BIDS" is the umbrella assumption for everything that is in the study but not a hard requirement.

2. This was just me voting for having a /phenotype folder at root level. Since the current phenotype BEP does not make any layout suggestions for /phenotype within bids-study, I feel we have some freedom here to decide where the /phenotype folder should/could be. I did not see anything in the current version of the phenotype BEP that would be violated by having /phenotype at root level in the bids-study layout.

3. No, we don’t have that yet. We can talk about this another time and separate this discussion from this one!

Your other questions:

where do we place non-BIDS derivatives?

This I addressed in my reply to 1.

where, for a BIDS derivative dataset, do we place BIDS input datasets used to produce it
* when they are "raw BIDS"
* when they are "derivative BIDS"
* when they are a mix of both

This is a point where you and I (and probably BIDS and I) don’t use the same meaning for sourcedata.

• Sourcedata for me is any data in it’s barest form, like right after its „birth“. And only this.
• Raw for me is non-processed data but data that is in a standard and interoperable format.
• Derivative for me is everything that was processed, no matter if it was processed directly from sourcedata or raw or other derivatives.

For me it does not make a difference if this dataset was used as an input, it is NOT sourcedata if it is in the raw or derivatives form. Whereas you use sourcedata for anything that was an input for something, no matter of its form.

So, for me, the answers would be:

• „where do we place BIDS input datasets used to produce it when they are „raw BIDS““ —> in /raw
• „where do we place BIDS input datasets used to produce it when they are „derivative BIDS““ —> in /derivatives
• „where do we place BIDS input datasets used to produce it when they are a mix of both“ —> separated in /raw and /derivatives.

We do have the Sources metadata for the derivatives, so it shouldn’t be an issue if inputs are in different places.

[jbpoline]

2cents : the very flexible specification comes at the cost of loss of useful information in the standard, I would certainly make "raw"/"rawbids" the only (first?) place to search for a BIDS validated dataset, that would be useful for the tools and mental clarity : source data == not BIDS, raw/rawbids == BIDS

[CPernet]

I like the idea, obviously, but I wonder how this conflicts with the existing structure —i.e., how the validator behaves and whether 'we' want to differentiate cases. Concretely,

MyData/
      |- source
      |- derivatives
      |- sub-01

vs

MyStudy/
       |- bids-source
       |- bids-derivatives
       |- bids-raw
                  |- sub-01

• source and bids-source are ignored
• derivatives and bids-derivatives are validated for whatever parts follow BIDS
• MyData (a BIDS dataset) is validated to same as bids-raw except for Modality agnostic files
• MyStudy has an extra validation ? and Modality agnostic files are at this level

[julia-pfarr]

@yarikoptic @effigies how is it going with this PR?

[effigies]

Regarding phenotype, pulling it out of the primary dataset seems to run counter to BEP36, which aims to put additional constraints on phenotype. Because many of those rules hinge on relationships between phenotype files and root-level participants.tsv/sessions.tsv, I think this would relax those new rules almost entirely.

I also have a pair of questions:

1. In each of sourcedata/, rawdata/ and derivatives/, is the contents of each of these directories supposed to be a dataset, or are they a collection of datasets, or are they potentially an arbitrary directory structure with datasets appearing somewhere within? Just to illustrate my meaning:

   study/
     sourcedata/
       dicoms/ <- second level
       physio/  <- second level
     rawdata/
       dataset_description.json <- first level
     derivatives/
       subdirA/
         subdirB/
           dataset_description.json <- third level
       subdirC/
         subdirD/
           subdirE/
             dataset_description.json <- fourth level
   

2. Is a validator expected to find BIDS datasets in any/all of these subdirectories and validate them, or do these all remain opaque, and it's the responsibility of the curator/archive to validate subdatasets?

The simple answers: 1) The contents are arbitrary (at least as far as this PR is concerned); 2) No (at least as far as this PR is concerned).

You could go a bit further and say that rawdata/ is expected to be a BIDS dataset, which is a very straightforward directive for a toolwriter to follow (see study, look in rawdata/), but that then rules out the possibility of multiple raw datasets.

I realize these aren't problems Nikhil is introducing, but up to now, there has been a validator behavior for sourcedata/ and derivatives/: ignore their contents. When re-introducing rawdata/, it's an opportunity to think through whether BIDS has anything to say inside these directories.

[nikhil153]

@effigies,
Re: phenotype, if that's the case, then wouldn't it be an issue for the derivative DatasetType as well? Happy to discuss the implication in the BEP36 thread, as I am not sure if it currently considers the newer DatasetTypes (i.e. derivative and study).

Re: questions, my view would be:
sourcedata : arbitrary
rawdata: single BIDS raw DatasetType (i.e. no collection of datasets)
derivatives: non-nested (i.e. no derivatives within derivatives) collection of BIDS derivative DatasetType.

I think allowing multiple raw datasets would create several complications that are better discussed at a later time - possibly under BIDS-MEGA BEP / BIDS2.0?
For derivatives directory, allowing nested sub-directory structure as in your example is okay. As long there there is no nesting of Datasets themselves i.e. no nesting of dataset_description.json files. Hopefully, this would allow enough flexibility for pipeline outputs while avoiding recursion and keeping the validator logic relatively simple.

[julia-pfarr]

@bids-maintenance please look at this PR and discussion and comment your thoughts! Thank you <3

[neuromechanist]

Late to the party, the change LGTM since:
Common Principles document defines source vs. raw vs. derivative data. Having raw directory (as raw, bidsraw, or rawdata) provides parity.

However, as a result the text under Study Dataset needs to be revised as well to match the suggested structure here.

[effigies]

Sorry for the long silence.

The reason for removing phenotype directory was that it would presumably appear in the raw dataset. But isn't that true for derivatives directory as well? It's a bit confusing having derivatives part of all three DatasetTypes but enforcing phenotype only be part of raw or derivative. In practice, phenotype data curation is likely to be more independent than that of derivatives, which is one of the reasons we prefer it to be at the top level.

So the logic in derivatives is that any raw dataset is a valid derivative dataset, so you cannot restrict, you can only expand. Furthermore, you can have derivatives of derivatives, so nesting is entirely reasonable. I suppose you could preprocess phenotypic data and include it in a derivative, so it might even make sense.

IMO, if phenotypic data is worth including in the study, it's worth including in the raw data, regardless of the independence of curation. I still don't see the use case for phenotype at the study level.

That said, I won't block this.

[julia-pfarr]

@effigies do you approve? @bids-maintenance could someone else take another look, since we need 2 approvals

[effigies]

LGTM. @yarikoptic?

[bendhouseart]

If the point of the rawdata/ folder is to hold bids data that is used to construct the parent dataset, then why can't we call it datasets/ and use the dataset description to describe the dataset type with rawdata, derivative, pheno etc?

Then, the parent dataset could be understood to have been constructed from any number of child datasets (raw or derivative) and further outputs from that parent dataset go in into derivative/ and so on and so forth?

I don't believe we ever define dataset in the spec, but maybe this is an opportunity to do so and have a flexible layout?

[effigies]

Given that the latest response was to the rawdata/ proposal, I want to try to reestablish the state of this discussion.

In February, we had an offline conversation that led to changing the proposal to rawbids/, but the reasoning wasn't shared. Here is my recollection of the discussion:

• Right now, you can stick source datasets (BIDS or pre-BIDS) in sourcedata/ and derivative datasets in derivatives/. For raw and derivative datasets, sourcedata/ and derivative/ are relative to the current dataset's contents.
• There are three problems with the status quo:
  1. For BIDS-Study datasets, the line between sourcedata/ and derivative/ is up to the curator. There is no blessed location for the BIDS dataset that is the subject of the study.
  2. When training people to use nipoppy (the project that was the original inspiration of of BIDS-Study, and which hopes to be able to fully adopt BIDS-Study), there was pretty strong resistance among trainees to use sourcedata/raw for the raw BIDS datasets. Due to the example of sourcedata/+rawdata/+derivatives/ that was in historical drafts of the BIDS spec, rawdata/ was familiar and made sense to users.
  3. rawdata/ is a bad name, with "raw" having different connotations for different people.

Because "raw BIDS" is a phrase that appears in Common principles and BIDS Derivatives, rawbids/ was proposed as being a compromise that addresses all three problems:

1. It unambiguously identifies the topic of a BIDS-Study dataset.
2. It is close enough to the historical rawdata/ example that it should face less user resistance.
3. It clarifies that "raw" is being used according to BIDS' idiosyncratic usage.

Finally, for the sake of consistency, rawbids/ is permitted for derivative datasets, while it is not meaningful for raw datasets.

Hopefully others will add to this if I have missed something.

[yarikoptic]

LGTM.

Thanks all the sharing pain and joy!

[nikhil153]

@effigies - thanks for the apt summary!

@effigies, @yarikoptic and other @bids-maintenance - greatly appreciate your efforts, patient explanations, and interesting conversations on the messy business of maintaining standards!

@michellewang, @mathdugre, @julia-pfarr, @nburgos, @AliceJoubert, @Adam-Ismaili-92, @surchs, @jbpoline - thanks for your feedback in shaping and fine-tuning this PR.

We look forward to the convergence of nipoppy and BIDS-Study that would grow the community adoption further!