feat: support tabular onlist projection

docs/CHANGELOG.md

@@ -19,11 +19,13 @@ authors:
 - `seqspec check` now emits warning diagnostics for overlapping read geometry, with guidance to use `seqspec index --no-overlap` when needed.
 - `seqspec check --skip external` runs schema and structural checks without accessing onlist or read resources.
 - A generated examples site under `docs/examples/site`, with rendered assay reports, read templates, region templates, and a searchable assay catalog.
+- Optional `sequence_column_index` and `skip_rows` onlist fields for extracting sequences from whitespace-delimited tabular source files.
 
 ### Changed
 
 - `seqspec upgrade` now upgrades `0.3.0` specs to `0.4.0` in both implementations.
 - Python and Rust now share the same core command surface for `auth`, `check`, `find`, `file`, `format`, `index`, `info`, `init`, `insert`, `methods`, `modify`, `onlist`, `print`, `split`, `upgrade`, and `version`.
+- Python and Rust apply the same onlist row-skipping and column-selection behavior to local, remote, and gzipped sources.
 - `seqspec build` is deprecated in both CLIs and remains as a compatibility stub.
 - Older specs are loaded more permissively before upgrade, which makes `0.2.x` and `0.3.x` specs easier to normalize.
 - `seqspec onlist -s region-type` now errors when matches span multiple reads in a modality. Use `-s read` or `-s region` to disambiguate.

docs/SEQSPEC_TOOL.md

@@ -635,7 +635,7 @@ spec = load_spec("spec.yaml")
 get_onlists(spec, modality="rna", selector="region-type", id="barcode")
 ```
 
-- optionally, `-o OUT` when set with `-f`, writes the joined onlist to this file; when set without `-f`, downloads remote onlists locally and prints paths.
+- optionally, `-o OUT` when set with `-f`, writes the joined onlist to this file; when set without `-f`, downloads remote onlists locally and prints paths. Local or remote onlists with `sequence_column_index` or `skip_rows` are written as normalized sequence-only files.
 - `-m MODALITY` is the modality in which you are searching for the region.
 - `-i ID` is the `id` of the object to search for the onlist.
 - `-s SELECTOR` is the type of the `id` of the object (default: read). Can be one of:
@@ -648,6 +648,11 @@ get_onlists(spec, modality="rna", selector="region-type", id="barcode")
 - optionally, `--auth-profile PROFILE` uses a named auth profile for protected remote onlists.
 - `yaml` corresponds to the `seqspec` file and may be plain YAML or `.yaml.gz`.
 
+Onlist rows are split on arbitrary whitespace. The optional onlist fields
+`sequence_column_index` (zero-based, default `0`) and `skip_rows` (default `0`)
+select sequences from tabular sources while retaining the source URL and checksum
+in the specification.
+
 _Note_: `-s region-type` is only valid when the matching regions come from one read geometry. If the same `region_type` appears across multiple reads in the modality, `seqspec onlist` errors and asks you to use `-s read` or `-s region` instead.
 
 ### Examples

docs/SPECIFICATION.md

@@ -179,6 +179,8 @@ Each `Region` has the following properties which are useful to annotate the elem
   - `url`: a freeform string that specifies either the url location of the file, or the local path of the file (relative to this seqspec file)
   - `urltype`: can be one of ["local", "ftp", "http", "https"] specifies the type of the `url`
   - `md5`: the md5sum of the uncompressed file in `filename`, must match the pattern `^[a-f0-9]{32}$`
+  - `sequence_column_index`: optional zero-based index of the whitespace-delimited field containing each sequence; defaults to `0`
+  - `skip_rows`: optional number of physical rows to skip before reading sequences; defaults to `0`
 - `regions` can either be `null` or contain a list of `regions` as specified above.
 
 Example:
@@ -202,6 +204,27 @@ onlist: !Onlist
 regions: null
 ```
 
+Tabular source files can be used without creating a sequence-only derivative. For
+example, an onlist whose first row is a header and whose second field contains the
+sequence uses:
+
+```yaml
+onlist:
+  file_id: vendor-plate.tsv
+  filename: vendor-plate.tsv
+  filetype: tsv
+  filesize: 120
+  url: https://example.org/vendor-plate.tsv
+  urltype: https
+  md5: 5b62453df2771f5aa856f78797f16591
+  sequence_column_index: 1
+  skip_rows: 1
+```
+
+Rows are split on arbitrary whitespace after the requested physical rows are
+skipped. The `md5` continues to describe the referenced source file, before this
+projection is applied.
+
 For more information about the various fields, please see the JSON schema specification (`seqspec/schema/seqspec.schema.json`). For consistency across assays I suggest following a standard naming conventions for common regions. I've made a collection of "named" regions available; please see `docs/examples/regions` for a list of example regions.
 
 ## `Read` Object

seqspec/Region.py

@@ -30,6 +30,8 @@ class Onlist(BaseModel):
     url: str
     urltype: str
     md5: str
+    sequence_column_index: int = Field(default=0, ge=0)
+    skip_rows: int = Field(default=0, ge=0)
 
     # add a update_spec attribute that computes the md5 for the object
 
@@ -82,6 +84,21 @@ class OnlistInput(BaseModel):
         default=None,
         description=("MD5 checksum of the on-list file if available; omit if unknown."),
     )
+    sequence_column_index: Optional[int] = Field(
+        default=None,
+        ge=0,
+        description=(
+            "Zero-based whitespace-delimited field index containing each sequence. "
+            "Defaults to 0."
+        ),
+    )
+    skip_rows: Optional[int] = Field(
+        default=None,
+        ge=0,
+        description=(
+            "Number of physical rows to skip before reading sequences. Defaults to 0."
+        ),
+    )
 
     def to_onlist(self) -> Onlist:
         return Onlist(
@@ -92,6 +109,8 @@ def to_onlist(self) -> Onlist:
             url=self.url or "",
             urltype=self.urltype or "local",
             md5=self.md5 or "",
+            sequence_column_index=self.sequence_column_index or 0,
+            skip_rows=self.skip_rows or 0,
         )
 
 

seqspec/_core.pyi

@@ -33,6 +33,8 @@ class Onlist:
     url: str
     urltype: str
     md5: str
+    sequence_column_index: int
+    skip_rows: int
 
     def __init__(
         self,
@@ -43,6 +45,8 @@ class Onlist:
         url: str,
         urltype: str,
         md5: str,
+        sequence_column_index: int,
+        skip_rows: int,
     ) -> None: ...
     @staticmethod
     def from_json(json_str: str) -> "Onlist": ...

seqspec/schema/seqspec.schema.json

@@ -409,6 +409,18 @@
               "description": "md5sum for the file pointed to by filename",
               "type": "string",
               "pattern": "^[a-f0-9]{32}$"
+            },
+            "sequence_column_index": {
+              "description": "Zero-based whitespace-delimited field index containing each sequence. Defaults to 0.",
+              "type": "integer",
+              "minimum": 0,
+              "default": 0
+            },
+            "skip_rows": {
+              "description": "Number of physical rows to skip before reading sequences. Defaults to 0.",
+              "type": "integer",
+              "minimum": 0,
+              "default": 0
             }
           },
           "required": [

seqspec/seqspec_onlist.py

@@ -242,19 +242,20 @@ def download_onlists_to_path(
 
     for onlist in onlists:
         if onlist.urltype == "local":
-            # Local file - just return the path
             local_path = base_path / Path(local_onlist_locator(onlist))
-            downloaded_paths.append({"file_id": onlist.file_id, "url": str(local_path)})
+            if onlist.sequence_column_index == 0 and onlist.skip_rows == 0:
+                downloaded_paths.append(
+                    {"file_id": onlist.file_id, "url": str(local_path)}
+                )
+                continue
+            onlist_elements = read_local_list(onlist, str(base_path))
         else:
-            # Remote file - download it
             onlist_elements = read_remote_list(onlist, auth_profile=auth_profile)
-            # Create unique filename for this onlist
-            filename = f"{onlist.file_id}_{output_path.name}"
-            download_path = output_path.parent / filename
-            write_onlist(onlist_elements, download_path)
-            downloaded_paths.append(
-                {"file_id": onlist.file_id, "url": str(download_path)}
-            )
+
+        filename = f"{onlist.file_id}_{output_path.name}"
+        download_path = output_path.parent / filename
+        write_onlist(onlist_elements, download_path)
+        downloaded_paths.append({"file_id": onlist.file_id, "url": str(download_path)})
 
     return downloaded_paths
 

seqspec/utils.py

@@ -525,9 +525,20 @@ def write_pydantic_to_file_or_stdout(
         print(yaml.dump(dump, sort_keys=False))
 
 
-def yield_onlist_contents(stream):
-    for line in stream:
-        yield line.strip().split()[0]
+def yield_onlist_contents(stream, sequence_column_index: int, skip_rows: int):
+    for row_number, line in enumerate(stream, start=1):
+        if row_number <= skip_rows:
+            continue
+
+        fields = line.strip().split()
+        if not fields:
+            continue
+        if sequence_column_index >= len(fields):
+            raise ValueError(
+                f"onlist row {row_number} has {len(fields)} field(s); "
+                f"cannot select zero-based column index {sequence_column_index}"
+            )
+        yield fields[sequence_column_index]
 
 
 def local_resource_url(url: str, filename: str, resource: str) -> str:
@@ -551,7 +562,9 @@ def read_local_list(onlist: Onlist, base_path: str = "") -> List[str]:
     stream = io.TextIOWrapper(stream)
 
     results = []
-    for i in yield_onlist_contents(stream):
+    for i in yield_onlist_contents(
+        stream, onlist.sequence_column_index, onlist.skip_rows
+    ):
         results.append(i)
     stream.close()
     return results
@@ -587,7 +600,9 @@ def read_remote_list(
         stream = io.TextIOWrapper(binary_stream)
 
         results = []
-        for i in yield_onlist_contents(stream):
+        for i in yield_onlist_contents(
+            stream, onlist.sequence_column_index, onlist.skip_rows
+        ):
             # add the new line when writing to file
             results.append(i)
     finally:

src/compat.rs

@@ -268,6 +268,8 @@ pub struct CompatOnlist {
     pub url: Option<String>,
     pub urltype: Option<String>,
     pub md5: Option<String>,
+    pub sequence_column_index: Option<usize>,
+    pub skip_rows: Option<usize>,
 }
 
 impl CompatOnlist {
@@ -280,6 +282,8 @@ impl CompatOnlist {
             self.url.unwrap_or_default(),
             self.urltype.unwrap_or_else(|| "local".to_string()),
             self.md5.unwrap_or_default(),
+            self.sequence_column_index.unwrap_or_default(),
+            self.skip_rows.unwrap_or_default(),
         )
     }
 }
@@ -362,3 +366,29 @@ impl CompatRegion {
         )
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_compat_onlist_preserves_projection_fields() {
+        let yaml = r#"
+file_id: plate
+filename: plate.tsv
+filetype: tsv
+filesize: 42
+url: plate.tsv
+urltype: local
+md5: abc123
+sequence_column_index: 1
+skip_rows: 1
+"#;
+        let compat: CompatOnlist = serde_yaml::from_str(yaml).unwrap();
+
+        let onlist = compat.into_onlist();
+
+        assert_eq!(onlist.sequence_column_index, 1);
+        assert_eq!(onlist.skip_rows, 1);
+    }
+}

src/models/onlist.rs

@@ -9,6 +9,10 @@ pub struct Onlist {
     pub url: String,
     pub urltype: String,
     pub md5: String,
+    #[serde(default)]
+    pub sequence_column_index: usize,
+    #[serde(default)]
+    pub skip_rows: usize,
 }
 
 impl Onlist {
@@ -20,6 +24,8 @@ impl Onlist {
         url: String,
         urltype: String,
         md5: String,
+        sequence_column_index: usize,
+        skip_rows: usize,
     ) -> Self {
         Self {
             file_id,
@@ -29,6 +35,8 @@ impl Onlist {
             url,
             urltype,
             md5,
+            sequence_column_index,
+            skip_rows,
         }
     }
 
@@ -53,6 +61,8 @@ mod tests {
             "barcodes.txt".into(),
             "local".into(),
             "abc123".into(),
+            0,
+            0,
         )
     }
 
@@ -66,16 +76,38 @@ mod tests {
         assert_eq!(ol.url, "barcodes.txt");
         assert_eq!(ol.urltype, "local");
         assert_eq!(ol.md5, "abc123");
+        assert_eq!(ol.sequence_column_index, 0);
+        assert_eq!(ol.skip_rows, 0);
     }
 
     #[test]
     fn test_onlist_json_roundtrip() {
-        let ol = sample_onlist();
+        let mut ol = sample_onlist();
+        ol.sequence_column_index = 1;
+        ol.skip_rows = 1;
         let json = ol.to_json().unwrap();
         let ol2 = Onlist::from_json(&json).unwrap();
         assert_eq!(ol, ol2);
     }
 
+    #[test]
+    fn test_onlist_json_defaults_projection_for_legacy_data() {
+        let json = r#"{
+            "file_id":"ol1",
+            "filename":"barcodes.txt",
+            "filetype":"txt",
+            "filesize":1024,
+            "url":"barcodes.txt",
+            "urltype":"local",
+            "md5":"abc123"
+        }"#;
+
+        let onlist = Onlist::from_json(json).unwrap();
+
+        assert_eq!(onlist.sequence_column_index, 0);
+        assert_eq!(onlist.skip_rows, 0);
+    }
+
     #[test]
     fn test_onlist_clone() {
         let ol = sample_onlist();

src/models/region.rs

@@ -562,6 +562,8 @@ mod tests {
             "list.txt".into(),
             "local".into(),
             "".into(),
+            0,
+            0,
         );
         let mut r = Region::new(
             "r".into(),
@@ -635,6 +637,8 @@ mod tests {
             "".into(),
             "local".into(),
             "".into(),
+            0,
+            0,
         );
         let parent = joined(
             "parent",
@@ -671,6 +675,8 @@ mod tests {
             "".into(),
             "local".into(),
             "".into(),
+            0,
+            0,
         );
         let r2 = Region::new(
             "bc".into(),

src/schema/seqspec.schema.json

@@ -409,6 +409,18 @@
               "description": "md5sum for the file pointed to by filename",
               "type": "string",
               "pattern": "^[a-f0-9]{32}$"
+            },
+            "sequence_column_index": {
+              "description": "Zero-based whitespace-delimited field index containing each sequence. Defaults to 0.",
+              "type": "integer",
+              "minimum": 0,
+              "default": 0
+            },
+            "skip_rows": {
+              "description": "Number of physical rows to skip before reading sequences. Defaults to 0.",
+              "type": "integer",
+              "minimum": 0,
+              "default": 0
             }
           },
           "required": [