`curl` manifest futilely includes external LungMAP files

[hannes-ucsc]

An external file is a file whose DRS URI

A) is compact and resolves to a host that is not TDR or
B) is host-based and has a host that is not TDR

At the moment we don't have files matching A) so currently, any file with a compact DRS URI is external.

I encountered this in dev so it may not be valid. These files are not downloadable so their azul_mirror_uri and azul_url should be None in the /index/files response and they shouldn't be included in the manifest.

[hannes-ucsc]

Spike to provide reproduction for dev and prod for both /index/files and manifest.

[nadove-ucsc]

Every file in lm2 on dev has either 1. a host-based DRS URI where the host is TDR or 2. a compact DRS URI where the host resolves to gen3.biodatacatalyst.nhlbi.nih.gov.

$ curl 'https://service.dev.singlecell.gi.ucsc.edu/index/files?catalog=lm2&size=350' >dev_lm2_files.json
$ cat dev_lm2_files.json | jq .pagination.total
348
$ cat dev_lm2_files.json | jq .hits[].files[].drs_uri | cut -f3 -d/ | cut -f1 -d: | sort | uniq -c
     74 dg.4503
    274 jade.datarepo-dev.broadinstitute.org

Every file's azul_url is not null. This confirms the bug.

$ cat dev_lm2_files.json | jq .hits[].files[].azul_url | cut -f3 -d/ | uniq -c
    348 service.dev.singlecell.gi.ucsc.edu

Every file's azul_mirror_uri is null, as expected.

$ cat dev_lm2_files.json | jq .hits[].files[].azul_mirror_uri | uniq -c
    348 null

Every file in the catalog is present in an unfiltered manifest. This necessarily includes the external files.

$ cat lm2_dev.curlrc | grep '^url=' | wc -l
348

[nadove-ucsc]

For prod, my analysis of the /index/files response only covers the first 999 out of 5,088 total.

Again, the DRS URIs are a mix of host-based URIs pointing to TDR and compact URIs pointing to biodatacatalyst. Unlike on dev, there are also a lot of null values:

$ cat prod_lm10_files.json | jq .hits[].files[].drs_uri | cut -f3 -d/ | cut -f1 -d: | sort | uniq -c
    450 data.terra.bio
     74 dg.4503
    475 null

The same bug appears to be present with azul_url being set for external files.

$ cat prod_lm10_files.json | jq .hits[].files[].azul_url | cut -f3 -d/ | cut -f1 -d: | sort | uniq -c
    475 null
    524 service.azul.data.humancellatlas.org

Here is an example of such a problematic file. Note the compact identifier. This demonstrates the bug on prod:

$ curl 'https://service.azul.data.humancellatlas.org/index/files/563ab2c6-6a71-474b-859b-206cb49ad5e3?catalog=lm10' | jq '.files[] | {drs_uri, azul_url}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3060  100  3060    0     0   2241      0  0:00:01  0:00:01 --:--:--  2241
{
  "drs_uri": "drs://dg.4503:d184f3fb-4544-4ba8-850f-3ce366e28901",
  "azul_url": "https://service.azul.data.humancellatlas.org/repository/files/563ab2c6-6a71-474b-859b-206cb49ad5e3?catalog=lm10&version=2025-01-07T21%3A14%3A38.102511Z"
}

Following that azul_url leads to

{
      "Code": "UnauthorizedError",
      "Message": "Unexpected response from https://gen3.biodatacatalyst.nhlbi.nih.gov/ga4gh/drs/v1/objects/d184f3fb-4544-4ba8-850f-3ce366e28901/access/gs"
}

As expected, all mirror_uri's are null:

$ cat prod_lm10_files.json | jq .hits[].files[].mirror_uri | uniq -c
    999 null

I created a single-row manifest containing the above file.

{
  "fileId": {
    "is": [
      "563ab2c6-6a71-474b-859b-206cb49ad5e3"
    ]
  }
}

The manifest contains the same non-functional Azul URL:

$ cat lm10_prod.curlrc | grep '^url='
url="https://service.azul.data.humancellatlas.org/repository/files/563ab2c6-6a71-474b-859b-206cb49ad5e3?catalog=lm10&version=2025-01-07T21%3A14%3A38.102511Z"

[dsotirho-ucsc]

@hannes-ucsc: "We're not sure what the estimate is. I'm guessing 4. Assignee to start work and retriage if it turns out to be more than that. "