Skip to content

Curl manifest IT fails on AnVIL #8063

Open
Defect
#8058
Open
Curl manifest IT fails on AnVIL#8063
Defect
#8058
Assignees
hannes-ucsc
Labels
service[subject] The service part of Azultest[subject] Unit and integration test code

Description

@achave11-ucsc
achave11-ucsc
opened on May 29, 2026

https://gitlab.explore.anvilproject.org/ucsc/azul/-/jobs/67063

2026-05-29 19:17:10,617    INFO MainThread test.integration_test: Beginning sub-test [curl_manifest] {'catalog': 'anvil13-it'}
2026-05-29 19:17:10,618    INFO MainThread test.integration_test: Looking for FASTQ files up to 536870912 bytes in size
2026-05-29 19:17:10,619    INFO MainThread azul.oauth2: Making GET request to 'https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil13-it&filters=%7B%22files.file_format%22%3A+%7B%22is%22%3A+%5B%22.fastq%22%2C+%22.fastq.gz%22%5D%7D%2C+%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size'
2026-05-29 19:17:10,619    INFO MainThread azul.oauth2: … without a request body
2026-05-29 19:17:10,620    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDxyu')]
2026-05-29 19:17:10,985    INFO MainThread azul.oauth2: Got 200 response after 0.366s from GET to https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil13-it&filters=%7B%22files.file_format%22%3A+%7B%22is%22%3A+%5B%22.fastq%22%2C+%22.fastq.gz%22%5D%7D%2C+%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size
2026-05-29 19:17:10,985    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '2380', 'Connection': 'keep-alive', 'Date': 'Fri, 29 May 2026 19:17:10 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a19e636-7634029d0287e2292ef8e604;Parent=70026fd95fc63f9c;Sampled=0;Lineage=1:c1bd52b0:0', 'x-amzn-RequestId': 'a7a769bb-adfd-4b37-ab0c-ee20325f3248', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'eJDomE-DoAMEsGg=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 1e5d213252e38a8aa2e2a927fd3c3754.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD55-P8', 'X-Amz-Cf-Id': 'v5LNfi7svHQecqCEiXO1n_EcYqDcTlUfCtKIXvV9f9-0FMy5pFP0LA=='})
2026-05-29 19:17:10,986    INFO MainThread azul.oauth2: … with a response body of length 2380 starting in b'{"hits":[],"pagination":{"count":0,"total":0,"size":100,"next":null,"previous":null,"pages":0,"sort":"files.file_size","order":"asc"},"termFacets":{"diagnoses.phenotype":{"terms":[],"total":0,"type":"terms"},"biosamples.disease":{"terms":[],"total":0,"type":"terms"},"diagnoses.disease":{"terms":[],"total":0,"type":"terms"},"diagnoses.phenopacket":{"terms":[],"total":0,"type":"terms"},"files.is_supplementary":{"terms":[],"total":0,"type":"terms"},"files.data_modality":{"terms":[],"total":0,"type":"terms"},"donors.reported_ethnicity":{"terms":[],"total":0,"type":"terms"},"datasets.consent_group":{"terms":[],"total":0,"type":"terms"},"activities.assay_type":{"terms":[],"total":0,"type":"terms"},"datasets.title":{"terms":[],"total":0,"type":"terms"},"biosamples.anatomical_site":{"terms":[],"total":0,"type":"terms"},"donors.organism_type":{"terms":[],"total":0,"type":"terms"},"datasets.data_use_permission":{"terms":[],"total":0,"type":"terms"},"donors.phenotypic_sex":{"terms":[],"total":0,"type":"terms"},"activ...'
2026-05-29 19:17:10,986    INFO MainThread test.integration_test: Looking for any files up to 536870912 bytes in size
2026-05-29 19:17:10,988    INFO MainThread azul.oauth2: Making GET request to 'https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil13-it&filters=%7B%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size'
2026-05-29 19:17:10,988    INFO MainThread azul.oauth2: … without a request body
2026-05-29 19:17:10,989    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDxyu')]
2026-05-29 19:17:11,622    INFO MainThread azul.oauth2: Got 200 response after 0.634s from GET to https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil13-it&filters=%7B%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size
2026-05-29 19:17:11,622    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '147324', 'Connection': 'keep-alive', 'Date': 'Fri, 29 May 2026 19:17:11 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a19e636-3396eebc200185281c7074be;Parent=3768788d632d2aaa;Sampled=0;Lineage=1:c1bd52b0:0', 'x-amzn-RequestId': '5deb4f8e-6438-4e9b-9bd2-5817d4001efc', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'eJDopHELoAMESRQ=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 1e5d213252e38a8aa2e2a927fd3c3754.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD55-P8', 'X-Amz-Cf-Id': 'HcM4PyNorhX_Is8vWnMDB2kJZcb58xpbKeaEq-kXoyoUf74ed5Tmdw=='})
2026-05-29 19:17:11,622    INFO MainThread azul.oauth2: … with a response body of length 147324 starting in b'{"hits":[{"entryId":"34a39d5d-0033-4478-abba-18e1cef130a6","sources":[{"source_prefix":"3/1","source_spec":"tdr:bigquery:gcp:datarepo-0a4ee218:ANVIL_HPRC_20260121_ANV5_202601222125","source_id":"55785ded-42a6-4808-ac64-b1a0928f8ac8"}],"bundles":[{"bundle_uuid":"87fd6129-37c5-a115-8d98-ff4ae69ffc15","bundle_version":"2022-06-01T00:00:00.000000Z"}],"activities":[],"biosamples":[],"datasets":[{"dataset_id":["59960255-cca9-fcd3-d8ca-eb88f2c72f00"],"consent_group":["NRES"],"data_use_permission":["NRES"],"registered_identifier":[null],"title":["ANVIL_HPRC"],"data_modality":["genomic"]}],"diagnoses":[],"donors":[],"files":[{"document_id":"34a39d5d-0033-4478-abba-18e1cef130a6","source_datarepo_row_ids":["file_inventory:0a55b6c5-ad26-469c-ab2b-6e522a513f9d"],"file_id":"ed5fb718-88de-3acd-9ac5-add9cb3ec9f8","data_modality":[null],"file_format":".hist","file_size":19,"file_md5sum":"d79a3e5d68ede9e7f9e8934e510b9c53","reference_assembly":[null],"file_name":"HG03453.f1_assembly_v2.merqury.asm.only.hist","is_supplementar...'
2026-05-29 19:17:11,627    INFO MainThread test.integration_test: Failed sub-test [curl_manifest] {'catalog': 'anvil13-it'}
  test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [curl_manifest] (catalog='anvil13-it') ... FAIL
…
2026-05-29 19:19:19,594    INFO MainThread test.integration_test: Beginning sub-test [curl_manifest] {'catalog': 'anvil14-it'}
2026-05-29 19:19:19,594    INFO MainThread test.integration_test: Looking for FASTQ files up to 536870912 bytes in size
2026-05-29 19:19:19,596    INFO MainThread azul.oauth2: Making GET request to 'https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil14-it&filters=%7B%22files.file_format%22%3A+%7B%22is%22%3A+%5B%22.fastq%22%2C+%22.fastq.gz%22%5D%7D%2C+%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size'
2026-05-29 19:19:19,596    INFO MainThread azul.oauth2: … without a request body
2026-05-29 19:19:19,596    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDxyu')]
2026-05-29 19:19:19,952    INFO MainThread azul.oauth2: Got 200 response after 0.356s from GET to https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil14-it&filters=%7B%22files.file_format%22%3A+%7B%22is%22%3A+%5B%22.fastq%22%2C+%22.fastq.gz%22%5D%7D%2C+%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size
2026-05-29 19:19:19,952    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '1597', 'Connection': 'keep-alive', 'Date': 'Fri, 29 May 2026 19:19:19 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a19e6b7-21ca2da93f90ae5e19ee6290;Parent=23889dd605c66d89;Sampled=0;Lineage=1:c1bd52b0:0', 'x-amzn-RequestId': '2ff6af0d-4b20-4db1-8712-c20eb23ad360', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'eJD8wGBIIAMEDww=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 1e5d213252e38a8aa2e2a927fd3c3754.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD55-P8', 'X-Amz-Cf-Id': 't7crcrKx2vhptB_8Vl5iHYxDn6ExF5AsRHI44KAVpeekmVrg4LkyTA=='})
2026-05-29 19:19:19,952    INFO MainThread azul.oauth2: … with a response body of length 1597 starting in b'{"hits":[],"pagination":{"count":0,"total":0,"size":100,"next":null,"previous":null,"pages":0,"sort":"files.file_size","order":"asc"},"termFacets":{"diagnoses.phenotype":{"terms":[],"total":0,"type":"terms"},"biosamples.disease":{"terms":[],"total":0,"type":"terms"},"diagnoses.disease":{"terms":[],"total":0,"type":"terms"},"diagnoses.phenopacket":{"terms":[],"total":0,"type":"terms"},"files.is_supplementary":{"terms":[],"total":0,"type":"terms"},"files.data_modality":{"terms":[],"total":0,"type":"terms"},"donors.reported_ethnicity":{"terms":[],"total":0,"type":"terms"},"datasets.consent_group":{"terms":[],"total":0,"type":"terms"},"activities.assay_type":{"terms":[],"total":0,"type":"terms"},"datasets.title":{"terms":[],"total":0,"type":"terms"},"biosamples.anatomical_site":{"terms":[],"total":0,"type":"terms"},"donors.organism_type":{"terms":[],"total":0,"type":"terms"},"datasets.data_use_permission":{"terms":[],"total":0,"type":"terms"},"donors.phenotypic_sex":{"terms":[],"total":0,"type":"terms"},"activ...'
2026-05-29 19:19:19,952    INFO MainThread test.integration_test: Looking for any files up to 536870912 bytes in size
2026-05-29 19:19:19,954    INFO MainThread azul.oauth2: Making GET request to 'https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil14-it&filters=%7B%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size'
2026-05-29 19:19:19,954    INFO MainThread azul.oauth2: … without a request body
2026-05-29 19:19:19,954    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDxyu')]
2026-05-29 19:19:20,585    INFO MainThread azul.oauth2: Got 200 response after 0.631s from GET to https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil14-it&filters=%7B%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size
2026-05-29 19:19:20,585    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '167597', 'Connection': 'keep-alive', 'Date': 'Fri, 29 May 2026 19:19:20 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a19e6b7-5051fe7a6838453d6f1e45ed;Parent=3391befa63fe8cd7;Sampled=0;Lineage=1:c1bd52b0:0', 'x-amzn-RequestId': '8c20ec24-f555-4e29-aa27-ce3a8546dafc', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'eJD8zHY3IAMEEzA=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 1e5d213252e38a8aa2e2a927fd3c3754.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD55-P8', 'X-Amz-Cf-Id': '1SHp6dzIURIbekldFnkDIHDldFTvj4WVCCdORkU9LfY5ebk45RYZ9Q=='})
2026-05-29 19:19:20,585    INFO MainThread azul.oauth2: … with a response body of length 167597 starting in b'{"hits":[{"entryId":"ae38890d-1a36-4c75-8aaf-951025c4c1b0","sources":[{"source_prefix":"3e/1","source_spec":"tdr:bigquery:gcp:datarepo-0677c238:ANVIL_IGVF_Mouse_R1_20260416_ANV5_202604162312","source_id":"d82fbf42-7655-4381-962e-c3cb280cccf3"}],"bundles":[{"bundle_uuid":"3e6fff0f-e55f-af58-95bd-0294176300e8","bundle_version":"2022-06-01T00:00:00.000000Z"}],"activities":[{"activity_type":["Sequencing"],"assay_type":[null],"data_modality":[null]}],"biosamples":[{"anatomical_site":["Kidney"],"biosample_type":["Unknown, Tissue"],"disease":[null],"donor_age_at_collection_unit":[null],"donor_age_at_collection":[{"gte":null,"lte":null}]}],"datasets":[{"dataset_id":["3201ae89-576f-04ff-14c5-30fdb1a2ac64"],"consent_group":["NRES"],"data_use_permission":["NRES"],"registered_identifier":["phs003472"],"title":["AnVIL_IGVF_Mouse_R1"],"data_modality":[null]}],"diagnoses":[],"donors":[{"organism_type":["Mouse"],"phenotypic_sex":["Female","Male"],"reported_ethnicity":[null],"genetic_ancestry":[null]}],"files":[{"document_...'
2026-05-29 19:19:20,590    INFO MainThread test.integration_test: Failed sub-test [curl_manifest] {'catalog': 'anvil14-it'}
  test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [curl_manifest] (catalog='anvil14-it') ... FAIL
…
FAIL: test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [curl_manifest] (catalog='anvil13-it')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/ucsc/azul/test/integration_test.py", line 423, in subTest
    yield
  File "/builds/ucsc/azul/test/integration_test.py", line 799, in _test_curl_manifest
    outer_file, inner_file = self._get_one_file(catalog,
                             ~~~~~~~~~~~~~~~~~~^^^^^^^^^
                                                max_size=max_size,
                                                ^^^^^^^^^^^^^^^^^^
                                                downloadable=True)
                                                ^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/test/integration_test.py", line 902, in _get_one_file
    self.fail('No files found')
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: No files found
======================================================================
FAIL: test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [curl_manifest] (catalog='anvil14-it')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/ucsc/azul/test/integration_test.py", line 423, in subTest
    yield
  File "/builds/ucsc/azul/test/integration_test.py", line 799, in _test_curl_manifest
    outer_file, inner_file = self._get_one_file(catalog,
                             ~~~~~~~~~~~~~~~~~~^^^^^^^^^
                                                max_size=max_size,
                                                ^^^^^^^^^^^^^^^^^^
                                                downloadable=True)
                                                ^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/test/integration_test.py", line 902, in _get_one_file
    self.fail('No files found')
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: No files found
----------------------------------------------------------------------
Ran 16 tests in 1557.950s
FAILED (failures=2)
make: *** [Makefile:288: integration_test] Error 1
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

The response's drs_uri field looked as follows:

❯ curl 'https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil14-it&filters=%7B%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size' | jq -r .hits[].files[].drs_uri

drs://drs.anv0:v2_8ff4ac75-c1cf-3ecc-8b90-526298ae624e
drs://drs.anv0:v2_bfb6302e-daef-3269-aff9-469e0b05f29f
drs://drs.anv0:v2_c37dff44-38ac-3bb6-b2d0-9dc659a6a7dd
drs://drs.anv0:v2_44c9b818-13ff-3f05-b6f2-4cf1cdae1f78
drs://drs.anv0:v2_72dfb360-15e3-3dcd-a290-2b2a0c697516
drs://drs.anv0:v2_edc5dc8f-cf75-3e94-9858-0f28d81eea27
drs://drs.anv0:v2_8f1abfcb-4e4e-30ae-a075-db2527a46b57
drs://drs.anv0:v2_3147dd95-1225-3043-9895-1c755b6ed67b

❯ curl 'https://service.hammerbox.explore.anvilproject.org/index/files?catalog=anvil13-it&filters=%7B%22files.file_size%22%3A+%7B%22within%22%3A+%5B%5B1%2C+536870912%5D%5D%7D%7D&size=100&order=asc&sort=files.file_size' | jq -r .hits[].files[].drs_uri

drs://drs.anv0:v2_ed5fb718-88de-3acd-9ac5-add9cb3ec9f8
drs://drs.anv0:v2_1583f5e1-2d79-3a73-ab6e-4145cd6b7673
drs://drs.anv0:v2_a1b7dad9-10f3-3fbb-86f9-8b42ee848204
drs://drs.anv0:v2_ed4c38f6-9b72-3791-901e-80ee7efa6345
drs://drs.anv0:v2_2ff43676-22d4-33cd-9d1d-8bc1c0be728a
drs://drs.anv0:v2_4e253c6d-09d9-37c7-816b-fdf1a29bf061

azul/test/integration_test.py

Lines 899 to 902 in 4d1092b

drs_uri = DRSURI.parse(json_str(inner_file['drs_uri']))
if downloadable == isinstance(drs_uri, HostBasedDRSURI):
return outer_file, inner_file
self.fail('No files found')

Introduced in PR #8010, checks that the response drs_uri is HostBasedDRSURI which isn't the case for the drs_uris in hammerbox and causes the test to fail.

drs_uri = DRSURI.parse('drs://drs.anv0:v2_8ff4ac75-c1cf-3ecc-8b90-526298ae624e')
isinstance(drs_uri, HostBasedDRSURI)
False

Metadata

Metadata

Assignees

Labels

service[subject] The service part of Azultest[subject] Unit and integration test code

Type

Defect

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions