Transient StorageObjectNotFound exception during info schema IT

[nadove-ucsc]

https://gitlab.explore.anvilproject.org/ucsc/azul/-/jobs/69296

2026-07-08T04:52:36.752628Z 01E 2026-07-08 04:52:36,752    INFO MainThread test.integration_test: Beginning sub-test [validate_info_schemas] {}
2026-07-08T04:52:36.774159Z 01E 2026-07-08 04:52:36,773    INFO MainThread azul.boto3: s3.GetObject:	Making GET request to REDACTED
2026-07-08T04:52:36.774172Z 01E 2026-07-08 04:52:36,773    INFO MainThread azul.boto3: s3.GetObject:	… without a request body
2026-07-08T04:52:36.835741Z 01E 2026-07-08 04:52:36,835    INFO MainThread azul.boto3: s3.GetObject:	Got 404 response
2026-07-08T04:52:36.835868Z 01E 2026-07-08 04:52:36,835    INFO MainThread azul.boto3: s3.GetObject:	… with a response body of length 321 being b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>REDACTED</Key><RequestId>WYETBHDWAC6EYYTN</RequestId><HostId>+pwFEjKieXNXIInnjPJ6+fpamfoYX1Z4GeTtEf1tFySXba6M4BxUkqSxaLa1WUOO1WBKCGgrWKM=</HostId></Error>'
2026-07-08T04:52:36.836312Z 01E 2026-07-08 04:52:36,836    INFO MainThread test.integration_test: Failed sub-test [validate_info_schemas] {}
2026-07-08T04:52:36.836316Z 01E 
2026-07-08T04:52:36.836483Z 01E   test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [validate_info_schemas] ... ERROR

...

======================================================================
ERROR: test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [validate_info_schemas]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/ucsc/azul/src/azul/service/storage_service.py", line 112, in _get_object
    response = self._s3.get_object(Bucket=self.bucket_name,
                                   Key=object_key)
  File "/build/.venv/lib/python3.14/site-packages/botocore/client.py", line 606, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/.venv/lib/python3.14/site-packages/botocore/context.py", line 123, in wrapper
    return func(*args, **kwargs)
  File "/build/.venv/lib/python3.14/site-packages/botocore/client.py", line 1094, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/builds/ucsc/azul/test/integration_test.py", line 434, in subTest
    yield
  File "/builds/ucsc/azul/test/integration_test.py", line 2006, in _test_mirroring
    info_objects = [service.info(file) for file in indexed_files.keys()]
                    ~~~~~~~~~~~~^^^^^^
  File "/builds/ucsc/azul/src/azul/indexer/mirror_service.py", line 524, in info
    return json.loads(storage.get_object(self._info_object_key(file)))
                      ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/service/storage_service.py", line 108, in get_object
    return self._get_object(object_key)['Body'].read()
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/service/storage_service.py", line 115, in _get_object
    raise StorageObjectNotFound(object_key)
azul.service.storage_service.StorageObjectNotFound: REDACTED
----------------------------------------------------------------------
Ran 16 tests in 1304.480s
FAILED (errors=1, skipped=1)

Edit: redacted S3 URI and object key due to an influx of alarms from unauthorized attempts to download the object.

[dsotirho-ucsc]

@hannes-ucsc: "Assignee to provide self-contained description. A mere link isn't sufficient as it might become stale or require additional permissions to access. After that, spike to diagnose."