HATS cloudtests

Integration tests for cloud read and write through HATS and LSDB libraries.

Why have this repo?

We have abstracted our entire i/o infrastructure to be read through the python fsspec library, using universal_pathlib. All that needs to be provided is a valid protocol pathway, and storage options for the cloud interface.

Despite the best efforts of everyone, there are nuances to different file systems. This repo reflects our own attempts to confirm that we're not introducing regressions in the HATS and LSDB code bases against continued support for various file systems.

Dev Guide - Getting Started

Pretty easy, really.

./setup_dev.sh

Performing HATS cloud tests locally

You'll notice that attempting >> pytest on its own doesn't work! You must specify the cloud you're targeting with tests, as fixtures will be different for each test. Some are pretty straightforward, but some require additional steps, documented here.

'local_s3'

A local AWS S3 bucket is created in the s3_server pytest fixture.

This requires NO additional setup, and runs all read and write tests. If you're looking for a quick test of some functionality, this is a good place to start.

'http'

A local HTTP server is instantianted via the http_server pytest fixture.

This requires NO additional setup, but ONLY runs read tests.

'local_gcs'

This test requires a local GCS server running at port 4443. The fake-gcs-server runs on docker, and so should be instantiated outside the pytest context.

You can do so with some lines like (on linux):

sudo snap install docker
sudo docker pull fsouza/fake-gcs-server
sudo docker run -d -p 4443:4443 --name fake-gcs-server -v ${PWD}/tests/cloud:/data fsouza/fake-gcs-server -scheme http -public-host 0.0.0.0:4443 -external-url http://localhost:4443

'abfs'

ABFS tests run against production resources on the Microsoft Azure bucket. In order to run the tests, you will need to (ask someone for the real values) and export the following environment variables in a command line:

export ABFS_LINCCDATA_ACCOUNT_NAME=lincc_account_name
export ABFS_LINCCDATA_ACCOUNT_KEY=lincc_account_key

This is currently disabled because the external bucket doesn't exist anymore.

Adding a new cloud

Ideally, you can add a new cloud test with a few config changes.

1. Pick a nice short name
2. In conftest.py
   1. add to ALL_CLOUDS list.
   2. determine if your cloud is going to be tested for read-only, or also for cloud writing. Add (or don't) to the READ_ONLY_CLOUDS list
   3. create a session-scoped pytest.fixture that creates a running instance of your cloud. should return a kwarg dict for connection.
   4. add case in cloud_path and storage_options methods to return appropriate UPath or kwarg dict.
3. Add cloud-specific test data (there are some tests that use absolute paths, and these will be different for each protocol)
   1. tests/data/indexed_files_<short_name> - new directory, mirroring existing
   2. tests/cloud/data/collection_absolute_paths/coll_<short_name> - new directory, with a single collection.properties file containing absolute paths to catalog tables.

Contributing

See the hats contribution guide for contribution best practices.

Acknowledgements

This project is supported by Schmidt Sciences.

This project is based upon work supported by the National Science Foundation under Grant No. AST-2003196.

This project acknowledges support from the DIRAC Institute in the Department of Astronomy at the University of Washington. The DIRAC Institute is supported through generous gifts from the Charles and Lisa Simonyi Fund for Arts and Sciences, and the Washington Research Foundation.