Garage ansible playbook and publish script

.github/workflows/oats.yml

@@ -153,9 +153,17 @@ jobs:
           require_tests: true
           fail_on_failure: true
 
-      # TODO: This will need updating once the self-hosted Garage storage
-      # service exists For now we just store the text artifacts as the whole
-      # results directory is far too big to store as an artifact.
+      # The whole run goes to the artifact store when one is configured; the
+      # GitHub artifact keeps the small text outputs close to the run page.
+      - name: Publish run to the store
+        if: always() && steps.rundir.outputs.run_dir != '' && vars.OATS_STORE_ENDPOINT != ''
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.OATS_STORE_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.OATS_STORE_SECRET_ACCESS_KEY }}
+        run: |
+          storage/publish_run.sh "${{ steps.rundir.outputs.run_dir }}" s3://oats-runs \
+            --endpoint "${{ vars.OATS_STORE_ENDPOINT }}"
+
       - name: Upload run artifact
         if: always() && steps.rundir.outputs.run_dir != ''
         uses: actions/upload-artifact@v7

README.md

@@ -118,6 +118,12 @@ If you want to aggregate all files into a single directory for ease of view, you
 
 This command will copy all odm_orthophoto.tif files from all test cases into `/my/path`. See `./harvest --help` for more options.
 
+### Publishing runs to durable storage
+
+`results/runs/` is local and prunable. `storage/publish_run.sh` publishes a run
+to the artifact store, which serves it read-only for the comparison viewer. See
+[`storage/README.md`](storage/README.md).
+
 ## Roadmap
 
 We have great plans for OATS. Some of them include:

docs/ci.md

@@ -10,8 +10,9 @@ outside it, namely the runner, the cross-repo token and the ODM-side trigger.
 Register a runner against this repository (*Settings → Actions → Runners*). The
 default `self-hosted` label is all the workflow asks for.
 
-It needs docker, plus `git`, `wget`, `rsync`, `sed`, `unzip` and `jq` on
-`PATH`; `./run` bootstraps bats itself. Give it plenty of RAM —
+It needs docker, plus `git`, `wget`, `rsync`, `sed`, `unzip`, `jq`, and `rclone`
+on `PATH`; `./run` bootstraps bats itself. Give it plenty of RAM —
+
 the suite is RAM-bound and the `all` group sets the ceiling. Each full run
 generates about 60GB of output right now so as a rough guide we should aim for
 about 1TB of disk space.
@@ -65,6 +66,18 @@ fails and the ODM publish job goes red.
 
 ## Results
 
-Currently we store the text outputs as a github artifact which will allow us to
-review any failures. This will be replaced by the s3-compatible Garage server
-once its set up.
+Every run uploads its reports and small text outputs as a GitHub artifact,
+kept for 14 days, which is enough to review a failure from the run page.
+
+The run's primary outputs go to the artifact store described in
+[`storage/README.md`](../storage/README.md) when the store is configured; the
+rest stays under `results/runs/` on the runner. The
+`Publish run to the store` step runs `storage/publish_run.sh`. To configure it,
+under *Settings → Secrets and variables → Actions* on this repository:
+
+- Variable `OATS_STORE_ENDPOINT`: the store's S3 API, e.g. `http://oats-vm.lan:3900`.
+- Secrets `OATS_STORE_ACCESS_KEY_ID` and `OATS_STORE_SECRET_ACCESS_KEY`: the
+  two values the storage playbook prints on its first run.
+
+The runner reaches the endpoint directly, so port 3900 on the store must be
+open to it. The step needs `rclone` and `jq` on the runner.

storage/.gitignore

@@ -0,0 +1 @@
+deploy/inventory.yml

storage/README.md

@@ -0,0 +1,150 @@
+# OATS run artifact storage
+
+Durable storage for OATS run directories, served read-only over HTTP(S).
+
+**Backend: [Garage](https://garagehq.deuxfleurs.fr/)** — S3-compatible API which
+means we keep the storage contract separate from the instance and can swap out
+the backend later if we need more storage etc.
+
+## Store contract
+
+Each run sits at its identity path, laid out as the harness writes it but
+holding only the primary outputs:
+
+```
+oats-runs (bucket root)
+  index.json                                    # every run, newest first
+  <odm_git_short>/<image_key12>/<timestamp>/    # one self-contained run
+    run_manifest.json
+    oats_manifest.tsv
+    reports/*.xml
+    tests/<dataset>/<test>/
+      *.json, *.txt                             # log, options, task output
+      odm_orthophoto/odm_orthophoto.tif
+      odm_dem/*.tif
+      odm_georeferencing/odm_georeferenced_model.laz
+      odm_report/
+```
+
+Input images, OpenSfM/OpenMVS intermediates, meshes and textures stay on the
+runner; the allowlist is `PUBLISH_FILTER` in `publish_run.sh`. A full run is
+about 2GB in the store against 50-60GB on the runner.
+
+Each run is stored under its `run_key` from `run_manifest.json`. `index.json` is
+all the viewer reads to find runs:
+
+```json
+{
+  "generated": "...",
+  "runs": [ {
+    "key": "...",
+    "timestamp": "...",
+    "image": "...",
+    "image_digest": "...",
+    "odm_git_revision": "...",
+    "group": "...",
+    "datasets_total": 2,
+    "passed": 0,
+    "failed": 2
+  } ]
+}
+```
+
+`passed`/`failed` aggregate per-dataset bats exit codes.
+
+## Deployment
+
+`deploy/` holds an idempotent Ansible role; re-run it for any change. The
+target needs systemd and ssh; the role installs the rest:
+
+- `service.yml` — `garage` binary, secrets, config, systemd unit
+- `bucket.yml` — anonymous read, CORS
+- `proxy.yml` — hostname alias, nginx vhost
+
+Step one is to set up a separate VM in proxmox that will host our data. It needs
+Debian or Ubuntu (apt), an ssh user with sudo, and the disk mounted at
+`/var/lib/garage` (you can change the path in the inventory if preferred).
+
+Add a vhost to the `nginx` proxy:
+
+```nginx
+server {
+    server_name oats-store.opendronemap.org;   # eg.
+    location / {
+        proxy_pass http://<vm>:3902;
+        proxy_set_header Host $host;   # Garage picks the bucket by Host
+        proxy_buffering off;           # ODM outputs are large
+        ...
+    }
+}
+```
+
+The S3 API listens on port 3900 and the web endpoint on 3902, on every
+interface; RPC and the admin API stay on loopback.
+
+Set `oats_public_hostname` in the inventory to the `server_name` in nginx: the
+role aliases the bucket to it, which is how Garage maps the domain to the
+bucket. Set `oats_nginx_manage` only when nginx runs on the store VM itself.
+
+```sh
+cd deploy
+cp inventory.example.yml inventory.yml     # update host, user, hostname
+ansible-playbook storage.yml
+```
+
+The role generates the writer key on the first run and prints it at the end, for
+the CI secret store. Afterwards, `sudo cat /etc/oats/writer.env` on the host.
+
+Garage runs as the `garage` user under systemd, configured by
+`/etc/garage.toml`. `garage status` on the host shows the node.
+
+To update versions, set `oats_garage_version` and `oats_garage_sha256` to the
+new release and re-run the playbook.
+
+## Testing
+
+The role has a [Molecule](https://ansible.readthedocs.io/projects/molecule/)
+scenario that applies it to a systemd container, applies it again to check
+nothing changes, then checks the bucket is served anonymously with CORS:
+
+```sh
+uv tool install molecule --with 'molecule-plugins[docker]' --with ansible-core
+uv tool install ansible-core
+ansible-galaxy collection install community.docker
+cd deploy/roles/oats_storage
+molecule test
+```
+
+`molecule converge` leaves the container running for inspection;
+`molecule destroy` removes it.
+
+## Publishing
+
+```sh
+./publish_run.sh <run_dir> s3://oats-runs --endpoint http://<vm>:3900
+```
+
+Idempotent per run key. Needs `rclone`, `jq`, and `AWS_ACCESS_KEY_ID` /
+`AWS_SECRET_ACCESS_KEY`. See `--help`.
+
+`index.json` is derived from the run manifests, so `--reindex` rebuilds it from
+whatever is already in the store.
+
+## Credentials
+
+- `rpc_secret`, `admin_token`: generated on first deploy into `/etc/oats`,
+  readable only by the `garage` user. The admin API listens on loopback
+  port 3903.
+- Writer key: `/etc/oats/writer.env`, root-only, and the CI secret store.
+- The viewer needs none.
+
+The writer key uses the standard S3 variable names, so any S3 tool reads it
+without translation. Name the CI secrets for OATS and map them to those names in
+the workflow.
+
+`deploy/inventory.yml` is git-ignored.
+
+## Durability
+
+Single node, single disk. Runs are re-derivable by re-running CI; sync offsite
+with `rclone sync`. Back up the guest with `vzdump`.

storage/deploy/ansible.cfg

@@ -0,0 +1,4 @@
+[defaults]
+inventory = inventory.yml
+roles_path = roles
+stdout_callback = yaml

storage/deploy/inventory.example.yml

@@ -0,0 +1,16 @@
+---
+# Copy to inventory.yml (git-ignored)
+all:
+  hosts:
+    oats-store:
+      ansible_host: oats-vm.lan  # VM hostname or IP
+      ansible_user: ubuntu       # SSH username (needs sudo)
+
+      # Set the nginx vhost so the store is accessible via HTTP.
+      oats_public_hostname: oats-store.opendronemap.org
+
+      # Change if you want to store the data somewhere else.
+      # oats_data_root: /var/lib/garage
+
+      # Install nginx alongside garage if there is no external proxy.
+      # oats_nginx_manage: true

storage/deploy/roles/oats_storage/defaults/main.yml

@@ -0,0 +1,46 @@
+---
+# Where publish_run.sh is deployed.
+oats_storage_dir: /opt/oats-storage
+
+# Garage's metadata and data. Mount a separate disk here first if wanted.
+oats_data_root: /var/lib/garage
+
+oats_bucket: oats-runs
+
+# The bucket's public face: served by the web endpoint, readable from any origin.
+oats_bucket_public:
+  websiteAccess:
+    enabled: true
+    indexDocument: index.html
+  corsRules:
+    - AllowedOrigin: ["*"]
+      AllowedMethod: [GET, HEAD]
+      AllowedHeader: ["*"]
+      ExposeHeader: [ETag, Content-Length, Content-Type]
+      MaxAgeSeconds: 3600
+
+# The S3 API (3900) and web endpoint (3902) listen here; RPC and admin stay on
+# loopback.
+oats_bind_addr: 0.0.0.0
+oats_web_endpoint: http://127.0.0.1:3902
+oats_admin_endpoint: http://127.0.0.1:3903
+
+# garage is installed here, which the role puts on PATH.
+oats_bin_dir: /usr/local/bin
+oats_garage_version: v2.3.0
+oats_garage_url: "https://garagehq.deuxfleurs.fr/_releases/{{ oats_garage_version }}/x86_64-unknown-linux-musl/garage"
+# sha256sum of the download; Garage publishes none, so compute it on a bump.
+oats_garage_sha256: f98d317942bb341151a2775162016bb50cf86b865d0108de03eb5db16e2120cd
+
+# Garage runs under systemd as this user.
+oats_service_user: garage
+
+# Secrets, generated on the first run and printed once.
+oats_conf_dir: /etc/oats
+oats_writer_env: "{{ oats_conf_dir }}/writer.env"
+
+# When set, the bucket is aliased to it. The vhost needs oats_nginx_manage.
+oats_public_hostname: ""
+oats_nginx_manage: false
+oats_nginx_site_dir: /etc/nginx/sites-available
+oats_nginx_enabled_dir: /etc/nginx/sites-enabled

storage/deploy/roles/oats_storage/handlers/main.yml

@@ -0,0 +1,11 @@
+---
+- name: Restart Garage
+  ansible.builtin.systemd:
+    name: garage
+    daemon_reload: true
+    state: restarted
+
+- name: Reload nginx
+  ansible.builtin.systemd:
+    name: nginx
+    state: reloaded

storage/deploy/roles/oats_storage/molecule/default/converge.yml

@@ -0,0 +1,6 @@
+---
+- name: Converge
+  hosts: all
+  become: true
+  roles:
+    - oats_storage

storage/deploy/roles/oats_storage/molecule/default/molecule.yml

@@ -0,0 +1,17 @@
+---
+# Run the role in a systemd container: molecule test
+driver:
+  name: docker
+platforms:
+  - name: oats-store
+    image: geerlingguy/docker-ubuntu2404-ansible:latest
+    command: ""
+    pre_build_image: true
+    privileged: true
+    cgroupns_mode: host
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw
+provisioner:
+  name: ansible
+  env:
+    ANSIBLE_ROLES_PATH: ${MOLECULE_PROJECT_DIRECTORY}/..

storage/deploy/roles/oats_storage/molecule/default/verify.yml

@@ -0,0 +1,47 @@
+---
+- name: Verify
+  hosts: all
+  become: true
+  tasks:
+    - name: Read the bucket through the admin API
+      ansible.builtin.shell: >-
+        curl -sS --fail -H "Authorization: Bearer $(cat /etc/oats/admin_token)"
+        "http://127.0.0.1:3903/v2/GetBucketInfo?globalAlias=oats-runs"
+      register: bucket
+      changed_when: false
+
+    - name: Check the bucket is served anonymously with CORS
+      ansible.builtin.assert:
+        that:
+          - (bucket.stdout | from_json).websiteAccess
+          - (bucket.stdout | from_json).corsRules | length == 1
+          - (bucket.stdout | from_json)['keys'] | length == 1
+
+    - name: Fetch a missing object from the web endpoint
+      ansible.builtin.uri:
+        url: http://127.0.0.1:3902/nothing
+        headers:
+          Host: oats-runs
+          Origin: http://example.com
+        status_code: 404
+      register: web
+
+    - name: Check the web endpoint answers with CORS headers
+      ansible.builtin.assert:
+        that: web.access_control_allow_origin == '*'
+
+    - name: Reach the S3 API and web endpoint on the host's address
+      ansible.builtin.uri:
+        url: "http://{{ ansible_default_ipv4.address }}:{{ item.port }}/"
+        headers:
+          Host: oats-runs
+        status_code: "{{ item.status }}"
+      loop:
+        - { port: 3900, status: 403 }
+        - { port: 3902, status: 404 }
+
+    - name: Check RPC and the admin API stay on loopback
+      ansible.builtin.uri:
+        url: "http://{{ ansible_default_ipv4.address }}:{{ item }}/"
+        status_code: -1
+      loop: [3901, 3903]