Repository navigation
Garage ansible playbook and publish script - #17
Open
MJohnson459 wants to merge 11 commits into
Open
MJohnson459 wants to merge 11 commits into
MJohnson459 wants to merge 11 commits into
Conversation
Store OATS run directories durably and serve them read-only over HTTP so the comparison viewer can enumerate and load runs. storage/ adds: - A single-node Garage (S3-compatible) service: compose.yaml, garage.toml (replication_factor = 1), and init.sh for the one-time layout / bucket / website / writer-key setup. Secrets are injected from a git-ignored .env; .env.example documents them. - publish_run.sh: idempotent, dual-target. Publishes a run directory to a filesystem root (rsync) or an S3 bucket (mc against Garage), keyed by the run's identity path, and regenerates a top-level index.json listing every run with its identity and aggregate pass/fail counts. The viewer reads only index.json plus the run objects; it never lists the bucket. - cors.xml and the website/anonymous-read setup give cross-origin GET for index.json, run_manifest.json, metrics and preview images. - README.md records the backend decision, local setup, publish workflow, credential handling and the runner deployment steps. Verified end-to-end locally against Garage: both publish targets, idempotent re-publish, and cross-origin browser fetch of the store over the web endpoint.
The store's deployment was a 89-line README procedure: pass the disk through, mkfs and fstab it, generate secrets, write a compose override, install mc, apply CORS, alias the bucket, add an nginx vhost. Every step was a copy-paste command with no way to re-run it safely. deploy/ replaces that with an idempotent Ansible role covering the whole host setup, so rolling out a config change is one command. init.sh is reused rather than reimplemented, keeping one definition of the cluster setup for both local development and deployment. Writer credentials are generated on first run and kept root-only in /etc/oats/writer.env, with an opt-in fetch for pasting into the CI secret store. Only the two Proxmox host operations stay manual, being one-shot and outside the VM. The README drops to the store contract, the deployment command and the parts that need explaining rather than executing. Verified against a local Garage: fresh deploy, a re-run reporting no changes, a run published through the deployed store, and a cross-origin read of index.json over the aliased public hostname.
init.sh was the last of the provisioning left in shell, which meant the cluster setup had two homes and the playbook shelled out to a script to do half its work. Its steps are all guarded Garage CLI calls, so they port directly to tasks: wait for the node, assign and apply the layout, create the bucket, enable anonymous web access, issue the writer key and grant it. The credentials are parsed straight from "key create" rather than scraped from a script's stdout. Local development runs the same role through inventory.local.yml, which puts the store under ~/.local/share, so dropping the script costs it nothing and the setup has one definition. The task list is split by what it sets up — disk, service, cluster, credentials, access, proxy — leaving main.yml as a table of contents. Verified on a local Garage: a fresh deploy from an empty state, a re-run reporting no changes (with the public-hostname path exercised), a run published through the result, and a cross-origin read of index.json over the aliased hostname.
The docs still carried the shape of earlier iterations: the top-level README advertised a filesystem publish target that no longer exists, the store contract diagram kept a <tag> level the harness has since dropped from run directories, and .env.example described hand-filling a secrets file the deploy role now generates and nothing referenced. Rewrite the prose to state what the store does rather than how it got here, correct the run layout to tests/<dataset>/<test>/, and drop the orphaned example file.
Cut the prose down to what a reader needs: the store contract, the deployment command, and the facts that are not obvious from the code. The tree and index.json shapes stay, since the viewer codes against them, as does publish_run.sh's usage text, which is its --help output.
A failed enumeration could replace a populated index.json with an empty one: mc find's stderr went to /dev/null, jq turned no input into an empty run list, and mc pipe wrote it before pipefail reported the failure. The index is now assembled locally and checked before upload — non-empty, and containing the run just published — and the enumeration's errors are left visible. --reindex rebuilds the index from the manifests already in the store, which are the source of truth. The writer key is read back from Garage with "key info --show-secret" rather than captured at creation, so /etc/oats/writer.env is a cache that is rewritten when it goes missing instead of a file whose loss forced the key to be reissued. Verified against a local Garage: publish and --reindex both report the run count; a rebuild that enumerates nothing refuses and leaves a populated index untouched; deleting writer.env and re-running restores the identical credentials.
Garage v2.3.0 replaces the docker compose stack: a checksummed binary under systemd, --single-node --default-bucket for the layout, bucket and writer key, and the admin API for website access and CORS. The role no longer formats disks, installs mc or needs Galaxy collections. The S3 API and web endpoint bind on all interfaces so CI and an external proxy can reach them; RPC and admin stay on loopback. A Molecule scenario runs the role in a systemd container and checks idempotence and the exposed ports. The workflow gains a publish step that is skipped until OATS_STORE_ENDPOINT is set.
`mc` was easy, but only had a denylist. When we upload, it will be much easier to start with what we want and add, rather than risk huge files being uploaded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds an ansible playbook that will set up a VM with garage and all the paths and config an oats CI job will expect. The readme explains all the steps not automated. Two things:
Once garage is set up and the secrets added to the oats repo, the
publish_run.shscript should store the major outputs (~2GB) for every run. I haven't added any cleanup yet to the garage store.AI Disclaimer: I did use AI to learn about garage and related tooling, rubber ducking the desired process, and helping me with running tests.