ci: implement the documented Velero restore drill in the system-test job

[devantler]

🤖 Generated by the Daily AI Assistant

Summary

docs/dr/restore-drill.md has documented a CI restore drill since it was written — but no workflow ever implemented it. The backup → data-loss → restore path was never regression-tested, so a Velero chart bump, RBAC drift, or MinIO credential break would only surface during a real disaster (as 2026-06-10 demonstrated for the adjacent vault-snapshot path, which turned out to be a health check in a trench coat).

This implements the drill as steps inside the existing system-test job, reusing the Talos+Docker cluster it just reconciled — a separate job (as the doc originally described) would pay a second ~10-minute cluster bootstrap for no extra signal. Added wall-clock: ~2-3 minutes.

The drill

1. Wait for BackupStorageLocation/default → Available (Velero → in-cluster MinIO, the local R2 stand-in — same S3 code path as prod)
2. Create dr-drill namespace + marker ConfigMap carrying run-id/sha
3. Backup CR scoped to the namespace; wait for Completed, failing fast on Failed/PartiallyFailed/FailedValidation
4. Delete the namespace, assert it's gone
5. Restore CR from the backup; wait for Completed
6. Assert the restored ConfigMap's run-id matches GITHUB_RUN_ID

Velero CRs are created with kubectl (no velero CLI install, can't drift from the deployed Velero version). On failure the step dumps BSL/Backup/Restore state + the Velero server log.

Doc truth-ups in the same change

• restore-drill.md described a standalone job with its own cluster and a timeout-minutes: 240 budget that never existed → now describes the implemented steps.
• runbook.md claimed the drill asserts etcd encryption-at-rest; restore-drill.md itself explicitly scopes that out → corrected.

Validation

• Workflow YAML parses (yq); the 90-line drill script passes bash -n; the heredoc Backup/Restore payloads verified as valid YAML at column 0.
• ⚠️ Expected CI status: system-test is currently red repo-wide from the pre-existing OpenBao label-patch 422 regression (being probed by ci: probe the OpenBao label-patch 422 with kubectl on system-test failure #1990); the drill steps run after reconcile, so this PR can't go green until that's resolved — the drill itself is unaffected.

🤖 Generated with Claude Code

[devantler]

🤖 Generated by the Daily AI Assistant

System-test status: the first run failed in ~2 min on a transient schema-fetch error during ksail workload validate (bases/apps/actual-budget ... validation failed: EOF — also hit other PRs and branches with no manifest changes). On rerun, validate passed and the job ran the full ~53 min, failing at reconcile with infrastructure/apps NotReady — the known repo-wide OpenBao label-patch 422 wedge being probed in #1990 (same signature and duration as #1987/#1989). The drill steps added by this PR run after reconcile, so they haven't executed yet; this PR can go green once the 422 regression is fixed.

[devantler]

🤖 Generated by the Daily AI Assistant

The drill's first real run (now that main's reconcile wedge is fixed) exposed a genuine bug — exactly the kind of regression this drill exists to catch, just in itself this time: CNPG also defines a backups resource, and kubectl resolves an unqualified backup to backups.postgresql.cnpg.io on this cluster. wait_phase therefore polled the wrong API group for its whole 300s timeout (backups.postgresql.cnpg.io \"dr-drill\" not found) while the actual Velero backup ran unobserved.

Fixed in 65815e7f: every get/describe/wait in the drill now uses the fully-qualified *.velero.io resource names, with a comment documenting the collision. The next system-test run will exercise the full drill with correct polling.

[botantler]

🎉 This PR is included in version 1.52.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀