diff --git a/docs/dr/openbao.md b/docs/dr/openbao.md index 6a89c9f80..d4cded066 100644 --- a/docs/dr/openbao.md +++ b/docs/dr/openbao.md @@ -2,10 +2,27 @@ ## Overview -OpenBao stores secrets in file-based storage. The Velero daily backup captures -the openbao namespace PVCs and the `openbao-unseal` Secret (which contains the -unseal key and root token). The `vault-config` Job auto-initializes OpenBao on -fresh clusters and auto-unseals on restarts. +OpenBao runs as a raft (Integrated Storage) cluster. Three artifacts make it +recoverable: + +1. **Raft snapshots** — the `vault-snapshot` CronJob saves a + `bao operator raft snapshot` to the `vault-snapshots` PVC daily (newest 14 + kept) and mirrors them to the S3 backup target under `openbao-snapshots/` + (Cloudflare R2 in prod, MinIO locally). The mirror exists because Velero's + file-system backup only captures volumes mounted by *running* pods — + nothing mounts this PVC outside the CronJob's brief run, so Velero alone + would never carry the snapshots off-cluster. +2. **`openbao-unseal` Secret** — unseal key + root token, captured by Velero's + daily resource backup. A snapshot is only usable together with the keys + that were current when it was taken. +3. **The `vault-config` Job** — auto-initializes OpenBao on fresh clusters, + auto-unseals on restarts, and **auto-restores**: when no pod reports an + initialized barrier but `openbao-unseal` still holds keys AND a snapshot + exists on the PVC, it temp-initializes, runs + `bao operator raft snapshot restore -force` with the newest snapshot, and + unseals with the stored key — no operator action. Only when no snapshot is + available does it abort and demand explicit data-loss acknowledgement + (the #1982 guard, unchanged). ## Recovery Scenarios @@ -21,50 +38,62 @@ No manual action needed. Verify: kubectl exec -n openbao openbao-0 -- bao status ``` -### Scenario 2: PVC data corruption +### Scenario 2: Raft data corruption or loss (Secret + snapshots intact) -**Symptom**: OpenBao fails to start, storage errors in logs. +**Symptom**: OpenBao fails to start with storage errors, or all pods report +`initialized: false` while `openbao-unseal` still exists (the 2026-06-10 +incident shape). -**Resolution**: +**Resolution** — automated. Reset the data volumes and let the `vault-config` +Job restore the newest snapshot: 1. Scale down the StatefulSet: ```bash kubectl scale statefulset -n openbao openbao --replicas=0 ``` -2. Delete the corrupted PVC: +2. Delete the corrupted data PVCs (NOT `vault-snapshots`, and do NOT delete + the `openbao-unseal` Secret — both are the restore inputs): ```bash - kubectl delete pvc -n openbao data-openbao-0 - ``` -3. Scale back up: - ```bash - kubectl scale statefulset -n openbao openbao --replicas=1 - ``` -4. Delete the stale `openbao-unseal` Secret (old keys are for the old storage): - ```bash - kubectl delete secret -n openbao openbao-unseal + kubectl delete pvc -n openbao data-openbao-0 data-openbao-1 data-openbao-2 ``` -5. Trigger Flux reconciliation (`ksail workload reconcile`) — the `vault-config` - Job re-runs, auto-initializes with fresh keys, and configures policies/roles. -6. PushSecrets re-seed the vault from SOPS variables on next reconciliation. - -### Scenario 3: Full cluster rebuild (Velero restore) - -**Symptom**: Entire cluster is lost (DR scenario), Velero backup available. - -**Resolution**: - -1. `ksail cluster create` — provisions infrastructure -2. Deploy Velero and restore from backup — this restores: - - OpenBao PVC (vault data) - - `openbao-unseal` Secret (unseal key + root token) -3. Flux deploys `infrastructure-controllers` → OpenBao starts -4. The `postStart` hook reads the restored `openbao-unseal` Secret → auto-unseals -5. `vault-config` Job runs → detects vault is already initialized → skips init → - converges policies/roles -6. ExternalSecrets resume syncing from the restored vault data - -**Key point**: Velero backs up both the PVC (vault data) and the `openbao-unseal` -Secret (unseal credentials). Both are needed for a complete restore. +3. Trigger Flux reconciliation (`ksail workload reconcile`) — the StatefulSet + scales back up with empty volumes, and the `vault-config` Job detects + uninitialized-pods + surviving-keys + available-snapshot and restores + automatically (worst-case RPO: 24 h, the snapshot cadence). +4. ExternalSecrets resume syncing; PushSecrets top up anything newer than the + snapshot on their next refresh. + +Only if **no snapshot exists** (PVC also lost and the R2 mirror is empty) does +the Job abort with the data-loss guard; acknowledge the loss explicitly with +`kubectl delete secret openbao-unseal -n openbao` and the next run +re-initializes from scratch (Scenario 4 then re-seeds the KV). + +### Scenario 3: Full cluster rebuild (backups available) + +**Symptom**: Entire cluster is lost (DR scenario); the R2 snapshot mirror +and/or Velero backups are available. + +**Sequencing caveat**: on a rebuilt cluster Flux stands OpenBao up *before* +any restore can run, so the `vault-config` Job auto-initializes a **fresh** +vault first (no `openbao-unseal` exists yet → the guard does not trigger). +Recovering the old data means deliberately resetting that fresh vault into +the Scenario 2 shape: + +1. `ksail cluster create` + `workload push`/`reconcile` — the platform + converges with a fresh, empty vault (PushSecrets re-seed SOPS-sourced + values, so the cluster is functional but generated secrets are new). +2. Restore the old `openbao-unseal` Secret (from the Velero backup) over the + fresh one, and copy the newest snapshot from the R2 `openbao-snapshots/` + mirror onto the `vault-snapshots` PVC. +3. Scale OpenBao to 0, delete the fresh `data-openbao-*` PVCs, reconcile — + the `vault-config` Job now hits the automated restore path (Scenario 2) + and brings back the pre-incident vault. +4. ExternalSecrets resume syncing from the restored data; consumers pick up + the old (matching) credentials. + +**Key point**: a snapshot and the `openbao-unseal` Secret must come from the +same generation (same backup day) — keys from one era cannot unseal a +snapshot from another. ### Scenario 4: Full cluster rebuild (no Velero backup) diff --git a/k8s/bases/infrastructure/vault-backup/cronjob.yaml b/k8s/bases/infrastructure/vault-backup/cronjob.yaml index 73577fa63..048b21817 100644 --- a/k8s/bases/infrastructure/vault-backup/cronjob.yaml +++ b/k8s/bases/infrastructure/vault-backup/cronjob.yaml @@ -5,8 +5,21 @@ # openbao-unseal Secret: restore via # bao operator raft snapshot restore # then unseal with the key that was current when the snapshot was taken -# (docs/dr/openbao-raft-ha-migration.md). +# (docs/dr/openbao-raft-ha-migration.md). The vault-config Job performs +# this restore automatically when it finds an uninitialized cluster, a +# surviving openbao-unseal Secret AND a snapshot on the PVC (docs/dr/ +# openbao.md). # Authenticates via Kubernetes auth (vault-snapshot role). +# +# Pod layout: the snapshot runs as an initContainer and the off-cluster +# mirror as the main container, so the upload only ever runs after a NEW +# snapshot succeeded (containers in one pod run in parallel; init -> main +# gives strict ordering). The mirror copies the PVC's snapshots to the +# S3 backup target (Cloudflare R2 in prod, in-cluster MinIO locally) — +# Velero's file-system backup CANNOT carry this PVC off-cluster, because +# FSB only backs up volumes mounted by RUNNING pods and nothing mounts +# this PVC outside the CronJob's own brief run. Without the mirror, a +# full-cluster loss would also lose every raft snapshot. apiVersion: batch/v1 kind: CronJob metadata: @@ -52,7 +65,16 @@ spec: - name: snapshots persistentVolumeClaim: claimName: vault-snapshots - containers: + # R2 credentials synced by ESO (external-secret.yaml). Optional: + # on a fresh cluster the mirror container skips with a log line + # until the sync lands; the local snapshot is unaffected. + - name: r2-credentials + secret: + secretName: vault-snapshot-r2 + optional: true + - name: mc-config + emptyDir: {} + initContainers: - name: snapshot image: quay.io/openbao/openbao:2.5.3@sha256:fdc6da21ca6963560c32336fd7feb9cf2d5e52668f1a1647205a4b41171f0806 securityContext: @@ -107,3 +129,52 @@ spec: rm -f "$OLD" done echo "Snapshot complete." + containers: + - name: mirror + image: quay.io/minio/mc:RELEASE.2025-04-08T15-39-49Z@sha256:7e3efb09c22c0882fbf341b9d99f61f94ae6c4c20a06f2f1a2b20ea8993d8952 + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + readOnlyRootFilesystem: true + env: + - name: MC_CONFIG_DIR + value: /tmp/.mc + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 128Mi + volumeMounts: + - name: snapshots + mountPath: /snapshots + readOnly: true + - name: r2-credentials + mountPath: /r2 + readOnly: true + - name: mc-config + mountPath: /tmp/.mc + command: + - /bin/sh + - -ec + - | + if [ ! -f /r2/access_key_id ]; then + echo "vault-snapshot-r2 Secret not synced yet — skipping the" + echo "off-cluster mirror. The local PVC snapshot above still ran." + exit 0 + fi + mc alias set backup "${r2_endpoint}" \ + "$(cat /r2/access_key_id)" "$(cat /r2/secret_access_key)" + # Copy-only mirror: NO --remove, so an empty/recreated PVC can + # never wipe the off-cluster history. Remote retention is + # pruned by age instead, and only after a successful upload in + # the same run (this container only starts when the snapshot + # initContainer succeeded), so the newest snapshot always + # survives the prune. + mc mirror --overwrite /snapshots/ "backup/${r2_bucket}/openbao-snapshots/" + mc rm --recursive --force --older-than 14d \ + "backup/${r2_bucket}/openbao-snapshots/" || true + echo "Mirrored snapshots:" + mc ls "backup/${r2_bucket}/openbao-snapshots/" diff --git a/k8s/bases/infrastructure/vault-backup/external-secret.yaml b/k8s/bases/infrastructure/vault-backup/external-secret.yaml new file mode 100644 index 000000000..75bfabd0a --- /dev/null +++ b/k8s/bases/infrastructure/vault-backup/external-secret.yaml @@ -0,0 +1,29 @@ +--- +# R2 credentials for the vault-snapshot CronJob's off-cluster mirror +# container — the same infrastructure/backup/r2 KV entry Velero and CNPG +# use, synced into a plain two-key Secret for `mc`. Deliberately optional +# on the consumer side (the CronJob mounts it `optional: true`): on a +# fresh cluster the mirror is skipped with a log line until ESO has +# synced, while the local PVC snapshot still runs. +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: vault-snapshot-r2 + namespace: openbao +spec: + refreshInterval: 1h + secretStoreRef: + name: openbao + kind: ClusterSecretStore + target: + name: vault-snapshot-r2 + creationPolicy: Owner + data: + - secretKey: access_key_id + remoteRef: + key: infrastructure/backup/r2 + property: access_key_id + - secretKey: secret_access_key + remoteRef: + key: infrastructure/backup/r2 + property: secret_access_key diff --git a/k8s/bases/infrastructure/vault-backup/init-job.yaml b/k8s/bases/infrastructure/vault-backup/init-job.yaml index 20cdacc8d..963f62314 100644 --- a/k8s/bases/infrastructure/vault-backup/init-job.yaml +++ b/k8s/bases/infrastructure/vault-backup/init-job.yaml @@ -14,6 +14,12 @@ # Same retry pattern as the vault-config Job: the script fails until # vault-config has created the vault-snapshot auth role and unsealed the # vault, and OnFailure + backoffLimit keep retrying until then. +# +# Pod layout mirrors cronjob.yaml: snapshot as initContainer, off-cluster +# mirror as the main container — the baseline snapshot reaches the S3 +# backup target immediately instead of waiting for the nightly mirror, +# which matters because the deploy-time baseline is often the ONLY +# snapshot during a change's first 24 hours. apiVersion: batch/v1 kind: Job metadata: @@ -47,7 +53,16 @@ spec: - name: snapshots persistentVolumeClaim: claimName: vault-snapshots - containers: + # R2 credentials synced by ESO (external-secret.yaml). Optional: on a + # fresh cluster the mirror container skips with a log line until the + # sync lands; the local snapshot is unaffected. + - name: r2-credentials + secret: + secretName: vault-snapshot-r2 + optional: true + - name: mc-config + emptyDir: {} + initContainers: - name: snapshot image: quay.io/openbao/openbao:2.5.3@sha256:fdc6da21ca6963560c32336fd7feb9cf2d5e52668f1a1647205a4b41171f0806 securityContext: @@ -100,3 +115,50 @@ spec: bao operator raft snapshot save "$SNAP" ls -l "$SNAP" echo "Initial snapshot complete." + containers: + - name: mirror + image: quay.io/minio/mc:RELEASE.2025-04-08T15-39-49Z@sha256:7e3efb09c22c0882fbf341b9d99f61f94ae6c4c20a06f2f1a2b20ea8993d8952 + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + readOnlyRootFilesystem: true + env: + - name: MC_CONFIG_DIR + value: /tmp/.mc + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 128Mi + volumeMounts: + - name: snapshots + mountPath: /snapshots + readOnly: true + - name: r2-credentials + mountPath: /r2 + readOnly: true + - name: mc-config + mountPath: /tmp/.mc + command: + - /bin/sh + - -ec + - | + if [ ! -f /r2/access_key_id ]; then + echo "vault-snapshot-r2 Secret not synced yet — skipping the" + echo "off-cluster mirror. The local PVC snapshot above still ran." + exit 0 + fi + mc alias set backup "${r2_endpoint}" \ + "$(cat /r2/access_key_id)" "$(cat /r2/secret_access_key)" + # Copy-only mirror (no --remove) + age-based prune, same + # rationale as cronjob.yaml: an empty/recreated PVC can never + # wipe the off-cluster history, and the prune only runs after a + # successful upload in the same pod. + mc mirror --overwrite /snapshots/ "backup/${r2_bucket}/openbao-snapshots/" + mc rm --recursive --force --older-than 14d \ + "backup/${r2_bucket}/openbao-snapshots/" || true + echo "Mirrored snapshots:" + mc ls "backup/${r2_bucket}/openbao-snapshots/" diff --git a/k8s/bases/infrastructure/vault-backup/kustomization.yaml b/k8s/bases/infrastructure/vault-backup/kustomization.yaml index 5a1892935..0b4ddea2b 100644 --- a/k8s/bases/infrastructure/vault-backup/kustomization.yaml +++ b/k8s/bases/infrastructure/vault-backup/kustomization.yaml @@ -4,6 +4,7 @@ kind: Kustomization resources: - serviceaccount.yaml - pvc.yaml + - external-secret.yaml - init-job.yaml - cronjob.yaml - networkpolicy.yaml diff --git a/k8s/bases/infrastructure/vault-backup/networkpolicy.yaml b/k8s/bases/infrastructure/vault-backup/networkpolicy.yaml index 507bbbb84..5d78df860 100644 --- a/k8s/bases/infrastructure/vault-backup/networkpolicy.yaml +++ b/k8s/bases/infrastructure/vault-backup/networkpolicy.yaml @@ -20,6 +20,21 @@ spec: # Kube API for ServiceAccount token authentication - toEntities: - kube-apiserver + # S3-compatible snapshot mirror target (Cloudflare R2 in prod) + - toEntities: + - world + toPorts: + - ports: + - port: "443" + protocol: TCP + # In-cluster MinIO mirror target for local/CI + - toEndpoints: + - matchLabels: + k8s:io.kubernetes.pod.namespace: minio + toPorts: + - ports: + - port: "9000" + protocol: TCP # DNS resolution - toEndpoints: - matchLabels: diff --git a/k8s/bases/infrastructure/vault-backup/pvc.yaml b/k8s/bases/infrastructure/vault-backup/pvc.yaml index 2a1bc09f5..10802c8b8 100644 --- a/k8s/bases/infrastructure/vault-backup/pvc.yaml +++ b/k8s/bases/infrastructure/vault-backup/pvc.yaml @@ -1,8 +1,12 @@ # Dedicated volume for OpenBao raft snapshots (written by the # vault-snapshot CronJob, newest 14 retained). Uses the cluster's default # StorageClass. Snapshots on this PVC are the first-line restore source -# after OpenBao data loss; Velero's daily namespace backup carries them -# off-cluster alongside the openbao-unseal Secret they pair with. +# after OpenBao data loss: the vault-config Job restores from the newest +# one automatically when it finds an uninitialized cluster alongside a +# surviving openbao-unseal Secret. Off-cluster durability comes from the +# CronJob's mirror container (S3/R2), NOT from Velero — Velero's +# file-system backup only captures volumes mounted by running pods, and +# nothing mounts this PVC outside the CronJob's brief daily run. apiVersion: v1 kind: PersistentVolumeClaim metadata: diff --git a/k8s/bases/infrastructure/vault-config/job.yaml b/k8s/bases/infrastructure/vault-config/job.yaml index 6e4f4e079..2cb5013b2 100644 --- a/k8s/bases/infrastructure/vault-config/job.yaml +++ b/k8s/bases/infrastructure/vault-config/job.yaml @@ -7,7 +7,11 @@ # init/unseal: consecutive requests land on different pods, # which is exactly the "double-init split-brain" and # "unseal-before-join" risk pair called out in -# docs/dr/openbao-raft-ha-migration.md) +# docs/dr/openbao-raft-ha-migration.md). When the raft data +# is gone but the openbao-unseal Secret survived AND a raft +# snapshot exists on the vault-snapshots PVC, it restores +# the newest snapshot automatically instead of aborting +# (docs/dr/openbao.md). # 2. store-keys — persists unseal key + root token in a K8s Secret # # Main container (vault-config): @@ -82,6 +86,15 @@ spec: secret: secretName: vault-config-oidc optional: true + # Raft snapshots written by the vault-snapshot CronJob (vault-backup/), + # mounted read-only as the vault-init container's automated restore + # source. RWO is fine: the CronJob only holds the volume for seconds + # around 03:30 UTC, so attach conflicts with this Job are practically + # impossible. + - name: snapshots + persistentVolumeClaim: + claimName: vault-snapshots + readOnly: true initContainers: # --- 1. Auto-init + unseal --- - name: vault-init @@ -104,6 +117,9 @@ spec: readOnly: true - name: shared mountPath: /shared + - name: snapshots + mountPath: /snapshots + readOnly: true resources: requests: cpu: 10m @@ -142,43 +158,101 @@ spec: done if [ "$ANY_INITIALIZED" = "false" ]; then - # Data-loss guard: keys in openbao-unseal mean a cluster was - # initialized before. If on top of that NO pod reports an + # Data-loss guard (#1982): keys in openbao-unseal mean a cluster + # was initialized before. If on top of that NO pod reports an # initialized barrier, the raft data is missing or unreadable — # auto-initializing here would silently bring up an EMPTY vault # and overwrite the previous unseal key + root token (exactly # how the 2026-06-10 incident destroyed the whole KV store). - # Fail loudly instead and make the operator acknowledge the - # data loss explicitly before a fresh init is allowed. + # With a raft snapshot on the vault-snapshots PVC the data is + # recoverable WITHOUT operator action: temp-init, restore the + # newest snapshot, then unseal with the stored (pre-incident) + # key. Only when no snapshot exists does the guard still abort. if [ -f /openbao/unseal/unseal-key ]; then - echo "ERROR: no pod reports an initialized vault, but the openbao-unseal" - echo "Secret still holds keys from a previously initialized cluster." - echo "Refusing to auto-initialize: that would create an empty vault and" - echo "overwrite the old keys, making any surviving data unrecoverable." - echo "" - echo "Recover the data first (Velero restore / reattach the old data" - echo "PVCs) and re-run this Job. If the data loss is confirmed and a" - echo "fresh vault is intended, acknowledge it explicitly with:" - echo " kubectl delete secret openbao-unseal -n openbao" - echo "and let the next Job run initialize from scratch." - exit 1 - fi - echo "No initialized pod found — running bao operator init on openbao-0..." - INIT_OUTPUT=$(BAO_ADDR="$(addr 0)" bao operator init -key-shares=1 -key-threshold=1) + NEWEST_SNAP=$(ls -1t /snapshots/openbao-*.snap 2>/dev/null | head -n 1 || true) + if [ -z "$NEWEST_SNAP" ]; then + echo "ERROR: no pod reports an initialized vault, but the openbao-unseal" + echo "Secret still holds keys from a previously initialized cluster, and" + echo "no raft snapshot is available on the vault-snapshots PVC." + echo "Refusing to auto-initialize: that would create an empty vault and" + echo "overwrite the old keys, making any surviving data unrecoverable." + echo "" + echo "Recover the data first (Velero restore / reattach the old data" + echo "PVCs / copy a snapshot from the R2 openbao-snapshots/ mirror onto" + echo "the vault-snapshots PVC) and re-run this Job. If the data loss is" + echo "confirmed and a fresh vault is intended, acknowledge it explicitly:" + echo " kubectl delete secret openbao-unseal -n openbao" + echo "and let the next Job run initialize from scratch." + exit 1 + fi - UNSEAL_KEY=$(echo "$INIT_OUTPUT" | grep "Unseal Key 1:" | sed 's/.*: //') - ROOT_TOKEN=$(echo "$INIT_OUTPUT" | grep "Initial Root Token:" | sed 's/.*: //') + echo "No initialized pod found, but openbao-unseal holds keys from a" + echo "previous cluster AND a raft snapshot exists:" + echo " $NEWEST_SNAP" + echo "Running the automated snapshot restore (docs/dr/openbao.md)." + + # 1. Temp-init openbao-0. These credentials live only in this + # container's memory and stop mattering the moment the + # snapshot is restored (the snapshot's barrier replaces + # them). If the restore below fails, the raft holds an + # empty temp vault: delete the openbao data PVCs and let + # the pods + this Job re-run — the Secret and the snapshot + # are untouched, so the restore retries from scratch. + INIT_OUTPUT=$(BAO_ADDR="$(addr 0)" bao operator init -key-shares=1 -key-threshold=1) + TMP_UNSEAL=$(echo "$INIT_OUTPUT" | grep "Unseal Key 1:" | sed 's/.*: //') + TMP_ROOT=$(echo "$INIT_OUTPUT" | grep "Initial Root Token:" | sed 's/.*: //') + if [ -z "$TMP_UNSEAL" ] || [ -z "$TMP_ROOT" ]; then + echo "ERROR: failed to parse temp init output (unseal key or root token missing)." + echo "Refusing to print raw output because it may contain secret material." + exit 1 + fi - if [ -z "$UNSEAL_KEY" ] || [ -z "$ROOT_TOKEN" ]; then - echo "ERROR: failed to parse init output (unseal key or root token missing)." - echo "Refusing to print raw output because it may contain secret material." - exit 1 - fi + # 2. Unseal openbao-0 with the temp key and wait for a leader. + BAO_ADDR="$(addr 0)" bao operator unseal "$TMP_UNSEAL" >/dev/null + TRIES=24 + until pod_status 0 | grep -q '"ha_mode": "active"'; do + TRIES=$((TRIES - 1)) + if [ "$TRIES" -le 0 ]; then + echo "ERROR: openbao-0 never became the active node after temp init." + exit 1 + fi + sleep 5 + done + + # 3. Restore the snapshot. -force skips the cluster-ID match + # (the temp cluster's ID can never match the snapshot's). + echo "Restoring $NEWEST_SNAP..." + BAO_ADDR="$(addr 0)" BAO_TOKEN="$TMP_ROOT" \ + bao operator raft snapshot restore -force "$NEWEST_SNAP" + + # 4. The cluster's barrier is now the snapshot's, so the + # stored pre-incident keys apply again — fall through to + # the shared unseal loop with them. newly-initialized is + # deliberately NOT written: store-keys must not overwrite + # the openbao-unseal Secret that pairs with this snapshot. + cat /openbao/unseal/unseal-key > /shared/unseal-key + if [ -f /openbao/unseal/root-token ]; then + cat /openbao/unseal/root-token > /shared/root-token + fi + echo "Snapshot restored; unsealing with the stored key." + else + echo "No initialized pod found — running bao operator init on openbao-0..." + INIT_OUTPUT=$(BAO_ADDR="$(addr 0)" bao operator init -key-shares=1 -key-threshold=1) + + UNSEAL_KEY=$(echo "$INIT_OUTPUT" | grep "Unseal Key 1:" | sed 's/.*: //') + ROOT_TOKEN=$(echo "$INIT_OUTPUT" | grep "Initial Root Token:" | sed 's/.*: //') - printf '%s' "$UNSEAL_KEY" > /shared/unseal-key - printf '%s' "$ROOT_TOKEN" > /shared/root-token - touch /shared/newly-initialized - echo "Vault initialized on openbao-0." + if [ -z "$UNSEAL_KEY" ] || [ -z "$ROOT_TOKEN" ]; then + echo "ERROR: failed to parse init output (unseal key or root token missing)." + echo "Refusing to print raw output because it may contain secret material." + exit 1 + fi + + printf '%s' "$UNSEAL_KEY" > /shared/unseal-key + printf '%s' "$ROOT_TOKEN" > /shared/root-token + touch /shared/newly-initialized + echo "Vault initialized on openbao-0." + fi else echo "Vault is already initialized." @@ -240,7 +314,10 @@ spec: done if [ "$UNSEALED_ANY" = "false" ]; then echo "ERROR: no pod could be unsealed — the stored key may not match this cluster." - echo "See docs/dr/openbao-raft-ha-migration.md for recovery." + echo "After an automated snapshot restore this means the openbao-unseal Secret" + echo "and the snapshot are from different generations — restore the matching" + echo "pair (same backup day) and re-run. See docs/dr/openbao.md and" + echo "docs/dr/openbao-raft-ha-migration.md for recovery." exit 1 fi echo "Unseal phase complete." diff --git a/k8s/providers/docker/infrastructure/controllers/minio/networkpolicy.yaml b/k8s/providers/docker/infrastructure/controllers/minio/networkpolicy.yaml index 4ea3967ee..3a2f729e1 100644 --- a/k8s/providers/docker/infrastructure/controllers/minio/networkpolicy.yaml +++ b/k8s/providers/docker/infrastructure/controllers/minio/networkpolicy.yaml @@ -6,12 +6,21 @@ metadata: spec: endpointSelector: {} ingress: - # Velero and CNPG backup/restore access + # Velero and CNPG backup/restore access, plus the OpenBao raft-snapshot + # mirror (vault-backup CronJob + init Job). Cilium drops a flow unless + # BOTH the client's egress AND the server's ingress allow it, so the + # egress rule in bases/infrastructure/vault-backup/networkpolicy.yaml is + # not sufficient on its own — its missing counterpart here crashlooped + # vault-snapshot-init's mirror container in CI (mc i/o timeout) and + # wedged the infrastructure health gate. - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: velero - matchLabels: k8s:io.kubernetes.pod.namespace: cnpg-system + - matchLabels: + k8s:io.kubernetes.pod.namespace: openbao + app: vault-snapshot toPorts: - ports: - port: "9000"