Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ Things not in any single grep-able file:
- **Wildcard DNS**: `*.live.k8s.phl.io` → the Envoy LB `45.79.246.168`. DNS is managed in OpenTofu at [CodeForPhilly/ops](https://github.com/CodeForPhilly/ops) → `tofu/dns`; a host with no specific record simply follows the wildcard, so new apps need no DNS change at all.
- **Apex domains in tree**: `balancerproject.org`, `choosenativeplants.com` (+ `www.`), `codeforphilly.org` (+ `www.`), `penn-chime.phl.io`, `vaultwarden.phl.io`, `bitwarden.phl.io`. Apex ACME challenges only work once DNS points at Envoy — plan cutover and cert issuance together for these. `choosenativeplants.com` is at **Namecheap**, not Cloud DNS, so it can't be moved from the ops repo.
- **A new hostname is briefly down between DNS and cert.** The cert can't issue until the hostname resolves to Envoy (Let's Encrypt has to reach the solver), and Envoy's HTTPS listener doesn't program until the cert Secret exists — meanwhile HTTP 301s into a listener that isn't there. Roughly 60–90s. Keep TTLs at 60s.
- **cnpg is landing.** `_infra/cloudnative-pg/` installs the operator (chart v0.29.1, operator 1.30.1), the Barman Cloud plugin (chart plugin-barman-cloud-v0.8.0), and a `shared-cluster` Cluster, mirroring cfp-sandbox-cluster. Backups go through the plugin, never the in-tree `barmanObjectStore` (removed in cnpg 1.31). `_infra/cloudnative-pg/backups.yaml` archives WAL continuously and takes a nightly base backup to the `cfp-live-cluster-cnpg` bucket (Linode Object Storage, us-east, CodeForPhilly account; key sealed in `cloudnative-pg.secrets/`), 30-day retention. Existing apps still ship their own database per-app (vaultwarden runs a PostgreSQL StatefulSet via the gissilabs chart; chime + third-places similar); nothing has been migrated onto the shared cluster yet.
- **cnpg is landing.** `_infra/cloudnative-pg/` installs the operator (chart v0.29.1, operator 1.30.1), the Barman Cloud plugin (chart plugin-barman-cloud-v0.8.0), and a `shared-cluster` Cluster, mirroring cfp-sandbox-cluster. Backups go through the plugin, never the in-tree `barmanObjectStore` (removed in cnpg 1.31). `_infra/cloudnative-pg/backups.yaml` archives WAL continuously and takes a nightly base backup to `gs://cnpg-live` in GCP project `openphl-1177` (us-east4), deliberately off Linode so backups survive losing the Linode account; the service-account key is sealed in `cloudnative-pg.secrets/`, 30-day retention. Existing apps still ship their own database per-app (vaultwarden runs a PostgreSQL StatefulSet via the gissilabs chart; chime + third-places similar); nothing has been migrated onto the shared cluster yet.

## Guardrails

Expand Down
24 changes: 11 additions & 13 deletions _infra/cloudnative-pg/backups.yaml
Original file line number Diff line number Diff line change
@@ -1,26 +1,24 @@
# Continuous WAL archiving + nightly base backups for shared-cluster, via the
# Barman Cloud plugin (the in-tree barmanObjectStore is removed in cnpg 1.31).
#
# Bucket: cfp-live-cluster-cnpg (Linode Object Storage, us-east, CodeForPhilly
# account). Barman keys everything under the Cluster's name inside the bucket,
# so any future Cluster on this k8s cluster can share it. The access key is
# limited to this one bucket and sealed in cloudnative-pg.secrets/.
# Bucket: gs://cnpg-live in GCP project openphl-1177 (us-east4) — deliberately
# OFF Linode, so backups survive the loss of the Linode account or region.
# Barman keys everything under the Cluster's name inside the bucket, so any
# future Cluster on this k8s cluster can share it. The service account
# cnpg-backup-live@openphl-1177 has objectAdmin + legacyBucketReader on this
# one bucket only; its JSON key is sealed in cloudnative-pg.secrets/.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: linode-cnpg
name: gcs-cnpg
namespace: cloudnative-pg
spec:
configuration:
destinationPath: s3://cfp-live-cluster-cnpg/
endpointURL: https://us-east-1.linodeobjects.com
s3Credentials:
accessKeyId:
destinationPath: gs://cnpg-live/
googleCredentials:
applicationCredentials:
name: cnpg-backup-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: cnpg-backup-creds
key: ACCESS_SECRET_KEY
key: gcsCredentials
wal:
compression: gzip
data:
Expand Down
4 changes: 2 additions & 2 deletions _infra/cloudnative-pg/shared-cluster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ spec:
# for as long as the Secret it names is absent. The role and the balancer
# Database CR land together in the follow-up PR.

# Backups: WAL archiving + nightly base backups to Linode Object Storage via
# Backups: WAL archiving + nightly base backups to GCS (off-Linode) via
# the Barman Cloud plugin. See backups.yaml for the ObjectStore + schedule.
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: linode-cnpg
barmanObjectName: gcs-cnpg
3 changes: 1 addition & 2 deletions cloudnative-pg.secrets/cnpg-backup-creds.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ metadata:
namespace: cloudnative-pg
spec:
encryptedData:
ACCESS_KEY_ID: AgDl1qmQyjZlUKn+J2a+6C5NXq66dDT7M94MY4KLWZVevFYvtqFJtnJAiEyx5Rzb+r0+MzNyff5Xu5B1Uj1ckVDPQTk89d6VdI0aA4xjhJH3ChCPa6AH06pJi5vWGL+TtJB8o5snQESx8D/TBy4by+lVSQL+/0OYIEfeWLodB9bUXtUzLq8cLZTmdjIJINoOW9E0LeRKORuFHrSS7fNwoVacDq091ES9gvvQlC/pyeRh2xrOyHxfYaIL5+tZ/fPFRtexrpXdx/sowpr+faquWVTo8oekHzsKY+PPbMQfjddbLhJIhQpFJ2cM2w7W1nhKNHRRziqt+anvECUMrozTpbzdYQxVD0BtNFNoESiOLoyU1ZZWSggCR9LtJt3KYNT/BdKghB+hOS+p5bDgo0WwKGoGljrNhPaYZ44j1JZ7i7WiufFp5SPMd6tDMxiX/dVKCEZuDAwpV4iBwmBQvwbeTw9UXrkhnEzlAnOj2tRMUL6RyXnTjNSmVfTQmpykKC389vgMjWQ7sUSKWiYQ2+oTCy/gVc4xOD7IJsLVyXVG3+58vLh7sl6UpCAiJj9LmfAqM+qFleFiZJYd8rvk2hY7qigrGVKNV/m8x+Gu5V2XxhOR3AmGQtncJackhphFeLe7QrBWMePha9ooWMA12H3NEg7Xh5/TEKa/tBLElR9pNql1grCr9zzZvsuMZnVUpoIr+as0ZM/PjJEJY8+ZxXeSxBHkdQjsXg==
ACCESS_SECRET_KEY: AgDHHHC63As/z2l1RVarZqH3eT05H44bDU+24d4/SA79K3JDdNhjzLhDwhoAsnI8DKykZsjlHgYh6SPCKsy1vmC4rmQZK21mD465rl2ZTF0JdFRsn/1cHOzEt7XGPTlkyBOkcY5y7IQHuvGTRVIrGmBr4dmjEcU/u4uxycLfp73yqHYfzgSpTgGzusQvqRRw99+OlW4+4HvWbf0X8Gk1EEZJ/jMK2s+4fo1RDBEr5dP3yvShz0XxQ/RlNyJlNreOSKOYWXzMj0UxgmItgiEsu0mjn6Y7yehmgtZxFMsOd3TfKvX6ZeWjns+icvryDleY29gbVxKQXWSXQ7zjz31TJfkgrVobwj9CjVxbKGv+oE0JWhycFFA0kyAn5FmusR2+SEnBnxy9SUizx3DxWMJSGldY0n4hkd93+s6pkASUp6mRs8JSnYBaZYUWbz7Pbj/Hx9HrJrpk2X2N0w0DvgWr+T8ziI1vTcke3WfVWglNIK5PlEDUsOnB8xetQMv2jzqvhKg/YA3hYgx0AuqhpEWJPnoLghyYiFUa6VXWyHtyWQpmDrp4j/MAuDOF3os0Y7i+rfvOJKpvPfQapQFoFSctB39/UJAp1iCY7o3NzgZ4y2CdhCDHixy7Vopg1EDncAHLAZXzcoSQpeN0hhtdaSHBv4IdArjiqMWaF4Rt7NL2IVJfsz2xIOKlSNPs9XiNLGpUQETkEPbuUTZ3gwr8gcnSzGsutDYQlZdGld2v5NovyJYGHL1JYuehxyyn
gcsCredentials: AgCgzk8MLk3gX0ZKWI7NuBXt68rXIvvhsk+yI13HKO+NoGaf+Mrm4wOHXf/ukObE/yoO0gy/Gcq7ItNeG5y6Uey8ZxUvjW9km62uoogvKFwWafMsgVst/+ZzGxUgnmKEVH1fohkXB7W2pvLqY8HoCBzNW96gmY41hFdhxaDqDQrURTI+fYtyfKNuwV2a/sXu87z5/zYb8zNKjJ5hKVx6jTYeiyIGBVvhASIvrc3ZDztBY5Pwv78EkY5z3y2oaBk3q48kaKUuJp2W2S6y5FWXC2ktUDhN7iRzSGk0qd2g7NAjj+jH5wBqCiWlVzFOwhJCeeazfylt3rOBe1W8E3BJhz1BAgTj8JRAtzs1J6M50Fh5TGUvHlwWlPQPEIeqjNjcGys2ekQ7BstX8jXU+UGwWHVCC9JJpyCEpJ/L7xe5pLqDu7oExf+COGohlmruwt6F2z7e9eWy/HkPhHyewbC0xiavT66fG+icWwtWPGtLE0WIvqexoYPmmlRG7RgPAtW6Kc5YCYnnqTjJhhzTmJna5VyVXRuZszmsmK45W9J/uiOv8sv+UkzB6l1Qqfi08tNCK4RYF0Y0fazze4VN3TuHiDeUYhxpmWAlwtkO5rrBrpKSgeV4pbYIwvm8ARAUqp7P624P4wR7w1qly8x6aOgyYtLpZj4z+RXS0r0eAfcU1dcYFxlqUZyeP8gFWXwD7sXx38w/ZshFuIIy/APNhPOhbqF2SXLxnIMgMSvOl0vnalapPAJvt1WNouK5/3lOfOd8IzUp7mjkwg3S4FHysSnsrYDtQSMc+vaX7nhxA/yp4blV946Y0su+V7cujVNLopPD28uezRIoQX0l9oj7ZIEYjPwmR8zu/8Sscsp9jAQZsMYud4eQMJW9afhq+RzrAPy1k3HGZ8uM/Knvg86F/8g9UxdalM+xM2r6Gycu1qn7cFkNskKEzWAuBDSLWAxHAE7w8vZJqC0JpQnZ0ykSaRNBaV3EfvxryyyWstUYuTnj46JgArtl57Y4edLL7zLbY3D61q1bVxIhZbKUhsvwxMtdXkZf8qUr0WLMJnc81g4lI1EpqcrC1V9dBSCKzZwksE1GvWhdIX9J3ZnKiwkCXuqu4gvdRbYY6o3Er/2cUFqsqaZLOwe5zgQ5KQtBXi9G1isdbzJFrrpnNMlDy0Vx+5XtBg04Qm8AKJKwVXYgPCaj4WRI+vnuhJprgVjcw2QzwIYCIZ4AT7s1YXeii4K39fG7u1BIgXqYriNJDZze9Y80e8Ssr6u9auQn1ZkD2bTYhitC5DTvMpU6Wtcne7e8qY4v4NCM7hD/ZsreaOwmrt8heVM6hXmpJJ4h2rOY7OVm0I+WfIPS9L6skrm5tH1k1wf2PKk4aKNCgUJN8BSYOMizeZeS5Q1f3I9XRw5m1DnjL7QLFh9kVU2ChqKnsqEb+rVrCLhHJfPe+NFr0w/+CL7rHH0PiODuBsWaXDgKASVcHtSNB7L/XHX1UY9PYj3xHSPlP0Z4M0dTtzLLSpVSovFwH1Xf1+IPv+bxXb89ReLVZW46cQ+Uj226205FRbDKVfJZc84LY+hAMw0AZ5TJBHGM+Qer4s4BtwJ+uCv8lq5bmi8btcchZQLyEcRusXapY8XIJ2siNNDcyuOySMBYZHKF4rfWf0QTzWX0W44RJjlXYrRa9ht4FUpRa/+PMwhs1ob2k0KnEw9ukBy5+Ffaat6qC7ZHRvFAaDLhvJ5bwkPC254uHLK2gwubjzp3ynEZ/U38yfsYasKEe4vmI0ppufYsonau+DfTwlQJJ6I3iTwZvJFBk/1Zy5ssh1eL8JhP9N5/6pbHvHJnQ1H6dEo5Kw0cVUAiqJJZZNSJMFxiKW5qenqeEHSlkSkOtq9Gm6RU1DXXPYjMgFlGlOQ+FOYnzXvwmqB1X/FJyc9+y5ZH0sr6OvyJkGJ67LAjeMjZGfEIlfG/Z7YuEHwF+eXcckNiOyUUtyKC7kES68BiOY52FpBbEhjJzWpcCTiAsMZ+Y6M7NBoU2X0FS9fb+YLCO1QWX6BtH6vMM/LW/dy/PhRNWfipdir1qbqkPD8zGc/LRkou6QMu3QHQ3dS3s3xKnONAiZeIl6ASKGZuU6jPLUc4kMiYHh0AXv8LfXrCIZ06nvAySuw+4mN0Y5h+009+fAg5j8t0w97GqO3Sr4KvXf/MqkMysMqHHuDE0FYIFbRi01NiQoutZncCDH2BK73QGJ/uEv5vOoVmqtj21tBKhV0ut2Liw0YwatRnAQL66F0lbFgktrEDm7gaiOfPJs1ObpTXa2sRj3UCRKdIuIov0VRj5g1gS8fdSWYF+4jrWj7L/4uwHMpVcENgiIaGSb7yV7MY9rfhIiDHv/dzSHr3cDj9Jb//M/GDgrazN5gQEGVwy/DstRg76BWFg3gmlS9FxbwWcL++dIsQUGiEWydy7RWlOJde1SES/zZSUVzFOz2FC7m2NZKcCI9CW3BhFCBxxSN4VV1WD18edKuyD9XaN82dXcRpJ72shXHTHawC/uB3YAk/3z70SifMytfoTym5Qihj8QQSyEQoVRMj8mV9cV8Qt9dwVQ02vfu6GPAkcssmGRTh5qP5KyY1MMVweVpw1a3F77Q+WkhkvVOpg45hOE2IsQB26TLMoI5stcdE3edFzuQe/tQ38kvwVHQtOIM3PM0XZxgqHl8irrWTQjURh9jWTOC1IiI3ue8JDf8N7LRuMr3wCTwHrYSUN097poQd+fROWlKziWxq9Bt39SjeeVtkS/TMqgS2CyP6Qf6WOCN+9Dpo2AchjcbjIPUG5PPQHK5ZjMmObuSRZp+aYMbo/2zKsuL/LW1q3NmfcHne34+MJkblwPxCX3Jp32zhagq5aphEKbBpKRileAGdubsUlQr6XykAtukn343KIoEV9gC1/hxplHkqsnwlmw5JNBJpselQ0Wcw/HzcPwyd+fZxPY5o43FG8KLWum4JzzbIMVXM146qmj3RjCktL1zi2SQMmzFwujdEklQSvHOa/k+gXvymZGo3jqJAUL9LggxH/vunDZURq0cY2+ko81rXMamqkWrno2ceUB2g2B/DI8rOnlzPbfVxSS825xetuzjFYtNV4jFX9UhdQxFBcS1UMjdlFF9eljuvuqRWpsQOz7A7x8LMv13rEfsJIkBjT6ogIsESg58wth0P22DHyrbW/+XQDFvzovykVjJssMzFvGv1EdAVZGj2kIyUWfZReES7egGlbsfFwxlCgxvG5vhQIlOocB7Dv4V9ilR7hR9cxTz1dzkgIEZHzug34NF+oVdAOkgiK7CmUDfHgIuOBj+bKz3YtAHGFcQyJ4hRvhl8ZAhhCFAE4uwuulTcawMsuBL1/hkTg/c0cBdVgpBA0FqEvVql66QpS811LCV9dl53jeg5bqAOMWNN8D8Ah8KcLGFBauB2TQO05pmm3iX2nhPOol/WEHVOgl6TY8KtVSqX1ZLc1EGKvprdRCGXwBasTkyd1rk+qIwHdDrWJnPtgqtcmMJ6K0+AnvobwcigT7cFu7OiSQRjaJYf+98mFEh+DT2bX4XCL+0IsjvdmLVyhDxpo/T5jIjEs7ZDl8OShQD4F+zluEVqcifQ7U0w3LUqGgVOLh2eAPTSJCWDoxrKRW0tzmpn9ZSgoZiomahsCdfoJakRoCYxkUXL43jVP21yUXTipkDJc7dkBu6eD5eCiX+24JhzeN7ATBudWjfzWx6jOeASMssmjzna/XnufIF3rNNq/rcTwOBNATU1QtEBP5oCOUl8Dq5eIC1PgcrUkOs9U0hEDnmAsgK0NlH/v+soNkMd4CVlCNbiytda+A6DAiSnviI7yjVBT/1bvPHEzE2pm6p+WBXPDAE4b7VaNCurbuNJnnkCmwYb
template:
metadata:
name: cnpg-backup-creds
Expand Down