Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions suggestions/website/backup-running-gate-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
title: Backup Running Gate Contract
description: Constructional contract so create_backup archives Docker volumes without a pipefail grep of docker compose ps.
last_updated: 2026-10-30
owner: maintainers
status: proposed
theme: operations
impact: high
---

# Backup Running Gate Contract

## Problem

`create_backup` in `tu-vm.sh` copies config files, then wraps the database dump and every volume tarball in one pipeline:

```text
if docker compose ps --format json | grep -q '"State":"running"'; then
```

`tu-vm.sh` starts with `set -euo pipefail`. Under `pipefail`, `grep -q` is not a reliable predicate. It exits 0 at the first match and closes the pipe. The producer then dies with SIGPIPE. The pipeline status becomes that non-zero status, and the `if` body is skipped while a container is running.

Replayed on 2026-10-30 with Bash 5.2.21 and `set -o pipefail`, searching for the same `"State":"running"` token the script uses:

| Producer | `if` result | Pipeline status |
|---|---|---|
| One line, `{"State":"running"}` | Body runs | 0 |
| 50000 copies of that line, then a short sleep | Body skipped | 1, producer `BrokenPipeError` |
| `yes '{"State":"running"}'` | Body skipped | 141 |

Compose on this host is `Docker Compose version v5.1.0`. Its `ps` help says `-a` shows stopped containers. In `cmd/compose/ps.go` at tag `v5.1.0`, `runPs` sets `All: opts.All || len(opts.Status) != 0`. The script passes neither `-a` nor `--status`, so a stopped project prints no JSON. `grep -q` exits 1, and the body is skipped for a deterministic reason as well as the race above.

The body holds `pg_dump`, the volume loop, and the MinIO, Pi-hole, Open WebUI, and n8n snippets. Config copies above the `if` still run. The function then builds `${backup_path}.tar.gz` and prints `Backup created`. A stopped stack, or a running stack that loses the `grep -q` race, therefore publishes a config-only archive.

`docker volume inspect` does not need a running container. `pg_dump` does, and it already has its own `pg_isready` check. Those are different gates. One pipeline currently serves as both.

Stage 63 (open PR #89) decides when a bad archive may replace the previous tar, including a missing `database.sql`. This page does not change rotation. It makes the volume loop run, and it lets the existing `pg_isready` check decide the dump.

Checked on 2026-10-30 by reading `create_backup` and Compose `v5.1.0` `runPs`, and by replaying the pipeline. `tu-vm.sh` was not executed. No volume was read.

## Existing scan (reuse first)

| Asset | Role |
|---|---|
| `create_backup` | The `if` to remove from around the volume loop |
| `docker volume inspect` in that loop | Already skips names that are absent |
| `pg_isready` in the same function | The live-server gate for `pg_dump` |
| `set -euo pipefail` | Why `grep -q` on a pipe can skip the body |
| Compose `v5.1.0` `ps` | `--status running` and `-q` already exist |
| Stage 46 volume prefix (open PR #72) | Which names are inspected. Unchanged here |
| Stage 47 restore (open PR #73) | How an archive is applied. Unchanged here |
| Stage 63 image pin, empty dump, archive mode (open PR #89) | Image, rotation, and file mode. Unchanged here |

Out of scope:

- A second backup product
- Keep-latest rotation rules (Stage 63)
- Replacing `alpine` in the volume loop (Stage 63)
- `chmod` of the outer tar (Stage 63)
- Pi-hole `pihole -g` (Stage 53, open PR #79)
- The `n8n-workflows.json` redirection ([n8n workflow file export](./n8n-workflow-file-export-contract.md))

## Proposal

Archive volumes whenever `docker volume inspect` succeeds. Decide the logical dump only with `pg_isready`.

### Contribution lanes

| Lane | Where | Suggested change |
|---|---|---|
| Volume loop | `create_backup` | Move the loop out of the `docker compose ps \| grep -q` body |
| Dump | Same function | Leave `pg_dump` behind `pg_isready` |
| Service snippets | Same function | Keep each `docker exec` behind its own container probe |
| Predicate | Delete the pipeline | Do not replace it with another `grep -q` under `pipefail` |
| Running check, if one remains | Same function | `docker compose ps --status running -q` captured into a variable, or `docker inspect` as `check_service_health` already does |

### Rules

1. **A stopped project still gets volume tarballs.** `docker volume inspect` is the presence test. `docker compose ps` without `-a` is not.
2. **`pg_dump` stays on `pg_isready`.** A stopped Postgres cannot take a logical dump. Its volume tarball is the copy for that run.
3. **`grep -q` is not the running test.** The three-row table above is the regression. A match in the output must take the true branch when `pipefail` is on.
4. **Config-only success is not a data backup.** The final `Backup created` line is allowed only after the volume loop has run. Stage 63 still decides whether that archive replaces the previous one.
5. **Leave Stage 63's three edits alone.** Image pin, empty-dump rotation, and archive mode stay on that PR.

### Suggested contributor checklist

```text
1. Confirm the volume loop is outside any docker compose ps | grep pipeline
2. Confirm a stopped project with an existing volume still writes that volume's tarball
3. Confirm pg_dump still runs only after pg_isready
4. Replay the pipefail table: a slow producer of "State":"running" must not skip the body
5. Confirm rotation, the alpine image line, and chmod stay unedited
6. Confirm the n8n find/cat line is not edited on this change
```

### Framework reuse

| Need | Prefer | Avoid |
|---|---|---|
| Volume bytes | `docker volume inspect` and the existing `docker run` loop | A volume-backup plugin |
| Logical dump | `pg_isready` already in this function | Gating the dump on `grep -q` |
| Running filter | Compose `--status running -q` | `grep -q` on a `pipefail` pipeline |
| Intake | GitHub Issue | A second backup command |
94 changes: 94 additions & 0 deletions suggestions/website/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
---
title: Community Suggestions Website
description: Publishable Stage 64 website markdown for the backup running-state gate, the n8n workflow file, and the status accessibility curl.
last_updated: 2026-10-30
owner: maintainers
status: proposed
---

# Community Suggestions Website

This directory holds **publishable website markdown**: curated pages a future static docs site can ship without a second editable copy.

Participation stays on GitHub. These pages say which existing TU-VM tools to extend, and which small contracts keep day-to-day backups and status accurate for a community of operators and contributors.

## Why this exists

TU-VM already has the community path and the operator tools: GitHub Issues, `CONTRIBUTING.md`, `tu-vm.sh`, the helper API, the Nginx landing page, the document processor, playbooks, and CI. Historical suggestion files in `suggestions/` and open website PRs #27–#89 already specify the docs framework, governance, and most product contracts.

**Stage 64** (this run) records three defects checked on 2026-10-30. They are not restatements of Stages 1–63.

- `create_backup` runs `pg_dump` and the volume loop only when `docker compose ps --format json | grep -q '"State":"running"'` succeeds. `tu-vm.sh` uses `set -o pipefail`. `grep -q` closes the pipe at the first match, and a slow producer then fails the `if` while a container is running. Compose v5.1.0 also omits stopped containers unless `-a` or `--status` is set, so a stopped project skips the volume tarballs entirely. `docker volume inspect` does not need a running container. `pg_isready` already gates the dump.
- The same function writes `n8n-workflows.json` with `find /home/node/.n8n -name '*.json' -exec cat`. Two JSON files become one member that `json.loads` rejects. No JSON files still create a zero-byte member, because `find` exits 0. Compose sets `DB_TYPE=postgresdb` on the database `pg_dump` already saves. n8n 2.25.7 already ships `export:workflow --all --output=<file>` for a single JSON array.
- `show_access_info` curls `https://$vm_ip` with `-k -s` and no `--fail` or `--max-time`, then prints "Services are accessible". A 500 response exits 0. A peer that accepts TCP and stalls was still running after 3 seconds. The landing server serves `index.html` itself, so that curl does not read the Tier 1 checks `show_status` already printed.

The contracts:

- Archive volumes when `docker volume inspect` succeeds. Leave `pg_dump` on `pg_isready`. Do not use `grep -q` on a `pipefail` pipeline as the gate.
- Delete the `find` and `cat` workflow member. Keep the Postgres dump. If a portable file remains, use `n8n export:workflow --all --output=<file>` and pack it only when it is one JSON array.
- Drive the status summary from `check_service_health`. A static dashboard response is the landing page, not "services".

No second backup product, workflow store, or monitoring agent.

## Boundaries (do not reinvent)

| Already exists | Website role |
|---|---|
| GitHub Issues + suggestion form | Sole proposal intake |
| [`CONTRIBUTING.md`](../../CONTRIBUTING.md) | Contributor path and labels |
| [`SECURITY.md`](../../SECURITY.md) | Vulnerability reporting |
| [`docs/playbooks/`](../../docs/playbooks/README.md) | Operator recipes |
| Nginx landing page | Local control plane |
| Historical files in [`suggestions/`](../) | Archive and planning source |
| Stage 1–63 website pages (open PRs #27–#89) | Implement those contracts in code; do not rewrite them here |
| `create_backup` in `tu-vm.sh` | The function the backup pages change |
| `check_service_health` | The status signal the curl page should print |
| `pg_isready` inside `create_backup` | The dump gate that stays |
| n8n `export:workflow` in the pinned 2.25.7 image | The optional portable file |

Stage 63 (open PR #89) owns the floating `alpine` volume helper, keep-latest promotion of an empty dump, and the archive mode. Stage 61 (open PR #87) owns creating the n8n Postgres schema. Stage 52 (open PR #78) owns `test_endpoints` SNI. Stage 57 (open PR #83) owns the `tu.lan` labels in `show_access_info`. This stage does not edit those lines.

## Page map (Stage 64)

| Page | Purpose |
|---|---|
| [Backup running gate](./backup-running-gate-contract.md) | Volume archives do not depend on `grep -q` of `docker compose ps` |
| [n8n workflow file export](./n8n-workflow-file-export-contract.md) | `n8n-workflows.json` is not a concatenation of every JSON file |
| [Status access curl](./status-access-curl-contract.md) | "Services are accessible" follows Tier 1 health, not a static curl |

## Expected sibling pages

Stages 1–63 live on open PRs #27–#89 and are not files on this branch. When those PRs merge, keep their pages as siblings in this folder. Do not copy them here.

Representative filenames only: `backup-volume-image-pin-contract.md`, `backup-rotation-empty-dump-contract.md`, `backup-archive-mode-contract.md`, `n8n-postgres-schema-contract.md`, `endpoint-probe-sni-contract.md`, `pihole-domain-suffix-contract.md`.

Extend `stage-merge-playbook.md` through **Stage 64** when reconciling hubs so earlier pages are not dropped.

## Framework reuse (summary)

| Need | Prefer | Avoid |
|---|---|---|
| Proposal intake | GitHub Issues | A suggestion database |
| Docs site | Static framework already proposed in Stage 3 (Starlight remains the default) | A custom community portal |
| Volume bytes | `docker volume inspect` and the existing loop | A volume-backup plugin |
| Workflow rows | `pg_dump` of `ai_platform` | `cat` of `*.json` |
| Status line | `check_service_health` | An unbounded curl of the dashboard |

## Navigation for a future docs site

Map this folder to a `Community` section:

1. How to submit, status, and decisions (Stages 1–4, when those PRs are merged)
2. Operator day-to-day: volume backups run when the volumes exist
3. Operator day-to-day: the n8n workflow member is one JSON array or is absent
4. Operator day-to-day: `tu-vm.sh status` agrees with the Tier 1 health rows

When a static site is adopted, move or mount these pages into the docs content root. Do not keep two editable copies.

## Related archive and planning docs

- [`../README.md`](../README.md) — suggestions hub
- [`../index.md`](../index.md) — community suggestions map
- [`../implementation-backlog.md`](../implementation-backlog.md) — trimmed execution backlog
- [`../historical-suggestions.md`](../historical-suggestions.md) — changelog ideas
- [`../website-roadmap-from-historical-suggestions.md`](../website-roadmap-from-historical-suggestions.md) — historical product direction
104 changes: 104 additions & 0 deletions suggestions/website/n8n-workflow-file-export-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
---
title: n8n Workflow File Export Contract
description: Constructional contract so create_backup does not concatenate every JSON file under n8n's data directory into n8n-workflows.json.
last_updated: 2026-10-30
owner: maintainers
status: proposed
theme: operations
impact: high
---

# n8n Workflow File Export Contract

## Problem

`create_backup` writes `n8n-workflows.json` by concatenating every JSON file in the n8n data directory:

```text
docker exec ai_n8n find /home/node/.n8n -name "*.json" -exec cat {} \; > "$backup_path/n8n-workflows.json" 2>/dev/null || warn "n8n workflows backup failed"
```

The comment above that line says workflows live in the `n8n_data` volume and mentions an API export. The command does neither. It cats files. The shell opens `n8n-workflows.json` before `docker exec` runs.

Compose sets `DB_TYPE: postgresdb` and `DB_POSTGRESDB_DATABASE` to the same database `pg_dump` already targets (`ai_platform`, default user `ai_admin`). With that driver, workflow rows are in Postgres. Stage 61 (open PR #87) owns creating the `n8n` schema those rows use. This page does not create a schema and does not add a workflow database.

Replayed on 2026-10-30 with the same `find -exec cat` shape, outside Docker:

| Fixture under `.n8n` | File written | `json.loads` |
|---|---|---|
| `custom/workflow.json` then `nodes/package.json` | 56 bytes, two JSON values back to back | `JSONDecodeError: Extra data` |
| Directory with no `*.json` | 0 bytes, and `find` still exits 0 | Empty file, not a workflow list |

`find` exits 0 when it searches and matches nothing, so the `|| warn` branch does not run. The archive member is named like a workflow export and is either empty or not one JSON value.

n8n image `n8nio/n8n@sha256:761374d4eb84` is the 2.25.7 release. Its `export:workflow` command at tag `n8n@2.25.7` (`packages/cli/src/commands/export/workflow.ts`) writes `JSON.stringify` of the workflow array when `--output` is a file. `--all` selects every workflow. `--backup` is a different switch: it forces `--separate` and one file per workflow. The no-match path throws `No workflows found with specified filters` instead of writing a zero-byte file. Without `--output`, the command sends the text through `logger.info`, so a stdout redirect is not the file format.

The volume loop already attempts `docker_n8n_data`. The logical copy is the `pg_dump` of `ai_platform` in the same function. The concatenated member is a third copy with the wrong bytes.

Checked on 2026-10-30 by reading `create_backup`, the `n8n` service environment, and the n8n 2.25.7 export command. The `find` shape was replayed in a temporary directory. `tu-vm.sh` was not executed and the n8n image was not pulled.

## Existing scan (reuse first)

| Asset | Role |
|---|---|
| `create_backup` `pg_dump` of `ai_platform` | Logical copy of the database n8n is configured to use |
| `n8n_data` volume mounted at `/home/node/.n8n` | Filesystem copy, once the volume loop runs |
| `DB_TYPE: postgresdb` | Why workflow rows are not the JSON files under `.n8n` |
| n8n `export:workflow` in the pinned 2.25.7 CLI | Optional portable file, one JSON array |
| Stage 28 Postgres DNS (open PR #54) | `DB_POSTGRESDB_HOST`. Unchanged here |
| Stage 61 schema (open PR #87) | Creating schema `n8n`. Unchanged here |
| Stage 46 volume prefix (open PR #72) | The `n8n_data` tarball name. Unchanged here |
| [Backup running gate](./backup-running-gate-contract.md) | The `if` around this line. That page does not rewrite the `find` |

The running-gate page moves the volume loop. It leaves this `find` line in place. This page owns the `find` line only.

Out of scope:

- A workflow marketplace or a second database
- `export:credentials`
- n8n's `--backup` switch (separate files, one per workflow)
- Decrypting credential material
- Schema creation (Stage 61)
- MinIO bucket `n8n-workflows` (artifact storage, not this member)

## Proposal

Stop writing `n8n-workflows.json` from `find` and `cat`. Keep the Postgres dump as the workflow backup.

### Contribution lanes

| Lane | Where | Suggested change |
|---|---|---|
| Delete | The `find` / `cat` line | Remove it. Do not replace it with another directory walk |
| Logical copy | Existing `pg_dump` | Leave it. It already names `ai_platform` |
| Optional portable file | Only if a JSON member is still required | `docker exec ai_n8n n8n export:workflow --all --output=/tmp/n8n-workflows.json`, then copy that file into the backup directory |
| Empty result | Same optional command | On `No workflows found`, warn and omit the member. Do not leave a zero-byte file |
| Format check | Optional file | The member parses as one JSON array before it is packed |

### Rules

1. **The archive member is one JSON value or it is absent.** Concatenated objects and a zero-byte file both fail that rule.
2. **Postgres remains the workflow store.** `DB_TYPE=postgresdb` plus `pg_dump` of `ai_platform` is the backup. Stage 61 still owns the schema.
3. **A portable file uses `export:workflow --all --output=<file>`.** Copy the file out of the container. Do not redirect `logger.info` stdout. Do not pass `--backup` or `--separate`.
4. **Do not export credentials here.** The 2.25.7 workflow command's flags are `all`, `backup`, `id`, `projectId`, `output`, `pretty`, `separate`, `version`, and `published`. Credential export is a different command and stays out of `create_backup`.
5. **Do not walk `.n8n` for `*.json`.** That tree holds node packages and local files. It is already the `n8n_data` volume tarball when the volume loop runs.

### Suggested contributor checklist

```text
1. Confirm create_backup has no find -exec cat of /home/node/.n8n
2. Confirm a fixture of two JSON objects is not written as n8n-workflows.json
3. Confirm a data directory with no JSON files does not produce a zero-byte member
4. If the optional export remains, confirm it uses --all --output and the member is one JSON array
5. Confirm export:credentials and --separate are not used
6. Confirm the volume-loop image, rotation, and the compose ps gate are not edited on this change
```

### Framework reuse

| Need | Prefer | Avoid |
|---|---|---|
| Workflow rows | `pg_dump` of `ai_platform` | `cat` of every `*.json` |
| Portable file | n8n `export:workflow` already in the pinned image | A custom exporter |
| File format | One JSON array from `--output` | Logger stdout, or `--separate` |
| Intake | GitHub Issue | A workflow catalog service |
Loading
Loading