Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions suggestions/website/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
---
title: Community Suggestions Website
description: Publishable Stage 54 website markdown for secret generation that keeps distinct passwords, PDF commands that probe the pipeline they already have, Tika metadata that fits object headers, and a MinIO mount config that follows password changes.
last_updated: 2026-10-10
owner: maintainers
status: proposed
---

# Community Suggestions Website

This directory holds **publishable website markdown**: curated pages a future static docs site can ship without a second editable copy.

Participation stays on GitHub. These pages say which existing TU-VM tools to extend, and which small contracts keep day-to-day operation accurate for a community of operators and contributors.

## Why this exists

TU-VM already has the community path and the operator tools: GitHub Issues, `CONTRIBUTING.md`, `tu-vm.sh`, the helper API, the Nginx landing page, the document processor, playbooks, and CI. Historical suggestion files in `suggestions/` and open website PRs #27–#79 already specify the docs framework, governance, and most product contracts.

**Stage 54** (this run) records four defects checked on 2026-10-10 against the tree on `dev`. They are not restatements of Stages 1–53.

- `generate_secrets` replaces every `CHANGE_ME_SECURE_PASSWORD` with the Postgres password before `set_env_var_if_default` runs. Redis, n8n, MinIO, the knowledge-base MinIO key, and Pi-hole keep that one value. The passwords the function prints for Redis, n8n, MinIO, and Pi-hole are not the values in `.env`. `MCP_PROOF_SIGNING_KEY` stays `CHANGE_ME_PROOF_HMAC`.
- `check_pdf_processing_status` curls `http://localhost:9998/tika`. The Tika service does not publish that port. The same function and `test_pdf_processing` talk to MinIO with `${MINIO_ROOT_PASSWORD:-minio123456}` and never read `.env`. `reset_pdf_pipeline` and `resolve_minio_password` already do.
- `store_processed_content` caps extracted metadata at 1500 characters and then percent-encodes it. A 709-character JSON value encodes to 2125 bytes. A Tika-shaped object of 2640 characters is cut to 1514 and still encodes to 2406 bytes. S3 user metadata is 2 KiB. `PROCESSOR_MAX_FILE_ATTEMPTS` is `0`, so the put keeps retrying.
- `mount_minio_storage` writes `/etc/rclone/rclone.conf` only when the `[minio]` section is absent. A later `MINIO_ROOT_PASSWORD` change leaves the host mounts on the old secret.

The contracts:

- Give each published placeholder its own generated value, write that value, and print what the file contains. Sign MCP proofs with a generated key.
- Point the PDF commands at the Compose network and at `resolve_minio_password`.
- Bound metadata after encoding, and keep the processed text when metadata does not fit.
- Refresh the rclone secret when the MinIO password changes.

No new suggestion queue, voting database, secret manager, PDF product, or mount daemon.

## Boundaries (do not reinvent)

| Already exists | Website role |
|---|---|
| GitHub Issues + suggestion form | Sole proposal intake |
| [`CONTRIBUTING.md`](../../CONTRIBUTING.md) | Contributor path and labels |
| [`SECURITY.md`](../../SECURITY.md) | Vulnerability reporting |
| [`docs/playbooks/`](../../docs/playbooks/README.md) | Operator recipes |
| Nginx landing page | Local control plane |
| Historical files in [`suggestions/`](../) | Archive and planning source |
| Stage 1–53 website pages (open PRs #27–#79) | Implement those contracts in code; do not rewrite them here |
| `set_env_var_if_default` | The per-key writer the global replace currently skips past |
| `resolve_minio_password` | The password lookup the PDF commands should call |
| `run_minio_mc` | The MinIO client wrapper for bucket checks |
| `ai_helper_index` with `curl` | A Tier 1 client already on the Docker network |
| `store_processed_content` | The uploader that should keep text even when metadata is large |
| rclone install inside `mount_minio_storage` | The mount tool to keep |

Do **not** republish Stage 53's update wrapper, probe credentials, Pi-hole gravity refresh, or start-gate default list. Stage 53 owns which strings the start gate recognizes. This stage owns what `generate_secrets` writes on the first copy of `env.example`.

## Page map (Stage 54)

| Page | Purpose |
|---|---|
| [Secret generation identity](./secret-generation-identity-contract.md) | One generated value per service, and a proof key that is not a public placeholder |
| [PDF command probes](./pdf-cli-probe-contract.md) | Pipeline checks reach Tika on the Docker network and MinIO with the password from `.env` |
| [Processor metadata header](./processor-metadata-header-contract.md) | Extracted text is stored when Tika metadata does not fit in the object header |
| [rclone MinIO config](./rclone-minio-config-contract.md) | Host mounts follow `MINIO_ROOT_PASSWORD` after the first write |

## Expected sibling pages

Stages 1–53 live on open PRs #27–#79 and are not files on this branch. When those PRs merge, keep their pages as siblings in this folder. Do not copy them here.

Representative filenames only: `startup-secret-gate-contract.md`, `check-config-secret-coverage-contract.md`, `tika-pipeline-readiness-contract.md`, `minio-mc-client-transport-contract.md`, `processor-object-byte-budget-contract.md`, `offsite-backup-rclone-contract.md`.

Extend `stage-merge-playbook.md` through **Stage 54** when reconciling hubs so earlier pages are not dropped.

## Framework reuse (summary)

| Need | Prefer | Avoid |
|---|---|---|
| Proposal intake | GitHub Issues | A suggestion database |
| Docs site | Static framework already proposed in Stage 3 (Starlight remains the default) | A custom community portal |
| First-run secrets | `generate_secrets` and `set_env_var_if_default` | A second secrets file or a vault |
| PDF checks | `resolve_minio_password`, `run_minio_mc`, helper `curl` | Publishing port 9998 on the host |
| Object metadata | The existing `put_object` call, with a smaller header | A new metadata database |
| Host mounts | The rclone config this script already writes | A second mount agent |

## Navigation for a future docs site

Map this folder to a `Community` section:

1. How to submit, status, and decisions (Stages 1–4, when those PRs are merged)
2. Operator day-to-day: first-run secrets that match the printed card
3. Operator day-to-day: PDF checks and MinIO mounts that use the password in `.env`
4. Contributor day-to-day: GitHub Issues and the existing playbooks

When a static site is adopted, move or mount these pages into the docs content root. Do not keep two editable copies.

## Related archive and planning docs

- [`../README.md`](../README.md) — suggestions hub
- [`../index.md`](../index.md) — community suggestions map
- [`../implementation-backlog.md`](../implementation-backlog.md) — trimmed execution backlog
- [`../historical-suggestions.md`](../historical-suggestions.md) — changelog ideas
- [`../website-roadmap-from-historical-suggestions.md`](../website-roadmap-from-historical-suggestions.md) — historical product direction
100 changes: 100 additions & 0 deletions suggestions/website/pdf-cli-probe-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
---
title: PDF Command Probe Contract
description: Constructional contract so tu-vm.sh PDF commands reach Tika on the Docker network and MinIO with the password already stored in .env.
last_updated: 2026-10-10
owner: maintainers
status: proposed
theme: tooling
impact: high
---

# PDF Command Probe Contract

## Problem

`check_pdf_processing_status` reports Tika as down while the Tier 1 container is up, and it reports the `tika-pipe` bucket as missing when the bucket exists.

The Tika block curls the host:

```bash
if curl -s -f "http://localhost:9998/tika" >/dev/null 2>&1; then
echo " ✅ Tika API is responding"
else
echo " ⚠️ Tika API not responding"
fi
```

The `tika` service in `docker-compose.yml` has a container port of 9998 and no `ports:` entry. MinIO publishes `127.0.0.1:9000` and `127.0.0.1:9001`. Tika does not. From the host, `localhost:9998` is closed whenever the stack is running as composed. The warning is what a healthy install prints.

The bucket check and `test_pdf_processing` start `minio/mc` with the shell's `MINIO_ROOT_PASSWORD` or the published default:

```bash
mc alias set local http://ai_minio:9000 admin '${MINIO_ROOT_PASSWORD:-minio123456}'
```

`check_pdf_processing_status` and `test_pdf_processing` never read `.env`. A normal operator session does not export `MINIO_ROOT_PASSWORD`. The client then authenticates as `minio123456`. MinIO, after `generate_secrets`, is not that password. `mc ls` fails, and the script prints `tika-pipe bucket not found`.

`reset_pdf_pipeline` already loads the key from `.env` when the shell variable is empty. `resolve_minio_password` does the same lookup for bucket setup. The PDF status and PDF test commands are the two that skipped it.

Checked on 2026-10-10 from `tu-vm.sh` and `docker-compose.yml`. The Docker daemon was not required. The compose service has no host port, and the shell default is the string in those two `mc` lines.

Stage 48 (open PR #74) owns the missing Tika healthcheck and the image that has no `curl`. Stage 46 (open PR #72) owns `run_minio_mc` and its host-network client. This page is the `pdf-status` / `pdf-test` path, which does not call either of those.

## Existing scan (reuse first)

| Asset | Role |
|---|---|
| `check_pdf_processing_status` | The status command that curls the host and uses the default MinIO password |
| `test_pdf_processing` | The upload test with the same password default |
| `reset_pdf_pipeline` | The PDF command that already reads `.env` |
| `resolve_minio_password` | Shared lookup for `.env`, the shell, then the container |
| `run_minio_mc` | The `mc` wrapper bucket setup already uses |
| `ai_helper_index` | Tier 1 container whose start command installs `curl` on `ai_network` |
| Compose DNS name `tika` | The service hostname on that network |
| Stage 48 `tika-pipeline-readiness-contract.md` (expected sibling, open PR #74) | Healthcheck and `service_started` |
| Stage 46 `minio-mc-client-transport-contract.md` (expected sibling, open PR #72) | Transport for `run_minio_mc` |

Out of scope:

- Publishing port 9998 on the host
- A new PDF application or a second MinIO client
- The Tika healthcheck binary (Stage 48)
- The `run_minio_mc` network mode (Stage 46)

## Proposal

Probe Tika from a container that is already on `ai_network`. Take the MinIO password from `resolve_minio_password`.

### Contribution lanes

| Lane | Where | Suggested change |
|---|---|---|
| Tika | `check_pdf_processing_status` | Replace the host curl with `docker exec ai_helper_index curl -fsS http://tika:9998/tika` |
| Password | `check_pdf_processing_status` and `test_pdf_processing` | `minio_pass="$(resolve_minio_password)"` and pass that value to `mc` |
| Client | those two functions | Call `run_minio_mc` for `ls`, `mb`, and `cp` once Stage 46's transport is the one in tree |
| Wording | the bucket failure branch | When `mc` cannot authenticate, say the password was rejected, and keep `bucket not found` for a real missing bucket |

### Rules

1. **A running Tika container with a closed host port is healthy.** The status command reaches `tika:9998` on the Docker network.
2. **The password is the one in `.env`.** An unset shell variable does not select `minio123456` when `.env` has `MINIO_ROOT_PASSWORD`.
3. **The host port stays closed.** The check does not add a `ports:` entry for Tika.
4. **Reset, status, and test agree.** All three use the same password lookup.

### Suggested contributor checklist

```text
1. With ai_tika and ai_helper_index running, pdf-status prints that the Tika API is responding
2. Nothing listens on host port 9998
3. With MINIO_ROOT_PASSWORD only in .env, pdf-status lists tika-pipe instead of saying the bucket is missing
4. pdf-test uploads with that same password
5. An unset .env and an unset shell variable still falls through to the published default inside resolve_minio_password
```

### Framework reuse

| Need | Prefer | Avoid |
|---|---|---|
| HTTP client on the Docker network | `curl` already installed in `ai_helper_index` | A host port or a new image |
| MinIO password | `resolve_minio_password` | `${MINIO_ROOT_PASSWORD:-minio123456}` in the PDF commands |
| `mc` invocation | `run_minio_mc` | A third `docker run minio/mc` copy |
99 changes: 99 additions & 0 deletions suggestions/website/processor-metadata-header-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
---
title: Processor Metadata Header Contract
description: Constructional contract so the document processor stores extracted text when Tika metadata does not fit in the S3 user-metadata limit.
last_updated: 2026-10-10
owner: maintainers
status: proposed
theme: tooling
impact: high
---

# Processor Metadata Header Contract

## Problem

`store_processed_content` in `tika-minio-processor/universal_processor.py` attaches Tika's metadata JSON to the processed object. The size check runs before percent-encoding. The encoded value is what S3 counts, and it can be larger than the 2 KiB user-metadata limit even when the check passed.

The function does this:

```python
metadata_json = json.dumps(metadata, ensure_ascii=True)
max_len = 1500
if len(metadata_json) > max_len:
metadata_json = metadata_json[:max_len] + "...(truncated)"
s3_metadata['extracted-metadata'] = quote(metadata_json, safe='')
```

`quote(..., safe='')` encodes every character that is not a letter, digit, or `_.-`. A quote becomes `%22`. A space becomes `%20`. A backslash becomes `%5C`. `json.dumps` writes quotes, colons, and spaces throughout, and `ensure_ascii=True` writes non-ASCII as `\uXXXX`, which then encodes the backslash as well.

Replayed on 2026-10-10 with the same `json.dumps` and `quote` calls. No object was uploaded.

| Payload | JSON length | After the 1500 cut | `quote()` length |
|---|---|---|---|
| One value of 350 quote characters | 709 | not cut | 2125 |
| `dc:title`, `dc:creator`, and 40 `pdf:docinfo:custom:field_N` values of `value N "quoted" and spaced` | 2640 | 1514 | 2406 |

2125 and 2406 are both past 2048 before the other user-metadata headers. The put also sends `processed-by`, `original-filename`, `processing-timestamp`, `processing-date`, `processing-method`, and `original-content-type`. On the second object those headers bring the block to about 2791 bytes.

`put_object` then fails. `process_file_from_minio` returns false. The watcher records a retry. Compose sets `PROCESSOR_MAX_FILE_ATTEMPTS=0`, and the watcher treats `0` as unlimited. The same file is attempted on every `CHECK_INTERVAL` (10 seconds) for as long as the processor runs. The extracted text never lands, so the pipeline looks busy and produces no `.txt`.

The 1500 comment says the cut is conservative enough to survive quoting. The table is the opposite: a body under 1500 can encode past 2048, and a body over 1500 is still past 2048 after the cut.

Stage 41 (open PR #67) owns how many bytes of the object body are read and how many archive members are expanded. Stage 52 (open PR #78) owns `.tar.gz` identity and the output stem. This page is the user-metadata header on a successful extract.

## Existing scan (reuse first)

| Asset | Role |
|---|---|
| `store_processed_content` | Builds `Metadata` and calls `put_object` |
| `quote` from `urllib.parse` | The encoder the size check ignores |
| `process_file_from_minio` | Treats a failed store as a failed file |
| `UniversalMinIOWatcher` | Retries forever when `PROCESSOR_MAX_FILE_ATTEMPTS` is `0` |
| Tika `/meta` response | The dict stored under `extracted-metadata` |
| Stage 41 `processor-object-byte-budget-contract.md` (expected sibling, open PR #67) | Body size and archive expansion |
| Stage 52 `processor-archive-identity-contract.md` (expected sibling, open PR #78) | Archive suffix and output stem |

Out of scope:

- A side database or a second bucket for metadata
- Raising MinIO's metadata limit
- Changing archive detection or the object-body cap
- Dropping extracted text to make the header fit

## Proposal

Measure the encoded header. Keep the text object when metadata does not fit.

### Contribution lanes

| Lane | Where | Suggested change |
|---|---|---|
| Budget | `store_processed_content` | Percent-encode first, then shorten until `extracted-metadata` plus the other `x-amz-meta-*` headers is at most 2048 bytes |
| Cut | the same function | Shorten on a JSON boundary, or drop `extracted-metadata` and set `metadata-omitted=too-large` |
| Store | `put_object` | Upload the extracted text either way. A metadata failure is a log line, not a failed file |
| Retry | watcher | Leave `PROCESSOR_MAX_FILE_ATTEMPTS` alone. This path should not be a retry |

### Rules

1. **The encoded size is the size that counts.** A JSON string under 1500 characters still fails the budget when `quote()` makes it longer than the remaining header room.
2. **Text is the product.** The `.txt` object is written when Tika returned text, including when metadata is omitted.
3. **Omission is visible.** The object metadata says the Tika dict was left off, so an operator can tell a short header from a failed extract.
4. **Retries stay for real failures.** Download errors and Tika errors still use the existing backoff.

### Suggested contributor checklist

```text
1. Replay quote(json.dumps({"k": '"' * 350}, ensure_ascii=True), safe='') and confirm the encoded length is 2125
2. A store call with that dict still puts the text body
3. The resulting user-metadata block is at most 2048 bytes
4. A store call whose metadata fits unchanged still sends extracted-metadata
5. The watcher does not schedule another attempt solely because metadata was omitted
```

### Framework reuse

| Need | Prefer | Avoid |
|---|---|---|
| Header limit | The 2 KiB S3 user-metadata budget `put_object` already has to satisfy | A custom metadata service |
| Encoding | `urllib.parse.quote`, already imported | A second encoder |
| Object body | The current `put_object` of the extracted text | A new bucket layout |
Loading
Loading