Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions suggestions/website/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
title: Community Suggestions Website
description: Publishable Stage 60 website markdown for a dead Open WebUI metadata flag, a Tika /meta call that receives CSV, and unpinned images in the upload sync.
last_updated: 2026-10-26
owner: maintainers
status: proposed
---

# Community Suggestions Website

This directory holds **publishable website markdown**: curated pages a future static docs site can ship without a second editable copy.

Participation stays on GitHub. These pages say which existing TU-VM tools to extend, and which small contracts keep day-to-day operation accurate for a community of operators and contributors.

## Why this exists

TU-VM already has the community path and the operator tools: GitHub Issues, `CONTRIBUTING.md`, `tu-vm.sh`, the helper API, the Nginx landing page, the document processor, playbooks, and CI. Historical suggestion files in `suggestions/` and open website PRs #27–#85 already specify the docs framework, governance, and most product contracts.

**Stage 60** (this run) records three defects checked on 2026-10-26 against the tree on `dev`, the pinned Open WebUI revision, and Apache Tika 3.3.1. They are not restatements of Stages 1–59.

- `docker-compose.yml` sets `TIKA_METADATA_EXTRACTION: "true"` on `open-webui`. A search of Open WebUI revision `02dc3e689ceac915a870b373318b99c029ddf603` finds no such name. `TikaLoader.load` reads `X-TIKA:content` from the `/tika/text` JSON body and returns a `Document` whose metadata is only the request headers plus `Content-Type`.
- `tika-minio-processor/universal_processor.py` `PUT`s the file to `/meta` with `Content-Type` and no `Accept`. `requests` 2.32.5 then sends `Accept: */*`. Tika 3.3.1 `MetadataResource` produces `text/csv` and `application/json`, and the 3.3.1 server README shows `curl -T file http://localhost:9998/meta` as CSV. `response.json()` fails, the exception is swallowed, and the stored object has no `extracted-metadata`.
- `scripts/sync-openwebui-minio.sh` lists the upload volume with `alpine:latest` and runs untagged `minio/mc` (Docker tag `latest`) three times. Compose pins service images by digest. Those two references are outside that pin list.

The contracts:

- Delete the unread metadata flag. Leave the Open WebUI loader file alone. Keep document metadata on the processor path.
- Send `Accept: application/json` on the processor `/meta` call. Do not put that JSON on the object until the Stage 54 header budget is in the same change.
- Stop pulling `alpine:latest`. List the volume with the pinned Open WebUI image. Pin `minio/mc` the same way Compose pins `minio/minio`.

No metadata database, Tika fork, or second sync service.

## Boundaries (do not reinvent)

| Already exists | Website role |
|---|---|
| GitHub Issues + suggestion form | Sole proposal intake |
| [`CONTRIBUTING.md`](../../CONTRIBUTING.md) | Contributor path and labels |
| [`SECURITY.md`](../../SECURITY.md) | Vulnerability reporting |
| [`docs/playbooks/`](../../docs/playbooks/README.md) | Operator recipes |
| Nginx landing page | Local control plane |
| Historical files in [`suggestions/`](../) | Archive and planning source |
| Stage 1–59 website pages (open PRs #27–#85) | Implement those contracts in code; do not rewrite them here |
| `TikaLoader` in the pinned Open WebUI image | Text extraction the community already selected |
| `universal_processor.py` `/meta` call | The metadata store this stack owns |
| `scripts/sync-openwebui-minio.sh` | The cron sync to keep |

Do **not** republish Stage 59's PDF engine keys, checkup `UPDATE`, or Tika `:latest` update pair. Stage 58 (open PR #84) owns `tika-config.xml` and the full Tika image. Stage 54 (open PR #80) owns the encoded S3 user-metadata length. Stage 46 (open PR #72) owns `MC_HOST_local` and `--network host`. Stage 44 (open PR #70) owns digest comparison for Compose `image:` lines. This stage does not restate those.

## Page map (Stage 60)

| Page | Purpose |
|---|---|
| [Open WebUI Tika metadata flag](./openwebui-tika-metadata-flag-contract.md) | The Compose flag is unread; the loader keeps text, not a metadata record |
| [Processor meta Accept](./processor-meta-accept-contract.md) | `/meta` is asked for JSON, so the CSV default is not parsed as an object |
| [Sync client image pin](./sync-client-image-pin-contract.md) | The upload sync does not pull `alpine:latest` or untagged `minio/mc` |

## Expected sibling pages

Stages 1–59 live on open PRs #27–#85 and are not files on this branch. When those PRs merge, keep their pages as siblings in this folder. Do not copy them here.

Representative filenames only: `openwebui-pdf-engine-keys-contract.md`, `tika-ocr-runtime-contract.md`, `processor-metadata-header-contract.md`, `minio-mc-client-transport-contract.md`, `image-update-compare-contract.md`.

Extend `stage-merge-playbook.md` through **Stage 60** when reconciling hubs so earlier pages are not dropped.

## Framework reuse (summary)

| Need | Prefer | Avoid |
|---|---|---|
| Proposal intake | GitHub Issues | A suggestion database |
| Docs site | Static framework already proposed in Stage 3 (Starlight remains the default) | A custom community portal |
| Chat-upload text | Pinned Open WebUI `TikaLoader` | A patched `loaders/main.py` |
| Document metadata | Processor `/meta` with `Accept: application/json` | A metadata sidecar |
| Upload sync images | The pinned Open WebUI image and a pinned `minio/mc` | `alpine:latest` and untagged `minio/mc` |

## Navigation for a future docs site

Map this folder to a `Community` section:

1. How to submit, status, and decisions (Stages 1–4, when those PRs are merged)
2. Operator day-to-day: Open WebUI metadata claims match the keys the pinned image reads
3. Operator day-to-day: processed objects can carry Tika metadata that was actually JSON
4. Maintainer day-to-day: the sync cron uses images this repo already pins

When a static site is adopted, move or mount these pages into the docs content root. Do not keep two editable copies.

## Related archive and planning docs

- [`../README.md`](../README.md) — suggestions hub
- [`../index.md`](../index.md) — community suggestions map
- [`../implementation-backlog.md`](../implementation-backlog.md) — trimmed execution backlog
- [`../historical-suggestions.md`](../historical-suggestions.md) — changelog ideas
- [`../website-roadmap-from-historical-suggestions.md`](../website-roadmap-from-historical-suggestions.md) — historical product direction
111 changes: 111 additions & 0 deletions suggestions/website/openwebui-tika-metadata-flag-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
---
title: Open WebUI Tika Metadata Flag Contract
description: Constructional contract to remove the unread TIKA_METADATA_EXTRACTION Compose flag and keep document metadata on the processor, without patching the pinned Open WebUI loader.
last_updated: 2026-10-26
owner: maintainers
status: proposed
theme: operations
impact: high
---

# Open WebUI Tika Metadata Flag Contract

## Problem

Open WebUI Compose tells operators that metadata extraction is switched on:

```text
CONTENT_EXTRACTION_ENGINE: "tika"
TIKA_SERVER_URL: "http://ai_tika:9998"
PDF_EXTRACT_IMAGES: "false"
TIKA_METADATA_EXTRACTION: "true"
```

The comment above the last key is "Enable Tika metadata extraction". The pinned image does not read that name.

`docker-compose.yml` pins:

```text
ghcr.io/open-webui/open-webui@sha256:7f1b0a1a50cfbac23da3b16f96bc968fd757b26dc9e54e93813d61768ea9184e
```

That image is git revision `02dc3e689ceac915a870b373318b99c029ddf603`. A search of that revision finds `CONTENT_EXTRACTION_ENGINE`, `TIKA_SERVER_URL`, and `PDF_EXTRACT_IMAGES`. It finds no `TIKA_METADATA_EXTRACTION`.

`TikaLoader.load` in `backend/open_webui/retrieval/loaders/main.py` at that revision does this:

```text
endpoint += 'tika/text'
r = requests.put(endpoint, data=data, headers=headers)
raw_metadata = r.json()
text = raw_metadata.get('X-TIKA:content', '<No text content found>').strip()
if 'Content-Type' in raw_metadata:
headers['Content-Type'] = raw_metadata['Content-Type']
return [Document(page_content=text, metadata=headers)]
```

`headers` starts empty unless `mime_type` was passed. The call site that builds `TikaLoader` passes `url`, `file_path`, and `extract_images`. It does not pass `mime_type`. The `Document` therefore keeps page text and, at most, `Content-Type`. Title, creator, and page count are in the JSON body and are dropped.

Apache Tika 3.3.1 `TikaResource` maps `PUT /tika/{handler}` with `@Produces("application/json")` onto metadata that includes `X-TIKA:content`. `/tika/text` is that JSON method. The loader is right to call it for text. The Compose flag does not change the call.

Checked on 2026-10-26 by reading Compose, `TikaLoader`, and `git grep` of revision `02dc3e689`. No container was started.

## Existing scan (reuse first)

| Asset | Role |
|---|---|
| `CONTENT_EXTRACTION_ENGINE` and `TIKA_SERVER_URL` | The engine switch this revision actually branches on |
| `TikaLoader.load` | Text extraction to keep. Do not edit it in this repo |
| `tika-minio-processor` `/meta` call | The metadata path this stack owns |
| Stage 59 PDF engine keys (open PR #85) | Which config paths the loader reads. This page does not write `rag.file.pdf_loader` |
| Stage 59 init script (same PR) | Deletes the `PyPDFLoader` search. This page does not add a metadata patch there |
| Stage 58 OCR runtime (open PR #84) | Tika image and `tika-config.xml`. Unchanged here |
| Processor meta Accept (this stage) | How `/meta` becomes a JSON object |

Out of scope:

- A fork or bind-mount of `loaders/main.py`
- A metadata database, search index, or second Tika container
- Changing `PDF_EXTRACT_IMAGES` (Stage 59 keeps it false)
- Rewriting `tika-config.xml` (Stage 58)
- Teaching Compose a new `TU_VM_TIKA_METADATA` name

## Proposal

Remove the flag that this image ignores. Keep chat-upload text on the pinned loader. Keep author, title, and similar fields on the processor, which already calls `/meta`.

### Contribution lanes

| Lane | Where | Suggested change |
|---|---|---|
| Compose | `open-webui` environment | Delete `TIKA_METADATA_EXTRACTION` and the comment that says it enables metadata |
| Docs | Operator note beside the Tika env block | Say this revision stores `X-TIKA:content` in the Open WebUI document and does not keep the rest of the Tika JSON |
| Processor | `universal_processor.py` | Metadata work stays on the `/meta` call covered by the Accept contract in this stage |
| Init script | `scripts/init-openwebui.sh` | Do not add a metadata patch. Stage 59 already removes the loader search |

### Rules

1. **Delete `TIKA_METADATA_EXTRACTION`.** Revision `02dc3e689` has no reader for it.
2. **Do not patch `TikaLoader`.** Stage 59 leaves `loaders/main.py` unmodified. A local patch would drift on the next image bump.
3. **Text stays on `/tika/text`.** That path is the JSON handler Tika 3.3.1 uses for `X-TIKA:content`.
4. **Metadata stays on the processor.** The community-owned code is `universal_processor.py`, not the upstream loader.
5. **Leave OCR and the PDF engine keys on Stages 58 and 59.**

### Suggested contributor checklist

```text
1. Confirm TIKA_METADATA_EXTRACTION is absent from open-webui environment
2. Confirm init-openwebui.sh does not write a metadata patch into loaders/main.py
3. Confirm CONTENT_EXTRACTION_ENGINE and TIKA_SERVER_URL are unchanged
4. Confirm PDF_EXTRACT_IMAGES stays false
5. Confirm a PDF upload still uses Tika for text after the flag is removed
```

### Framework reuse

| Need | Prefer | Avoid |
|---|---|---|
| PDF and office text | Pinned `TikaLoader` and `CONTENT_EXTRACTION_ENGINE` | A custom loader image |
| Document metadata | Existing processor `/meta` call | A new metadata service |
| Config honesty | Delete the unread key | A second flag with a TU-VM name |
| Upstream drift | The digest pin | A bind-mounted `main.py` |
| Intake | GitHub Issue | A forked Open WebUI |
124 changes: 124 additions & 0 deletions suggestions/website/processor-meta-accept-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
---
title: Processor Meta Accept Contract
description: Constructional contract so the document processor asks Tika /meta for JSON, and does not store that JSON in S3 user metadata until the encoded header fits.
last_updated: 2026-10-26
owner: maintainers
status: proposed
theme: tooling
impact: high
---

# Processor Meta Accept Contract

## Problem

The document processor reports "extracting metadata" and then stores no Tika metadata.

`tika-minio-processor/universal_processor.py` builds the text request with an explicit Accept header:

```text
headers = {"Content-Type": content_type, "Accept": "text/plain"}
PUT {TIKA_URL}/tika
```

The metadata request a few lines later does not:

```text
metadata_headers = {"Content-Type": content_type}
PUT {TIKA_URL}/meta
if metadata_response.status_code == 200:
metadata = metadata_response.json()
```

`Content-Type` on that call is the uploaded file type, not the response type. On this host, `requests` 2.32.5 `default_headers()` sends `Accept: */*`. Preparing a `PUT` with only `Content-Type: application/pdf` keeps that Accept value.

Apache Tika 3.3.1 `MetadataResource` is `@Path("/meta")`. The `PUT` method is:

```text
@Produces({"text/csv", "application/json"})
```

`text/csv` is listed first. The Tika 3.3.1 server README shows the unadorned call as CSV:

```text
curl -T price.xls http://localhost:9998/meta
```

`curl` and `requests` both send `Accept: */*` when the caller does not set one. A CSV body makes `response.json()` raise. The processor catches `Exception`, logs a warning, and leaves `metadata` as `{}`.

`store_processed_content` only copies metadata when the dict is non-empty:

```text
if metadata:
s3_metadata['extracted-metadata'] = quote(metadata_json, safe='')
```

An empty dict skips the header. The `.txt` object can still be written. The status line said metadata was extracted. The object has none.

The text path in this same file, and the older `tika_processor.py` comment ("without Accept header, Tika returns HTML"), already treat Accept as required. The `/meta` call is the one that skipped it.

Checked on 2026-10-26 by reading the processor, replaying `requests` header preparation, and reading Tika 3.3.1 `MetadataResource.java` plus the server README. No Tika container was started.

## Existing scan (reuse first)

| Asset | Role |
|---|---|
| `universal_processor.py` text `PUT /tika` | Already sends `Accept: text/plain` |
| `universal_processor.py` `PUT /meta` | The call that needs `Accept: application/json` |
| `tika_processor.py` comment | In-repo note that a missing Accept selects the wrong Tika body |
| `store_processed_content` | Writes `extracted-metadata` only when the dict is non-empty |
| Stage 54 metadata header (open PR #80) | Encoded user-metadata length. This page does not redo that table |
| Stage 58 OCR runtime (open PR #84) | `tika-config.xml` and the full image. Unchanged here |
| Stage 41 byte budget (open PR #67) | How many object bytes are read. Unchanged here |
| Open WebUI metadata flag (this stage) | The unread Compose key. Metadata stays in this processor |

Out of scope:

- A metadata database or a wrapper in front of Tika
- Changing the `/tika` text Accept header
- Editing `tika-config.xml` or OCR strategy headers
- A new S3 metadata schema
- Restating the Stage 54 percent-encoding length table

## Proposal

Ask `/meta` for the JSON media type it already produces. Keep the text object storable when that JSON is large.

### Contribution lanes

| Lane | Where | Suggested change |
|---|---|---|
| Request | `PUT /meta` headers | Set `Accept: application/json` next to `Content-Type` |
| Parse | Same block | Keep `response.json()` only after that header. A non-JSON 200 stays a warning and an empty dict |
| Store | `store_processed_content` | Do not attach `extracted-metadata` unless the encoded value fits the limit Stage 54 already specifies |
| Order | One PR | Ship the Accept header together with that omit-or-cap. Accept alone turns today's empty dict into a body that can fail `put_object` |

Stage 54 owns the arithmetic (the cut runs before `quote()`, and the encoded value can pass 2048 bytes). This page does not replace that arithmetic. It says the Accept fix is unsafe to ship by itself, because a parsed dict reaches the header Stage 54 already shows can reject the upload. Until that cap lands, omit `extracted-metadata` and still store the text.

### Rules

1. **Send `Accept: application/json` on `/meta`.** `Accept: */*` is the CSV example in the Tika 3.3.1 README.
2. **Leave `Accept: text/plain` on `/tika`.** That call already matches the plain-text method.
3. **Do not fail the file when metadata is CSV or oversized.** Text storage is the pipeline result. Metadata is additional.
4. **Use Stage 54's header budget.** Do not invent a second byte limit in this page.
5. **Do not add a Tika proxy.** The processor already has the HTTP client.

### Suggested contributor checklist

```text
1. Confirm the /meta headers include Accept: application/json
2. Confirm the /tika text headers still include Accept: text/plain
3. Confirm a CSV or invalid JSON body logs a warning and leaves metadata empty
4. Confirm an oversized metadata dict does not fail put_object; the text object is still stored
5. Confirm tika-config.xml is unchanged
```

### Framework reuse

| Need | Prefer | Avoid |
|---|---|---|
| Metadata media type | Tika's existing `application/json` producer | A CSV parser in TU-VM |
| Text media type | The Accept header the text call already sets | A second Tika endpoint |
| Header size | Stage 54 encoded-length rule | A new object store |
| Failure mode | Store the text, warn on metadata | Retry forever because user metadata was too big |
| Intake | GitHub Issue | A forked Tika server |
Loading
Loading