feat(storage): make SQLite a profile the app can run on

[Yuge Zhang (ultmaster)]

Note

Scope grew past the original preview. Phase 5 no longer stops at "an isolated adapter that proves the ports"; SQLite is now a profile the application can actually run on, with no Workspace folder and no Space directories.

What this is

HUABU_STRUCTURED_BACKEND=sqlite is a selectable profile. Every record is a row in one database file; a Space's bytes are ordinary files, because the blob axis is always a file system:

<HUABU_DATA_DIR>/storage/
  sqlite/huabu.sqlite                       every record — override with HUABU_SQLITE_PATH
  disk/
    workspaces.json                         Disk structured store's Workspace registry
    blobs/<workspaceId>/<canvasId>/         every byte   — override with HUABU_BLOB_ROOT
      {skill.md, .artifacts/, .memory/, .upload/}

Each directory under storage/ is named for the backend that owns it, and one file per backend decides its layout (backends/disk/data-dir.ts, backends/sqlite/database.ts); the composition root asks and builds no path of its own, which a module-boundary census pins. storage/disk/ has two owners, so they get separate subtrees — the registry is never inside blobs/, because the blob store deletes whole directories and the registry is not its to delete.

That split is the one design correction this branch makes against its own first draft: an earlier revision had HUABU_BLOB_BACKEND=sqlite keep bytes in a blobs table, with a cross-axis rule forcing the two together. Bytes are files — a local directory now, an object store later — so no structured backend is asked to hold them, the two axes share nothing, and any structured backend pairs with any blob backend. sqlite records beside disk bytes is an ordinary profile, not a special case.

Verified by hand against a running Server on structured=sqlite blobs=disk: create a Space, add nodes through the executor, upload an image and fetch the bytes back, start an agent thread, create and activate a second Workspace by name, restart the process, delete a Space and watch its byte directory go with it.

Supported

Area	On SQLite
Workspaces	Rows. Created and activated on first start; created by name, listed, renamed, activated, and forgotten by id. One connection serves all of them; switching re-scopes rather than reconnects.
Spaces	Create, list, rename, delete, World bootstrap and protection, title de-duplication identical to Disk. Each Space belongs to exactly one Workspace, by foreign key.
Nodes	Read, readMany, list, stream, put with revision CAS, strict/non-strict label allocation, delete
Ordered writes	Node mutations + record replacement + delta journal in one immediate transaction; same-baseline writers have one winner
Events, changes, Tasks/Runs	Full parity, including atomic Run completion
Blob bytes	artifacts, guide, memory, uploads — the same Disk adapter and the same area layout, under a Server-owned root instead of a Space folder
Extension namespaces	Memory bookkeeping, debug prompt log, agent conversations — all cascade away with the Space
Agent chat	Threads, the Tier-1 event log and Tier-2 folded turns, surviving a restart

Not supported — declared, logged at startup, refused in the same words

Each row is an entry in storage/capabilities.ts, printed at boot for the selected profile and reused verbatim by the refusal at its own call site.

The matrix is keyed on the profile, both axes. Most entries need a Space or Workspace to be a real directory (a structured-backend property); four need the Space's bytes to be in that directory too, which is the blob backend's. Omitting an axis means every backend on it serves the feature — so a blob-agnostic row never needs editing when a blob backend lands, and the rows that do depend on it are exactly the ones forced to decide.

On the hybrid profile the answer is unchanged: a real file system for the bytes hands nothing back, because every row still needs the record and node documents to be files. detached-blobs.test.ts pins that, and capabilities.test.ts runs the whole matrix against sqlite/disk plus the disk/azure pairing the two-axis keying exists for.

Capability	What is lost	Why not emulated
workspace-directory	Picking / creating / revealing a Workspace folder	A Workspace is a row; the Server opens its own and the client is told canChangeWorkspace: false. Creating one by name is supported — it is only the folder that is gone.
space-bundle-export / space-bundle-import	.huabu.zip round-trip	The bundle is the Space directory, archived
reveal-space-folder	"Show me this in Finder"	What a user means by "this" is the Space — its record and node documents — and those are rows. Its byte directory is not the Space
builtin-file-tools	Agent read / write / glob / grep	The documents they edit are the node sidecars under nodes/, which are rows; the byte areas hold artifacts, not those
space-file-plane	RFS, the HTTP file plane external agents mount	New row. The old builtin-file-tools rationale named RFS as the fallback; RFS needs the same directory, so that claim was wrong
external-note-discovery	Adopting .md dropped into a Space from outside	A database has no such arrival path
workspace-user-memory	setting/user.md	A file the user edits at the root of a Workspace they chose. The blob port has no Workspace-level scope, so it has none to live in. A Space's own memory body is unaffected — it is a blob
workspace-user-skills	setting/skills/<id>/SKILL.md	Same arrival path as external notes; bundled and Agent Team skills are unaffected

Two further limits are properties of the backend rather than refusals, and a row nothing can refuse is not a capability:

• Windows directory-handle coordination. It exists so renaming a Space directory can succeed against a live fs.watch handle. A Space that is a row is never filed under its title and nothing watches it, so nothing ever asks. It was a capability row; printing "unavailable" told an operator they had lost something when the profile simply does not have the problem.
• Multi-process access. One process, one connection. WAL plus busy_timeout make a second reader survivable; no multi-process deletion fence or distributed transaction is promised.

Two rules keep the call sites honest, and module-boundaries.test.ts checks the first: a refusal asks the matrix (storageServes(id) on the composition root) rather than re-deriving the requirement from diskTree; a degradation does not, because it is not making the profile's promise.

Postgres and Azure Blob still have no adapter.

Two defects found by testing the preview against Disk

1. SQLite rejected a record Disk accepts. Disk persists through JSON.stringify, which drops an undefined own property; the SQLite encoder called it a non-JSON value and failed the write. An optional field spread onto a node is all it takes — exactly the silent divergence §13 warns about. The encoder now follows JSON.stringify; cycles, non-finite numbers, and non-plain objects still reject.
2. stream() did not stream. It materialized the whole collection, then delivered — the opposite of the latency shape the port describes, and an aborted scan had already paid for every row. It reads off a cursor now, and readMany went from one statement per id to one per chunk.

Also added while making the profile selectable: WAL, synchronous = NORMAL, and a bounded busy_timeout, none of which the isolated preview needed.

Design notes worth a reviewer's attention

• blobs=disk names a medium, not a directory. Bytes are local files; where is composition's, under one rule — a Space's bytes live with the Space — and the Space has two possible homes. On Disk records it is a folder, so the bytes stay inside it: that is not just compatibility with existing Workspaces, it is what space-bundle-export, space-bundle-import, reveal-space-folder and builtin-file-tools are made of, since each of them is the Space folder being complete. On rows there is no folder to be inside, so the adapter gets storage/disk/blobs/<workspaceId>/<canvasId>/. The adapter's root argument is required rather than defaulted, so that cross-axis choice can't be made by a caller that forgot to pass one.
• The capability matrix is keyed on the profile. A blob backend that cannot co-locate — an object store — would put bytes outside the Space folder even on Disk records, so bundle export/import, the file tools and RFS name the blob axis too. Every refusal asks storageServes(id) instead of re-deriving the requirement from diskTree, and a boundary census fails if a declared row has no refusal at all.
• Each backend owns its area of the data directory, in one file. workspaceRegistryPath moved out of the Disk workspace repository and sqliteDatabasePath out of the composition root; both now sit with their backend. That is what lets the Disk structured store's registry and the Disk blob store's byte roots share storage/disk/ without either being able to grow into the other, and detached-blobs.test.ts asserts the separation as the path fact it is.
• Scope binding is getWorkspaceKey(), not the workspace path. A Workspace that is a row has no path to compare; the key means the same thing on both kinds and the Disk behaviour is unchanged (there it is the resolved path).
• The delete saga removes the detached Space directory. Sweeping the areas is the blob port's contract; where the record is a row, no structured delete will ever remove the directory those areas sat under, so composition does it.
• createNamedWorkspace() is on the composition root, not the port. The counterpart to adoptWorkspaceDirectory: Disk could only serve "create by name" by inventing a folder the user never picked, and the port deliberately says nothing about where a Workspace is. workspaceCreateSchema.path becomes optional so the Server can say which form its backend requires.
• remove() on the Workspace repository is a forget, not a delete. The port says "without deleting any Workspace-owned data", which Disk honours for free because the folder outlives the registry entry. A database says it with a forgotten_at column.
• Space.sqliteTree is a second diskTree-shaped member. Agenetes's three storage ports are synchronous and extension() is not; rather than have that owner keep a cache warmed from an unrelated code path, the composition root exposes the synchronous form, typed by its absence. It has its own census in module-boundaries.test.ts with exactly one production consumer — a second one would mean the justification had drifted.
• World identity moved onto the composition root. isWorldCanvasId used to scan the Disk directory index; the World portal rules now take the live Space ids as an argument, so the policy is pure and the census of Disk-layout consumers shrank by two.

Scope

• 80 files, +7,846 / −594 against current main
• Confined to apps/server/src/modules, one optional field in packages/shared's workspace-create schema, docs/architecture/canvas-storage.md, and docs/proposals/multi-backend-storage.md
• No UI changes, no Disk-to-SQLite migration or import, no Postgres or Azure adapter
• Disk remains the default profile; nothing changes for a deployment that does not set HUABU_STRUCTURED_BACKEND

Note

The web client's Workspace UI is still folder-shaped, so on this profile it hides the picker (canChangeWorkspace: false) and offers no way to add a Workspace. Multi-Workspace support here is complete at the storage and HTTP layers; surfacing it is a separate UI change.

Proof

• The reusable contracts — structured store, Space repository, nodes, ordered write, logs, Tasks, extension substrate, and Workspace repository — run against Disk and real temporary SQLite files. The blob contract runs once, against the one blob adapter there is.
• PRODUCT_STORAGE_PROFILES gains sqlite/disk, so the §12.8 product-boundary suite runs against it unchanged — that was the point of writing it that way — plus a new case that everything survives a restart.
• What is about placement gets its own small suite (detached-blobs.test.ts): bytes land as real files under the Workspace-scoped root, one Workspace's root is not another's, deleting a Space leaves no directory behind, and the Workspace registry sits outside the blob root so no byte sweep can reach it.
• SQLite integration tests add WAL/foreign-key pragmas read on a second connection, JSON.stringify encoding parity, incremental streaming with early abort, batched readMany, Workspace scoping and handle invalidation across a switch, and forget-without-delete.
• The Agenetes conversation stores have their own suite against a mounted profile: round-trip, isolation, restart, and destruction with the Space.
• Full monorepo lint, format:check, typecheck, and test are green.

Docs

docs/proposals/multi-backend-storage.md §12.9 is rewritten around the selectable profile, its byte root, and its capability table; docs/architecture/canvas-storage.md gains the SQLite layout beside the Disk one.

🤖 Generated with Claude Code

https://claude.ai/code/session_017Howod9Qp6y2sc4jPJY7Zw

[Yuge Zhang (ultmaster)]

Pushed 30972f41 and bf505369 on top of 55699138. The description above is rewritten to match, because the scope changed rather than grew incrementally.

What moved. Phase 5 used to end at an isolated adapter that proved the ports and could not be selected. It is now a profile the application runs on: HUABU_STRUCTURED_BACKEND=sqlite HUABU_BLOB_BACKEND=sqlite, with no Workspace folder and no Space directories — one database file under <data dir>/storage/sqlite/.

That required a Workspace to become a row, a SQLite BlobStore, the Server to stop assuming a Workspace is a place, World identity to move onto the composition root, and SQLite-backed Agenetes conversation stores so agent chat survives a restart.

Two adapter defects, both found by running the preview against Disk and comparing:

1. SQLite rejected a record Disk accepts. Disk persists through JSON.stringify, which drops an undefined own property; the encoder called it a non-JSON value and failed the whole write. An optional field spread onto a node is all it takes — §13's silent-divergence risk in its most ordinary form. The encoder follows JSON.stringify now; cycles, non-finite numbers, and non-plain objects still reject.
2. stream() materialized the whole collection before delivering the first node, so an aborted scan had already paid for every row. It reads off a cursor now, and readMany went from one statement per id to one per chunk.

What the profile does not do is the half worth reviewing: nine capability rows in storage/capabilities.ts, printed at boot and reused verbatim by the refusal at each call site. space-file-plane is new — the old builtin-file-tools rationale named RFS as the fallback, and RFS needs the same directory, so that claim was wrong. The table is in the description.

Verification. pnpm run check is green on bf505369 (server 1354 passed / 21 skipped / 2 todo; web 1149; shared 379; lint, format, typecheck, i18n parity, Agent Team skills, and license headers all pass). PRODUCT_STORAGE_PROFILES gains sqlite/sqlite, so the product-boundary suite runs against the new profile unchanged, and the blob-store and Workspace-repository contracts now run on both backends.

Known gap, deliberate. Creating additional Workspaces by name has no UI on SQLite — the client sees canChangeWorkspace: false and one auto-created Workspace. The API supports list / activate / rename / forget by id; only folder-shaped creation is refused. A name-based create flow would have meant changing apps/web, which is outside this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B66r5euwUnNTmeNYjob1n3

[Yuge Zhang (ultmaster)]

Pushed a revision: blobs are always a file system, and the capability matrix is keyed on the profile

Seven commits (00cf9860..dee49199). The description is rewritten to match. Summary of what changed and why, for anyone who reviewed the earlier tree.

1. There is no SQLite blob store any more

HUABU_BLOB_BACKEND=sqlite kept every artifact, upload, guide document and memory body in a blobs table, and a cross-axis rule then forced structured=sqlite to go with it. That was the wrong shape. Bytes are files — a local directory now, an object store later — so no structured backend is asked to hold them, the two axes share nothing, and any implemented pairing is a valid deployment.

BlobBackendKind is 'disk'; RequestedBlobKind is 'disk' | 'azure'; the cross-axis rule is gone. sqlite records beside disk bytes is now the profile, and it is what PRODUCT_STORAGE_PROFILES runs the §12.8 suite against.

2. Where the bytes go, and who decides

The Disk blob adapter takes its Space root as a (required) argument. Composition chooses it, under one rule — a Space's bytes live with the Space — and a Space has two possible homes:

<HUABU_DATA_DIR>/storage/
  sqlite/huabu.sqlite                      every record       HUABU_SQLITE_PATH
  disk/
    workspaces.json                        Disk structured store — Workspace registry
    blobs/<workspaceId>/<canvasId>/        every byte         HUABU_BLOB_ROOT
      skill.md  .artifacts/  .memory/space.md  .upload/

On Disk records the Space is a folder, so bytes stay inside it — unchanged for every existing Workspace, and load-bearing: space-bundle-export is that folder archived, import unzips into it, RFS projects it, the file tools sandbox on it. Relocating artifacts would hollow out all four while each still reported as available.

Each directory under storage/ is named for the backend that owns it, and one file per backend decides its layout (backends/disk/data-dir.ts, backends/sqlite/database.ts). The composition root builds no backend path; a module-boundary census pins that. storage/disk/ has two owners, so the registry is never inside blobs/ — the blob store deletes whole directories and the registry is not its to delete.

3. Multiple Workspaces on SQLite were half-built

The schema, repository and activation all handled many, but POST /api/workspaces returned 409 because creating one meant adopting a folder. It now creates one from a name where there is no folder to adopt (createNamedWorkspace, the counterpart to adoptWorkspaceDirectory). workspaceCreateSchema.path becomes optional so the Server can say which form its backend requires.

Note

The web client's Workspace UI is still folder-shaped, so on this profile it hides the picker and offers no way to add a Workspace. Multi-Workspace support is complete at the storage and HTTP layers; surfacing it is a separate UI change.

4. The capability matrix is keyed on the profile, and gated on

Previously keyed on the structured backend alone. That stops being defensible the moment a second blob backend exists: disk records with an object store would keep every "needs a Space directory" row available, and bundle export would archive a Space folder its artifacts had never been written to — reporting success while producing an incomplete bundle.

Each row now states a requirement with one clause per axis. Every clause present must hold (and across axes), any listed backend satisfies its own clause (or within one), and an absent clause requires nothing. That last default is the design: a row that does not touch a Space's bytes must not need editing when a blob backend lands, and the rows that do are exactly the ones forced to decide then.

Capability	requires
space-bundle-export, space-bundle-import, builtin-file-tools, space-file-plane	{ structured: ['disk'], blobs: ['disk'] }
reveal-space-folder, external-note-discovery, workspace-directory, workspace-user-memory, workspace-user-skills	{ structured: ['disk'] }

Defects this review turned up, all fixed:

• Gates re-derived the requirement. Export, import, reveal, the file tools and the external-note claim all inferred availability from diskTree !== null — a second copy of the rule, and how a row could grow a blob-axis requirement its call site never learned about. Every refusal asks storageServes(id) now; diskTree is left to supply the path it was always for.
• RFS asked the wrong profile. hasStorageCapability(parseStorageProfile(), …) read the environment rather than the profile storage was opened with. storageServes binds the active profile, and hasStorageCapability left the barrel so no call site can pick one again.
• One row could never be refused. space-directory-handle-coordination had no consumer anywhere — a Space with no directory registers no handle owner, so nothing asks. It was printed at boot as something an operator had lost, when the profile simply does not have the problem. Removed; recorded as a backend property. module-boundaries.test.ts now fails if any declared row has no refusal outside storage/.
• The startup line named one axis — "unavailable on the sqlite structured backend" for rows that may be unavailable for either. It names the profile now.
• reveal-space-folder's summary was wrong. It claimed "Reveal a Space in the OS file manager"; the route opens <space>/nodes/ so a user can settle a duplicate-markdown collision by hand. Off Disk a node is a row — no folder of documents, and the collision cannot arise, because label uniqueness is a constraint rather than a filename. That is also why the row has no blob clause.

Proof

• Product-boundary suite runs against sqlite/disk unchanged, plus restart survival.
• detached-blobs.test.ts covers what is about placement: bytes land as real files under the Workspace-scoped root, one Workspace's root is not another's, deleting a Space leaves no directory behind, the Workspace registry sits outside the blob root, and — the fact all ten gates rest on — bytes exist on disk while Space.diskTree is still null.
• capabilities.test.ts asserts disk/azure: bundle export/import, file tools and RFS unavailable; reveal, note discovery and workspace-directory still available. No adapter serves that pairing and validateStorageProfile refuses it, which is exactly why the matrix has to answer it correctly first.
• Verified by hand on a running Server at structured=sqlite blobs=disk: create a Space, upload and fetch an artifact, create and activate a second Workspace by name, confirm the two byte roots are separate, delete a Space and watch its directory go with it, and see /export, /reveal-nodes and /import refuse in the matrix's own words.
• Full monorepo lint, format:check, typecheck, test green.

Still open, deliberately

The Agenetes conversation stores (agent/agenetes/sqlite-stores.ts, ~560 lines with its dispatcher) remain the largest single block of new implementation, along with the Space.sqliteTree synchronous escape hatch they need. They could be deleted by pointing the existing File* stores at the Space's byte directory — but that bets the blob backend is always a local file system, and Azure Blob is the settled next one. Happy to take that trade if reviewers prefer it.

🤖 Generated with Claude Code

https://claude.ai/code/session_017Howod9Qp6y2sc4jPJY7Zw