Skip to content

feat: SQLite persistence core#1358

Open
kevin-dp wants to merge 147 commits intomainfrom
kevin/persistence
Open

feat: SQLite persistence core#1358
kevin-dp wants to merge 147 commits intomainfrom
kevin/persistence

Conversation

@kevin-dp
Copy link
Contributor

@kevin-dp kevin-dp commented Mar 12, 2026

Summary

Replaces #1230 — this PR contains the core persistence packages only, split out for easier review:

  • db-sqlite-persisted-collection-core — shared SQLite adapter, persistence runtime, tx pruning, and coordinator protocol
  • db-browser-wa-sqlite-persisted-collection — browser persistence via wa-sqlite (OPFS) with multi-tab coordinator (BroadcastChannel + Web Locks)
  • db-react-native-sqlite-persisted-collection — React Native/Expo persistence via expo-sqlite
  • db — index lifecycle events, removeIndex API, and hydration support for persistence

Runtime-specific persistence layers are in follow-up PRs:

Implements #865 (comment) from #865

Test plan

  • Unit and contract tests pass for db-sqlite-persisted-collection-core
  • Browser wa-sqlite persistence tests pass (single-tab + multi-tab coordinator)
  • React Native/Expo persistence and lifecycle tests pass
  • db package index lifecycle tests pass
  • pnpm test:pr passes

🤖 Generated with Claude Code

cursoragent and others added 30 commits February 10, 2026 09:28
cursoragent and others added 20 commits February 12, 2026 13:52
…cate tx skipping

The PersistedCollectionRuntime never restored its stream position from
the database on startup, always beginning at localTerm=1, localSeq=0.
After a page reload, the first new mutation would collide with a
previously applied transaction (term=1, seq=1), causing the SQLite
adapter's applyCommittedTx to silently skip it as a duplicate.

Add getStreamPosition to PersistenceAdapter (optional) and implement it
in SQLiteCorePersistenceAdapter. Call it from startInternal() so that
observeStreamPosition seeds the local counters before any mutations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Include full row values in the tx:committed message so receiving tabs
can apply changes directly without a SQLite round-trip via loadRowsByKeys.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ti-tab support

Web Locks for per-collection leadership election, BroadcastChannel for
cross-tab RPC transport, DB writer lock for SQLite write serialization,
envelope dedup for exactly-once mutations, and leader heartbeats.
Includes 15 unit tests with Web Locks/BroadcastChannel mocks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, `state.isLeader = true` was set before the setup code that
calls `getStreamPosition()`. If `getStreamPosition` threw (e.g. due to
a UNIQUE constraint violation from React StrictMode double-mounting),
`isLeader` remained permanently stuck at `true` because the `finally`
block that resets it was inside an inner try/finally that was never
entered.

Fix: Wrap the entire lock callback body in a single try/finally. Set
`state.isLeader = true` only after successful setup (stream position
restore and term increment). The finally block always runs and resets
`isLeader = false` + cleans up the heartbeat timer.

Also refactors the coordinator to support lazy adapter wiring via
`setAdapter()`, allowing `createBrowserWASQLitePersistence` to inject
the adapter after construction. This enables the demo to construct the
coordinator without requiring the adapter upfront.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nt seq collisions

The leader tab had two mutation paths: a "direct" path (write to SQLite
and broadcast) and an RPC path (through the coordinator). Previously,
only follower tabs used the RPC path — the leader bypassed the
coordinator and wrote directly.

This caused a seq collision: the leader's direct writes incremented the
runtime's `localSeq` but left the coordinator's `state.latestSeq` at 0.
When a follower later sent an RPC, the coordinator assigned seq starting
from 1 again, producing duplicate seq numbers. The leader then skipped
these "already-seen" tx:committed messages, causing follower mutations
to silently disappear.

Fix: Always route through `requestApplyLocalMutations` when available,
regardless of leader/follower status. This keeps the coordinator's seq
counter in sync with all writes.

Also removes `requestApplyLocalMutations` from `SingleProcessCoordinator`
— it was a stub that returned success without persisting, which would
break now that the leader uses this path. Single-process mode correctly
falls back to the direct path since it has no multi-tab coordination.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mmitted messages

The leader tab's `onCoordinatorMessage` handler skipped ALL messages
where `senderId` matched the coordinator's own node ID. But when the
coordinator processes a follower's RPC in `handleApplyLocalMutations`,
it delivers the resulting `tx:committed` to local subscribers using the
coordinator's own `senderId`. This caused the leader's runtime to
silently ignore follower mutations — they were written to SQLite but
never applied to the leader's in-memory collection.

Fix: Allow `tx:committed` messages from self to pass through the filter.
The seq dedup logic in `processCommittedTxUnsafe` already prevents
double-processing: when the leader's own mutations go through the
coordinator, `observeStreamPosition` is called with the response's
term/seq before the local `tx:committed` delivery runs under the mutex,
so the duplicate is detected via `txCommitted.seq <= this.latestSeq`.
Other message types (heartbeats, resets) from self are still skipped.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…riteMessage

When queryCollectionOptions detects a server-side deletion, it sends
{ type: 'delete', value: oldItem } through the sync. The persistence
layer only checked for 'key' in message to detect deletes, causing
value-based deletes to be misclassified as updates. Also use optional
chaining for process.versions in React Native where process exists but
versions may not.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ss-window sync

Add ElectronCollectionCoordinator using BroadcastChannel + Web Locks for
leader election and cross-window coordination in Electron renderer windows.
Wire coordinator into renderer persistence via setAdapter(), add
getStreamPosition to the IPC protocol, and export from package index.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… follow-up branches

Remove db-electron-sqlite-persisted-collection, db-node-sqlite-persisted-collection,
and db-cloudflare-do-sqlite-persisted-collection from this branch so it contains only
the core persistence packages (db, db-sqlite-persisted-collection-core,
db-browser-wa-sqlite-persisted-collection, db-react-native-sqlite-persisted-collection).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@changeset-bot
Copy link

changeset-bot bot commented Mar 12, 2026

⚠️ No Changeset found

Latest commit: 9ce6ae0

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@pkg-pr-new
Copy link

pkg-pr-new bot commented Mar 12, 2026

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/angular-db@1358

@tanstack/db

npm i https://pkg.pr.new/TanStack/db/@tanstack/db@1358

@tanstack/db-browser-wa-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-browser-wa-sqlite-persisted-collection@1358

@tanstack/db-ivm

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-ivm@1358

@tanstack/db-react-native-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-react-native-sqlite-persisted-collection@1358

@tanstack/db-sqlite-persisted-collection-core

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-sqlite-persisted-collection-core@1358

@tanstack/electric-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/electric-db-collection@1358

@tanstack/offline-transactions

npm i https://pkg.pr.new/TanStack/db/@tanstack/offline-transactions@1358

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/powersync-db-collection@1358

@tanstack/query-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/query-db-collection@1358

@tanstack/react-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/react-db@1358

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/rxdb-db-collection@1358

@tanstack/solid-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/solid-db@1358

@tanstack/svelte-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/svelte-db@1358

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/trailbase-db-collection@1358

@tanstack/vue-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/vue-db@1358

commit: 4f50320

@kevin-dp kevin-dp mentioned this pull request Mar 12, 2026
4 tasks
@github-actions
Copy link
Contributor

github-actions bot commented Mar 12, 2026

Size Change: +1.93 kB (+2.07%)

Total Size: 95.1 kB

Filename Size Change
./packages/db/dist/esm/collection/events.js 434 B +46 B (+11.86%) ⚠️
./packages/db/dist/esm/collection/index.js 3.56 kB +236 B (+7.1%) 🔍
./packages/db/dist/esm/collection/indexes.js 2.35 kB +1.25 kB (+113.57%) 🆘
./packages/db/dist/esm/index.js 2.8 kB +74 B (+2.72%)
./packages/db/dist/esm/indexes/lazy-index.js 1.24 kB +135 B (+12.24%) ⚠️
./packages/db/dist/esm/query/compiler/evaluators.js 1.62 kB +189 B (+13.23%) ⚠️
ℹ️ View Unchanged
Filename Size
./packages/db/dist/esm/collection/change-events.js 1.39 kB
./packages/db/dist/esm/collection/changes.js 1.22 kB
./packages/db/dist/esm/collection/lifecycle.js 1.75 kB
./packages/db/dist/esm/collection/mutations.js 2.34 kB
./packages/db/dist/esm/collection/state.js 3.49 kB
./packages/db/dist/esm/collection/subscription.js 3.71 kB
./packages/db/dist/esm/collection/sync.js 2.41 kB
./packages/db/dist/esm/deferred.js 207 B
./packages/db/dist/esm/errors.js 4.83 kB
./packages/db/dist/esm/event-emitter.js 748 B
./packages/db/dist/esm/indexes/auto-index.js 742 B
./packages/db/dist/esm/indexes/base-index.js 766 B
./packages/db/dist/esm/indexes/btree-index.js 2.17 kB
./packages/db/dist/esm/indexes/reverse-index.js 538 B
./packages/db/dist/esm/local-only.js 808 B
./packages/db/dist/esm/local-storage.js 2.1 kB
./packages/db/dist/esm/optimistic-action.js 359 B
./packages/db/dist/esm/paced-mutations.js 496 B
./packages/db/dist/esm/proxy.js 3.75 kB
./packages/db/dist/esm/query/builder/functions.js 733 B
./packages/db/dist/esm/query/builder/index.js 4.1 kB
./packages/db/dist/esm/query/builder/ref-proxy.js 1.05 kB
./packages/db/dist/esm/query/compiler/expressions.js 430 B
./packages/db/dist/esm/query/compiler/group-by.js 2.23 kB
./packages/db/dist/esm/query/compiler/index.js 2.05 kB
./packages/db/dist/esm/query/compiler/joins.js 2.11 kB
./packages/db/dist/esm/query/compiler/order-by.js 1.45 kB
./packages/db/dist/esm/query/compiler/select.js 1.09 kB
./packages/db/dist/esm/query/expression-helpers.js 1.43 kB
./packages/db/dist/esm/query/ir.js 673 B
./packages/db/dist/esm/query/live-query-collection.js 360 B
./packages/db/dist/esm/query/live/collection-config-builder.js 5.55 kB
./packages/db/dist/esm/query/live/collection-registry.js 264 B
./packages/db/dist/esm/query/live/collection-subscriber.js 2.42 kB
./packages/db/dist/esm/query/live/internal.js 145 B
./packages/db/dist/esm/query/optimizer.js 2.62 kB
./packages/db/dist/esm/query/predicate-utils.js 2.97 kB
./packages/db/dist/esm/query/query-once.js 359 B
./packages/db/dist/esm/query/subset-dedupe.js 927 B
./packages/db/dist/esm/scheduler.js 1.3 kB
./packages/db/dist/esm/SortedMap.js 1.3 kB
./packages/db/dist/esm/strategies/debounceStrategy.js 247 B
./packages/db/dist/esm/strategies/queueStrategy.js 428 B
./packages/db/dist/esm/strategies/throttleStrategy.js 246 B
./packages/db/dist/esm/transactions.js 2.9 kB
./packages/db/dist/esm/utils.js 924 B
./packages/db/dist/esm/utils/browser-polyfills.js 304 B
./packages/db/dist/esm/utils/btree.js 5.61 kB
./packages/db/dist/esm/utils/comparison.js 952 B
./packages/db/dist/esm/utils/cursor.js 457 B
./packages/db/dist/esm/utils/index-optimization.js 1.51 kB
./packages/db/dist/esm/utils/type-guards.js 157 B

compressed-size-action::db-package-size

@github-actions
Copy link
Contributor

github-actions bot commented Mar 12, 2026

Size Change: 0 B

Total Size: 3.85 kB

ℹ️ View Unchanged
Filename Size
./packages/react-db/dist/esm/index.js 225 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.32 kB
./packages/react-db/dist/esm/useLiveQuery.js 1.34 kB
./packages/react-db/dist/esm/useLiveSuspenseQuery.js 559 B
./packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@kevin-dp kevin-dp force-pushed the kevin/persistence branch 2 times, most recently from 38ea25f to 9ce6ae0 Compare March 12, 2026 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants