migration: batched postgres inserts and validation

[djkazic]

Background

Migrating an lnd bolt database (for example channel.db) into postgres is very slow. We measured about 4.5 hours for ~15M keys. Profiling showed the process was almost entirely idle waiting on the network (CPU around 6 percent), not busy doing work.

The cause is that the migration talks to postgres one row at a time, and it does so twice. The migrate phase issues one INSERT per key, and the verify phase reads the target back one query at a time. Each of those is a separate round trip on a single connection, so total time is roughly round_trips * round_trip_latency, where round_trips scales with the number of keys and buckets. It scales linearly with database size and with network latency, which is why co-locating the DB helps a little but does not fix the underlying problem.

This PR reduces the number of round trips in both the migrate and verify phases.

Changes

1. Batched write path (migratekvdb/bulk.go, new MigrateBulkSQL)

Adds a postgres write path that inserts leaf rows in batches (1000 rows per statement by default) using a raw pgx transaction, instead of one Put per key through kvdb. It walks the source depth first, creates bucket rows as it goes so the parent_id foreign key is always satisfied, and streams leaf key/values into multi-row INSERTs. The whole migration runs in a single transaction. It does not modify or fork the kvdb dependency.

This path is for fresh migrations only. The existing Migrate still handles resume.

Because the SQL target and the bolt meta database cannot share a transaction, completion is not perfectly atomic across the two. To make an interrupted run recoverable instead of leaving it stuck, MigrateBulkSQL writes an in-progress marker to the meta DB before it starts writing data, and clears it after the SQL commit and the finished-state write. On a re-run, if the marker is still set, it truncates the target table and starts over, which is safe because the marker means the previous attempt owned the table. This adds two small meta writes and no measurable overhead on the hot path.

Batch size is clamped to 65535 / 3 rows so a caller cannot exceed the postgres bind-parameter limit.

2. Streaming verification and progress logging (migratekvdb/migration.go)

Two changes to the existing verify path:

• The fresh-verification path now reads each target bucket with a single ForAll query instead of a per-key cursor, and compares it in lockstep against the source cursor. Nested buckets are collected during the streamed read and recursed into afterward, since ForAll does not allow issuing queries from inside its callback. This applies to fresh verifications against a backend that implements kvdb.ExtendedRBucket (postgres and sqlite). Resume and bolt targets keep using the original per-key cursor path, which is unchanged apart from a rename.
• Progress is now logged once per bucket (time-gated), in both migrate and verify. Previously the log fired only every N keys within a single bucket, so on a database with many small buckets it went silent for a long time and the process looked hung even though it was making progress.

3. Batched verification (migratekvdb/verify_batched.go, new VerifyBatchedSQL)

The single-ForAll-per-bucket verify above still issues a few queries per bucket, so on a database with a very large number of buckets (a real channel.db has on the order of a million) the verify phase is dominated by per-bucket round trips.

VerifyBatchedSQL walks the tree breadth first and fetches the direct children of up to batchSize buckets in one query (WHERE parent_id = ANY(...)), instead of one query per bucket. That reduces the query count from roughly bucket_count to roughly bucket_count / batchSize. The batch result is streamed (ORDER BY parent_id, key) and compared against each parent's source cursor inline, so a batch that happens to include a very wide bucket does not buffer that bucket's rows in memory. Every query is a plain index scan on (parent_id, key), so there is no server-side sort and no work_mem pressure, and client memory is bounded by one BFS level. The parent_id array is passed as a single bound parameter so the SQL text is constant and pgx reuses one prepared statement. Verification error messages print keys and paths only, never values.

This is a fresh-only path for a postgres target reached through a raw *sql.DB. Bolt targets and resume continue to use the hierarchical path.

4. CLI wiring (cmd_migrate_db.go)

A new --bulk-writes flag enables the fast postgres path for a fresh migration: MigrateBulkSQL for the migrate phase and VerifyBatchedSQL for the verify phase. It is opt-in and postgres only. Any other destination logs a warning and falls back to the standard Migrate and VerifyMigration path. A prior interrupted bulk run self-recovers as described above; a prior interrupted non-bulk run that already wrote data will surface a "target not empty" error rather than silently proceeding.

Results

We ran the actual lndinit migrate-db --bulk-writes command end to end against a copy of a real mainnet data directory.

• Full command, all mandatory databases, completed successfully in 8m12s.
• channeldb verify: 21m50s with the single-ForAll-per-bucket path, 1m22s with the batched path (about 16x), byte for byte identical both times.

The dominant remaining cost is bucket creation during the migrate phase, which still issues one INSERT ... RETURNING id per bucket. Batching that is a sensible follow-up but is out of scope here.

One behavior change worth calling out: a fresh sqlite or postgres verify no longer checkpoints mid-bucket, so if it is interrupted while verifying a large top-level bucket, the resume re-verifies that bucket from the start. Resume correctness is preserved, only repeated work changes. Bolt was never on this path.

Testing

New: an edge-case source builder (empty values, binary keys, prefix-overlapping keys, explicit sequences, an empty bucket, three-level nesting with leaves and sub-buckets interleaved) plus an independent byte-for-byte comparator.

A sqlite test verifies the empty-value handling (regression test for the nil-versus-empty case, where sqlite decodes an empty stored value as nil while bolt and postgres return an empty slice). A postgres test covers bulk migrate correctness, batch-size clamping, refusal on a non-empty target, crash recovery via the in-progress marker, batched verification (including a small batch size to exercise batch and level boundaries), and negative cases confirming both verify paths detect a missing row, a mutated value, and an extra row. An end-to-end test runs the full command with --bulk-writes. All pass under the base, kvdb_sqlite, and kvdb_postgres tags.

[ziggie1984]

Really nice optimization, thank you.

Lets split this up in 3 PRs:

1. lnd PR

   Add the SQL-KV migration primitives in lnd, near kvdb/sqlbase:

   • empty target check
   • target reset for recovery
   • bucket insert with returned row ID
   • leaf bulk insert
   • batched verification reads
   • explicit bucket/leaf marker

This keeps SQL schema knowledge inside lnd instead of lndinit. It also fits the longer-term plan to move migration code into lnd, so this becomes reusable migration infrastructure rather than a lndinit-only hot fix.

2. lndinit PR

Keep lndinit as orchestration only:
- add --bulk-writes
- call the new lnd migration API
- keep the BulkInProgress recovery marker
- add wiring/tests/docs

3. Optional follow-up

Treat the ForAll verifier optimization as separate. It helps the legacy/non-bulk verification path, but the new bulk verifier already uses batched reads, so it does not need to be part of this split.

[ziggie1984]

replaced with #94