test(python): add merge_insert benchmarks to ci_benchmarks

[wjones127]

merge_insert has two execution paths — a legacy indexed-scan path and a newer DataFusion path — and the ci_benchmarks suite measured neither. There was no way to see how they compare, and no regression signal for either. The one existing merge_insert benchmark lives in python/benchmarks/ (not run in CI) and covers a single 1K-row update.

This PR adds a merge_insert benchmark module to ci_benchmarks, plus the datasets it needs. Every benchmark that can run on both paths is parametrized on use_index, which is the only knob that selects between them today: use_index(True) on an indexed key takes the legacy path, use_index(False) disables the index gate and takes the DataFusion path. That gives a side-by-side number for each shape rather than a single blended one.

Coverage, in groups:

• Cost model — source/target ratio sweep (1K/10K/100K rows against 10M), across three key columns that separate key type (int64 vs string) from key distribution (clustered vs random); the same sweep with a cold index cache; all-new-keys ingest; and a no-index baseline.
• Write path — partial-column updates over a row-fraction sweep and a projection sweep, against a wide target that carries both many narrow scalar columns and one vector column. Those two exert different pressures: field count and buffer scheduling versus raw byte volume. write_bytes is the metric of interest.
• Clause shapes — insert-if-not-exists, update-only, delete-by-source, conditional update, and composite keys (fully and partially indexed).
• Target layouts — deletion file on every fragment, rows appended after the index was built, and many small fragments.
• Edges — a streaming (one-shot) source, and a source the same size as the target.

Timing benchmarks report through pytest-benchmark; the cost-model and write-path groups also run under the io_memory_benchmark fixture so IO counts, write bytes, and peak memory are tracked.

Targets are mutated by definition, so each dataset carries a merge_insert_base tag marking a pristine version. Benchmarks restore to that tag before every measured round, in an untimed setup hook, and the module teardown restores and cleans up the versions the run produced. Restoring from a tag rather than from "whatever the latest version was" means a crashed run cannot silently leave a drifted dataset behind for the next one.

That required one change to shared infrastructure: the io_memory_benchmark fixture had no way to reset state between its runs, and it always made an unmeasured warmup call. It now takes an optional setup callable, invoked before the warmup and before the measured run, and before IO stats are reset so the reset's own writes are never counted. Existing callers are unaffected.

Example

Two results from a local run, which show the kind of signal this is meant to surface. Both are legacy path vs DataFusion path on the same data.

Single-row upsert into a 1.1M-row target — the legacy index probe wins by 6.6x:

test_upsert_point[v1_indexed]      1.99 ms
test_upsert_point[v2_hash]        13.18 ms

Updating one scalar column across every row of a wide target — the legacy path patches the column while the DataFusion path rewrites whole rows, including the vector column:

test_io_mem_update_subset_row_fraction[100pct-v1_indexed]    81.3 KB written
test_io_mem_update_subset_row_fraction[100pct-v2_hash]       21.5 MB written

Not included

test_upsert_source_equals_target runs on the DataFusion path only. The legacy path cannot execute that shape at all — its source-side hash join requests more than the whole memory pool and the operation fails with "Resources exhausted" — so there is no baseline to compare against.

Benchmarks around repeated steady-state upsert loops and concurrent writers are deliberately out of scope; they pull compaction policy and conflict resolution in with them.

Testing

.github/workflows/ci-benchmarks.yml only runs on workflow_dispatch and on push to main, so PR CI will not exercise any of this. Verified locally instead: ran all 111 cases against scaled-down datasets (1.1M-row narrow target, 20K-row wide target), covering dataset generation, the restore-before-every-round setup, the row-count assertions, and the teardown cleanup. All pass.

Also confirmed that dataset.io_stats_incremental() attributes merge_insert's writes when read through the handle captured before the commit, so the io_memory_benchmark variants report real write_bytes rather than zero.

[hamersaw]

Do we have a sense of how long they will take from local runs? IIUC we didn't extend this benchmarking suite extensively because it's sensitive to longer delays.

[wjones127]

Do we have a sense of how long they will take from local runs? IIUC we didn't extend this benchmarking suite extensively because it's sensitive to longer delays.

These should all be fast (like milliseconds). I think the things we are reluctant to add are lots of FTS and vector index builds, whose runtimes can be measured in minutes.