perf(segment): reuse first vector index file as merge base during compaction

[JalinWang]

Closes #98.

Summary

ReduceVectorIndex always rebuilt merged vector indexes from scratch during compaction/optimize, even when no filtering was needed and the segments were layout-compatible. This PR adds a fast path that copies the first segment's vector index file as the merge base and only merges the tail segments into it, falling back to a full rebuild when reuse is unsafe.

Changes

Reuse path in segment compaction

src/db/index/segment/segment_helper.{h,cc}

• New MergeWithOptionalReuse(output_path, field, source_indexers, filter, concurrency, *merged_indexer) encapsulates the copy-or-rebuild decision. Called by both the unquantized branch and both flat/quantize legs of the quantized branch in ReduceVectorIndex.
• CanReuseFirstIndexer gates reuse on:
  • no filter (filter must be nullptr; not an empty RowIdFilter)
  • non-empty input
  • output index type ∈ { HNSW, HNSW_RABITQ, FLAT } — i.e. streaming-style indexes whose Merge appends to an in-memory graph. IVF/VAMANA rebuild from scratch in Merge (dump-then-reopen) and would silently drop the base file's docs, so they always take the full-rebuild path.
  • first indexer's index_type and quantize_type match the output field's
• On reuse: FileHelper::CopyFile → Open(create_new=false) → Merge(tail_indexers, …). On any copy failure, logs a warning and falls back to the rebuild path.
• ExecuteCompactTask now leaves row_id_filter null when the delete bitmap is empty, so the no-deletes compaction can actually take the reuse path.

Append-not-overwrite when merging into a non-empty target

src/core/mixed_reducer/mixed_streamer_reducer.cc

• reduce() initializes next_id from target_streamer_->create_(sparse_)provider()->count() when merging into a non-empty target (the reuse case). Previously it started at 0, which would stamp the new docs over the base data.
• Guarded against null provider so IVF and other not-yet-loaded streamers on the rebuild path are unaffected (also fixes a SIGSEGV in that path). This should be improved later.

Public accessor

src/db/index/column/vector_column/vector_column_indexer.h

• Added field_schema() getter so CanReuseFirstIndexer can compare the source indexer's schema (index/quantize type) against the output field.

Tests

tests/db/index/segment/segment_helper_test.cc

• Extracted a SegmentCompactReuseTest fixture; rewrote the merge-reuse cases as parameterized regressions covering:
  • HNSW / HNSW_RABITQ / FLAT — reuse path
  • IVF / VAMANA — fallback path (reuse must NOT trigger)
  • single-segment compaction (best case for file-copy reuse)
  • multi-segment compaction with a filter (fallback path)
  • mixed-type merge: writing-segment FLAT indexers → compacted HNSW/IVF/… output

Gains

zvec_segment_indexer_reuse_benchmark.md

Dataset	Total docs	Init Index	CompactTask (no reuse)	CompactTask (reuse)	Saved	Reduction	Speedup
int8-hnsw-1w	20,000	0.36 s	9.46 s	3.58 s	5.89 s	62.2%	2.65×
int8-hnsw-10w	110,000	11.31 s	65.53 s	4.45 s	61.08 s	93.2%	14.74×
int8-hnsw-100w	1,010,000	123.75 s	541.33 s	5.87 s	535.46 s	98.9%	92.2×

References

• Started from fix: optimize segment compaction by reusing first vector index #114 — kept as the base, rewrote the implementation (layering, type-safety, dead-code, IVF-correctness fixes).

[feihongxu0824]

有实测过比如hnsw 100w+1w这种case的optimize性能收益吗？

[feihongxu0824]

最好记录好测试结果，并且可以在commit message里面体现吧

[JalinWang]

report link:
zvec_segment_indexer_reuse_benchmark.md

[feihongxu0824]

可以直接放到commit message里面，这个link是在个人仓库单独维护的？

[feihongxu0824]

OK，看到commit message里面已经有了

[feihongxu0824]

建议去掉外部link，把test design+result放到commit message里面即可

[feihongxu0824]

这里的CreateVectorIndexTask和CompactTask (no reuse)的耗时为啥相差这么大？