Fix ivf index bugs#313
Open
ivoras wants to merge 4 commits into
Open
Conversation
Four bugs in the IVF index could corrupt data or return wrong results: - Slot allocation was append-only (slot = n_vectors), so after any delete left an interior hole, a later insert overwrote a live vector — corrupting its data and dropping the row from KNN results. Allocate the first free slot from the validity bitmap instead. - assign-vectors walked the centroid array with a D-byte stride instead of D*sizeof(float), so every centroid past index 0 was garbage, and it mapped the nearest-centroid array index directly to a centroid id (wrong for the non-contiguous ids that set-centroid allows). It also treated quantized cell blobs as float32. Now operates entirely in quantized space with the correct stride and an id lookup array. - clear-centroids re-inserted raw float32 bytes into quantized cells without re-quantizing, corrupting every vector on a quantized index. - set-centroid stored full-precision centroids that mismatched the quantized cell vector size, breaking subsequent inserts. Now stores the quantized representation. Also make ivf_distance dispatch int8 to L1/cosine/L2 by the column's distance metric (was always L2), and remove a dead undefined-behavior ivf_ensure_stmt call in ivf_cell_find_or_create. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tests/test-unit.c had been mangled by a bad rebase — function bodies were interleaved and the file never compiled, so its IVF/rescore/DiskANN unit tests were never actually run. Reconstruct it from git history. While fixing it: - tests/sqlite-vec-internal.h re-declares struct VectorColumnDefinition for the test translation unit; its Vec0RescoreConfig was missing the oversample_search field present in sqlite-vec.c. The size mismatch shifted the ivf/diskann fields and made vec0_parse_vector_column scribble past the caller's struct (stack smash). Add the missing field. - The test-unit target now defines SQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1 so the IVF unit tests link and run. - The rescore int8 quantizer is a fixed [-1,1] -> [-128,127] map, not the adaptive min/max the historical test assumed; update those assertions. - Add the #included index sources (sqlite-vec-ivf*.c, sqlite-vec-rescore.c, sqlite-vec-diskann.c) as prerequisites of the loadable/static targets so edits to them trigger a rebuild. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regression tests covering each IVF bug fix: - test-ivf-mutations.py: insert-after-delete slot reuse (no overwrite), randomized insert/delete churn consistency, and assign-vectors correctness including non-contiguous centroid ids. - test-ivf-quantization.py: clear-centroids / set-centroid / assign-vectors on int8 and binary quantized indexes preserve vectors and recall. New test-ann-embeddings.py exercises IVF, DiskANN, and rescore end to end with real text embeddings from a local llama-server endpoint (localhost:2235), checking recall against exact brute-force results. The whole module skips when the server is unreachable, so it is safe to run in CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an "Approximate nearest neighbor (ANN) indexes" section describing the rescore, ivf, and diskann index types, their compile flags, and usage. Include a latency/speedup/recall benchmark table at both 256 and 1024 dimensions and a discussion of how the trade-offs shift as dimensionality grows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains fixes to the vector indexes code, worked out with Fable and Opus 4.8, and adds a chapter on using the indexes to the README file. It contains commits of different stages of the work, so you can cherry pick those that you're comfortable with.