Skip to content

feat: expose embedding chunk provenance - #209

Draft
sanikolaev wants to merge 9 commits into
masterfrom
feat/embedding-chunk-provenance
Draft

sanikolaev wants to merge 9 commits into
masterfrom
feat/embedding-chunk-provenance

Conversation

@sanikolaev

@sanikolaev sanikolaev commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator
  • preserve UTF-8 byte spans for automatically chunked text embeddings
  • return the winning row-local vector slot from grouped KNN search
  • validate FFI ownership, row offsets, spans, and vector-count bounds
  • update HNSW to upstream commit 72bb56474fc8f7942a433ba310db44391c075dd3 from feat: return the winning vector in grouped search hnswlib#2
  • bump embeddings ABI to 10 and KNN ABI to 19 without changing the KNN storage format

truncate and mean intentionally return no source spans because neither vector represents one truthful source chunk.

Blocked by manticoresoftware/hnswlib#4

Related issue: manticoresoftware/manticoresearch#4968

@sanikolaev
sanikolaev marked this pull request as draft September 30, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant