Skip to content

fix: align custom IDs in array-built indexes - #422

Merged
ASuresh0524 merged 2 commits into
StarTrail-org:mainfrom
dafyy321-pixel:fix/custom-array-ids
Sep 26, 2026
Merged

ASuresh0524 merged 2 commits into
StarTrail-org:mainfrom
dafyy321-pixel:fix/custom-array-ids

Conversation

@dafyy321-pixel

@dafyy321-pixel dafyy321-pixel commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Context

build_index_from_arrays() accepts IDs alongside precomputed embeddings. A caller can also use add_text() to supply the passages those embeddings belong to. In that combination, add_text() gives the passages sequential IDs, while the array build sends the caller's IDs to the vector backend. A search can find the vectors but cannot resolve their text from the passage store. The build itself completes, which makes the empty search results hard to trace.

For example, two add_text() calls followed by build_index_from_arrays(path, ["doc-a", "doc-b"], embeddings) used to write passage IDs 0 and 1 while indexing the vectors as doc-a and doc-b.

Change

  • Convert the supplied IDs to strings once and use them consistently for passages, offsets, the ID map, and the vector backend.
  • Reject IDs that collide after string conversion before writing index files.
  • Preserve caller-provided chunk metadata.
  • Add mock-backend regression tests for custom-ID search results and ID collisions.

Verification

  • The two new regression tests passed without a model download.
  • Ruff lint and format checks passed for the changed files.

Fixes #424.

ASuresh0524 added a commit that referenced this pull request Sep 26, 2026
`Type Check with ty` fails on main itself, so every PR opened against it
inherits a red check for code it never touched. #420, #421 and #422 are each
blocked this way, on files none of them modify.

Reproduced with CI's exact invocation (`ty check packages/leann-core/src apps
tests`, ty pinned to 0.0.17) against a pristine checkout: 5 diagnostics, the
same 5 CI reports.

- Three `# type: ignore[...]` comments use mypy codes ty does not recognise, so
  it treats them as blanket suppressions, finds nothing suppressed, and reports
  them unused. Removed, which is ty's own suggested fix.
- `tests/test_chrome_history_reader.py` assigns attributes onto bare
  `types.ModuleType` stubs when llama_index is absent. Those are genuine
  `unresolved-attribute` errors. The stubs now come from a helper returning
  `Any`. A suppression comment would not do: CI type-checks with no project
  dependencies installed while a developer machine has llama_index, so an
  ignore would be needed in one environment and flagged unused in the other.
  Annotating the variables as `Any` is also insufficient -- the assignment
  narrows the type straight back to `ModuleType`.

ty now passes. ruff is unchanged at 17 pre-existing findings, none new, and the
14 tests in the two touched files pass.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@ASuresh0524
ASuresh0524 merged commit 7c268aa into StarTrail-org:main Sep 26, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build_index_from_arrays loses passages when custom IDs follow add_text

2 participants