Skip to content

docs: README claims "BM25 fallback when embeddings off" — BM25 was tried and dropped #337

Description

@kaghni

README.md describes retrieval as:

Searches traces and skills with hybrid lexical + semantic retrieval (BM25 fallback when embeddings off)

That is not what the code does.

BM25 was tried and removedsrc/shell/grep-core.ts:351-355:

// BM25 tried and dropped (PR-NOTES F4c): score scale (~1..3) overpowered
// cosine in UNION, semantic hits were pushed out of top-K. LIKE is a
// better fit for "find any session mentioning X" which is the actual
// plugin use case.

BM25 index creation is also disabledsrc/deeplake-api.ts:548-549:

// BM25 index disabled — CREATE INDEX causes intermittent oid errors on fresh tables.
// See bm25-oid-bug.sh for reproduction. Re-enable once Deeplake fixes the oid invalidation.

The actual hybrid (grep-core.ts:343-350) is a single UNION ALL of lexical LIKE/ILIKE rows emitting a sentinel score of 1.0 (capped by HIVEMIND_HYBRID_LEXICAL_LIMIT, default 20) with semantic rows emitting real cosine, ordered by score — so exact substring matches always outrank semantic ones.

src/embeddings/disable.ts:26 carries the same stale wording ("falls back to BM25 / ILIKE matching").

Suggested fix: update the README bullet and the disable.ts comment to say lexical LIKE/ILIKE, and describe the sentinel-score fusion. Worth stating the rationale too — it's a good design note: BM25's unbounded score scale is incompatible with cosine in a single ORDER BY without rank-based fusion (RRF) or normalization.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions