README.md describes retrieval as:
Searches traces and skills with hybrid lexical + semantic retrieval (BM25 fallback when embeddings off)
That is not what the code does.
BM25 was tried and removed — src/shell/grep-core.ts:351-355:
// BM25 tried and dropped (PR-NOTES F4c): score scale (~1..3) overpowered
// cosine in UNION, semantic hits were pushed out of top-K. LIKE is a
// better fit for "find any session mentioning X" which is the actual
// plugin use case.
BM25 index creation is also disabled — src/deeplake-api.ts:548-549:
// BM25 index disabled — CREATE INDEX causes intermittent oid errors on fresh tables.
// See bm25-oid-bug.sh for reproduction. Re-enable once Deeplake fixes the oid invalidation.
The actual hybrid (grep-core.ts:343-350) is a single UNION ALL of lexical LIKE/ILIKE rows emitting a sentinel score of 1.0 (capped by HIVEMIND_HYBRID_LEXICAL_LIMIT, default 20) with semantic rows emitting real cosine, ordered by score — so exact substring matches always outrank semantic ones.
src/embeddings/disable.ts:26 carries the same stale wording ("falls back to BM25 / ILIKE matching").
Suggested fix: update the README bullet and the disable.ts comment to say lexical LIKE/ILIKE, and describe the sentinel-score fusion. Worth stating the rationale too — it's a good design note: BM25's unbounded score scale is incompatible with cosine in a single ORDER BY without rank-based fusion (RRF) or normalization.
README.mddescribes retrieval as:That is not what the code does.
BM25 was tried and removed —
src/shell/grep-core.ts:351-355:BM25 index creation is also disabled —
src/deeplake-api.ts:548-549:The actual hybrid (
grep-core.ts:343-350) is a singleUNION ALLof lexicalLIKE/ILIKErows emitting a sentinel score of 1.0 (capped byHIVEMIND_HYBRID_LEXICAL_LIMIT, default 20) with semantic rows emitting real cosine, ordered by score — so exact substring matches always outrank semantic ones.src/embeddings/disable.ts:26carries the same stale wording ("falls back to BM25 / ILIKE matching").Suggested fix: update the README bullet and the
disable.tscomment to say lexicalLIKE/ILIKE, and describe the sentinel-score fusion. Worth stating the rationale too — it's a good design note: BM25's unbounded score scale is incompatible with cosine in a singleORDER BYwithout rank-based fusion (RRF) or normalization.