fix: support quoted exact-phrase full text search

[adityasingh2400]

Fixes #137

Root cause

The full text search indexes text with flexsearch using the LatinBalance encoder:

const options: IndexOptions = {
  tokenize: "forward",
  encoder: Charset.LatinBalance,
};

LatinBalance is a phonetic-style encoder that maps similar-looking words to the same token. That is great for fuzzy recall, but it means a query can match words the user did not intend. As reported in the issue, searching for "aldi" returns "ALDEA HOMES" rows ahead of the real "ALDI" rows because both encode to overlapping tokens. The reporter asked for a way to express an exact match, and a collaborator confirmed the encoder is the cause.

Fix

This adds an opt-in exact-match path. A query wrapped in double quotes, for example "aldi", is treated as a case-insensitive substring match against the original text rather than the fuzzy token search. Unquoted queries keep the existing fuzzy behavior, so nothing changes for current searches.

To support this, the index now keeps the original text per id alongside the flexsearch index, and the exact path scans those texts while still respecting the result limit.

To make the logic testable in a node environment, the SearchIndex class moves out of the worker entry into a standalone module (search_index.ts) that the worker imports and re-exports. The public worker surface is unchanged.

Verification

Added test/search_index.test.ts. One test reproduces the issue by showing the fuzzy search returns the "ALDEA" rows for "aldi", and another asserts the quoted query "aldi" returns only the rows that actually contain that substring. The exact-match test fails before this change and passes after it.

Commands run in packages/viewer:

• npx vitest run, 40 passed across 3 files
• npx prettier -c on the changed files, all clean

The unquoted fuzzy path is left untouched, so this is additive behavior gated behind quoting.

[domoritz]

What if I want to search for "aldi" store? I guess that's not supported?

[domoritz]

Would it be simpler to check for longer than 2 above instead of longer or equal?

[adityasingh2400]

Good catch, and your other comment about "aldi" store pushed me to rethink this whole parse step. I replaced parseExactPhrase with a parseQuery that walks the string quote by quote, so the special-case length guard is gone entirely. A quoted run becomes a phrase, anything outside the quotes stays free text, and empty quotes just contribute nothing. That removed the >= 2 vs > 2 ambiguity you flagged here.

[adityasingh2400]

What if I want to search for "aldi" store? I guess that's not supported?

It is now, I pushed 9d5bcaa to support exactly that. The query parser splits a string into quoted phrases plus the leftover free text, so "aldi" store parses to one required phrase (aldi) and the free text store. The phrase is matched as an exact case-insensitive substring, the free text goes through the normal fuzzy index, and a row has to satisfy both. You can also stack phrases, "aldi" "downtown" requires both substrings.

Concretely, with rows ALDI Supermarket, Corner ALDI, and ALDI store downtown, searching "aldi" returns all three, while "aldi" store narrows to just ALDI store downtown. A plain unquoted query is unchanged, it is all free text with no phrases, so the existing fuzzy path is untouched. New tests cover the mixed, multi-phrase, and unterminated-quote cases.

[donghaoren]

Here we are storing all the texts another time, would it be possible to run the exact search as a query in DuckDB so we don't have to store them another time?

[adityasingh2400]

Good call, done. The phrases are now matched in DuckDB and nothing extra is stored.

search.worker.ts is back to being byte-identical to main, so the worker holds only the flexsearch index as before. The exact match runs as lower(text) LIKE '%phrase%' ESCAPE '\\' against the table, with the predicate applied in the same query.

Two things fell out of it that I think make this better than what I had:

A query made entirely of phrases is now answered by the database alone. It skips building the fuzzy index completely, so I made the worker lazy rather than constructed eagerly, and that path no longer pays the indexing pass at all.

For a mixed query like "aldi" store, the fuzzy hits for the free text become the candidate set and the SQL narrows them with an IN list, so the phrase filters the ranking rather than replacing it. One detail worth flagging: flexsearch treats limit: 0 as its default of 100 rather than as unlimited, so the candidate search passes an explicit large bound to avoid silently truncating before the filter runs.

Since the phrase is user input I escape %, _ and backslash so wildcards match literally, with a test covering "50%".

Locally: 8 search tests pass, and the viewer suite is at 146 passed with the same 2 pre-existing failures that are on main (the timing-sensitive retry tests in inference_utils and inference_embedding).

[Yigtwxx]

I went through this while looking at the search code for an unrelated reason and the latest revision reads well — the rewrite after @donghaoren's feedback is a real improvement over the in-worker version. Two things in particular stood out:

• Making the worker lazy is a nice consequence of pushing the match into SQL. A phrase-only query now skips the whole indexing pass, which on a large table is the difference between an instant result and a visible "Indexing..." wait.
• Escaping %, _ and the escape character itself is the kind of detail that usually gets missed until someone searches for a percentage.

I left four comments. The one about the unbounded candidate set in the mixed path is the only one I'd consider blocking; the rest are smaller.

One thing that isn't in the diff: this adds a user-facing query syntax with no way to discover it. packages/docs/overview.md mentions full-text search but doesn't describe how queries are interpreted, and the search input gives no hint that quoting does anything. A couple of sentences in the docs plus a placeholder or title on the input would go a long way — right now a user has to read query_parser.ts to learn the feature exists.

Not a maintainer, so take this as one reader's opinion rather than a gate.

[Yigtwxx]

Moving the phrase match into DuckDB solved the duplicate-text-storage problem, but the mixed path reintroduces a similar cost in a different place: the candidate set is now unbounded, and it gets serialized into the SQL text.

backend.query(freeText, UNLIMITED_SEARCH_RESULTS) returns every id flexsearch matches — search.worker.ts passes the limit straight to index.search, so there is no cap. Those ids cross the worker boundary as a structured clone, and then queryPhrases turns each one into a literal in id IN [...].

Concretely: on a 1M-row table, a query like "aldi" the matches the free text almost everywhere, so the candidate list is close to the whole table and the generated statement is tens of megabytes of id literals handed to duckdb-wasm. The IN [...] pattern in querySearchResultItems looks the same but is safe because it is only ever called with a limit-sized list (~100).

One way out is to drop the IN-list entirely and intersect on the JS side:

let candidateIDs = await backend.query(freeText, UNLIMITED_SEARCH_RESULTS);
let matched = new Set(
  await this.queryPhrases(phrases, this.predicateString(predicate), null, UNLIMITED_SEARCH_RESULTS),
);
let resultIDs = candidateIDs.filter((id) => matched.has(id)).slice(0, limit);

Same ranking, same semantics, and the SQL string stays constant-size. It costs one full LIKE scan — but the phrase-only path already pays exactly that scan, so it is not new work for the database. The candidate list is still unbounded in memory, which I think is acceptable since the flexsearch result set is already materialized in the worker either way.

Worth noting the id types line up for the Set here: both sides originate from the same coordinator.query, so a BIGINT column stays bigint on both.

[adityasingh2400]

Agreed, and done in 0255f2a the way you sketched it. queryPhrases no longer takes a candidate list. The mixed path runs the phrase scan unbounded, puts the ids in a Set, and filters the fuzzy candidates against it in JS, so the SQL text is now a constant size whatever the table holds. The candidate ids never leave the JS side.

I also added an early return when the fuzzy search comes back empty, since the phrase scan cannot add anything to an empty candidate set.

On the id types: I checked flexsearch 0.8 round-trips bigint ids unchanged (index.add(1n, ...) comes back as 1n from search), and the new duckdb-wasm test in 4703f1b feeds the searcher bigint ids taken from a real BIGINT column on both sides and asserts the intersection is non-empty and in fuzzy order.

Unit test a mixed query never serializes the candidate ids into the SQL asserts the phrase query contains neither IN [ nor LIMIT.

[Yigtwxx]

LIMIT without an ORDER BY makes the phrase-only results nondeterministic. DuckDB gives no row-order guarantee here, so searching "aldi" twice on a table with more than limit matches can legitimately return two different sets of 100 rows — and the order can shift as the table is scanned differently.

There is also an asymmetry between the two paths that users will notice: the fuzzy path returns results ranked by relevance, while this one returns whatever 100 rows the scan reached first. For a phrase match there is no relevance signal to rank by, so I don't think you need to invent one, but an explicit ORDER BY ${idColumn} would at least make repeated searches stable and make the "first 100" mean something.

[adityasingh2400]

Done in 0255f2a. When a limit is set the phrase query is now ORDER BY "id" LIMIT n, and the docstring on queryPhrases says why. For the mixed path, which now runs unbounded and intersects in JS, there is no ORDER BY and no LIMIT, since the fuzzy ranking decides the order there and sorting the full match set would be wasted work.

Covered twice: the unit test asserts the SQL shape, and the duckdb-wasm test runs "aldi" with limit: 2 twice against a real table with three matches and asserts both calls return [2, 3].

[Yigtwxx]

This stub is a small SQL engine implemented in regexes (/like '%(.*?)%' escape/g, / IN \[/, /LIMIT (\d+)/), which means the tests verify the shape of the string you build but never that it is valid DuckDB.

That matters most for the two least obvious pieces of the generated SQL:

• id IN [a, b, c] — the list-literal form. It matches the existing usage in querySearchResultItems, so it is almost certainly right, but nothing here would catch it if it weren't.
• ESCAPE '\\' in the template literal, which reaches the database as ESCAPE '\'. That relies on DuckDB not applying backslash escaping inside single-quoted strings. The "50%" test passes because the stub strips \\(.) in JS, not because DuckDB agreed.

The parser tests and the ranking tests are genuinely good and I'd keep them as they are — my concern is only the SQL-generating half. Would it be feasible to add one test against a real duckdb-wasm coordinator covering the phrase query, or failing that, note in the PR description how you verified the escaping end to end in the app?

[adityasingh2400]

You were right that the stub only proved the shape. 4703f1b adds test/search_duckdb.test.ts, which runs FullTextSearcher against @duckdb/duckdb-wasm/blocking, the node build of the same package the viewer ships (v1.5.1 engine here). It instantiates in about 300 ms so it sits in the normal vitest run rather than behind a flag.

It covers the pieces the stub could not:

• "50%", "a_b" and "back\slash" each match exactly the one row that contains the literal characters, so ESCAPE '\' reaches DuckDB the way the template literal intends and \%, \_ and \\ are taken literally.
• a predicate composes with the phrase condition.
• the id-ordered limit returns the same rows on a repeat.
• the mixed path intersects bigint ids from a real BIGINT column with bigint ids standing in for the fuzzy index.

The IN [...] list-literal form no longer appears in this PR's SQL after 0255f2a, so it is not exercised here, though I did confirm DuckDB accepts "id" IN [1, 3] while checking the escaping.

I also ran the new tests against the previous search.ts to make sure they discriminate: 4 of the 23 fail there (the two mixed-path assertions and the ordered limit, in both files), and all 23 pass on the branch. The stub's IN handling is gone since nothing generates it any more, and it now honours ORDER BY "id" so the unit-level limit test checks order too.

[Yigtwxx]

Joining the fragments with a space means a quote in the middle of a word silently splits it: re"d"x parses to phrase d plus free text re x, so the free-text side searches for two tokens the user never typed. Related, there is no way to search for a literal " at all, since every quote is consumed as a delimiter.

Neither is worth adding syntax for in my opinion — a quote mid-word is a weird thing to type, and an escape character would be more machinery than the feature needs. But the docstring above is careful about the other edge cases (empty quotes, unterminated quote), so these two belong in the same list, if only so the next reader knows the behavior is deliberate rather than an oversight.

[adityasingh2400]

Agreed that neither wants syntax. 2dfe255 adds both to the docstring's list of edge cases, stated as deliberate, and a unit test pins re"d"x to { phrases: ["d"], freeText: "re x" } so a later reader sees the behaviour is chosen rather than accidental.

[adityasingh2400]

@Yigtwxx thanks for the read. All four threads are addressed, replies inline, and the discoverability point is done in d6d0e51: packages/docs/overview.md now says search is fuzzy by default and that a double-quoted run requires an exact, case-insensitive substring, with the "aldi" store example, and the full-text search input has the placeholder Search, or "quote" text to match it exactly.

I also rebased onto current main (clean, no conflicts). Commits on top of the previous three:

• 0255f2a: intersect mixed-query candidates in JS, order phrase matches by id
• 2dfe255: document the mid-word quote and literal quote edge cases
• 4703f1b: cover the generated phrase SQL against duckdb-wasm
• d6d0e51: docs paragraph and search box placeholder

Verification on the branch:

packages/viewer: vitest run
 Test Files  12 passed (12)
      Tests  166 passed (166)

The two timing-sensitive retry tests I mentioned earlier as failing on main pass now. prettier -c is clean on every changed file. svelte-check reports 18 errors, all Cannot find module '@embedding-atlas/component' from not having the wasm build chain locally, none in the touched files.