Add semantic (vector) search support via Solr's text-to-vector module - #3707
Open
luis100 wants to merge 2 commits into
Open
Add semantic (vector) search support via Solr's text-to-vector module#3707luis100 wants to merge 2 commits into
luis100 wants to merge 2 commits into
Conversation
Adds a knn_vector Solr field type and query-time vectorization via
Solr's language-models module, so free-text queries can be matched by
meaning rather than only by keyword overlap.
- New TextToVectorFilterParameter FilterParameter subtype (field, query,
model, topK) - vectorizes the query at search time via Solr's
{!knn_text_to_vector} query parser and restricts results to the topK
nearest neighbours of a knn_vector field. Composes with every other
filter type the same way (AND/OR/nesting).
- AIP/File/Representation collections each get an embedding_vector
field (knn_vector, not stored - vectors are large and only useful for
the KNN search itself) and a vectorized_b flag (boolean, default
false) that an external enrichment service can poll
(fq=vectorized_b:false) to find documents still awaiting
vectorization. RODA itself never populates embedding_vector - the
enrichment pass is a separate service by design, since running
embedding calls synchronously on every index write would be far too
slow for ingest.
- SolrBootstrapUtils registers the embedding model with Solr's
text-to-vector-model-store from new core.index.embedding.* config
(enabled flag, base URL, model name, dimensions, Solr model name,
default top-K) - an OpenAI-compatible /embeddings endpoint, the same
contract any external enrichment service should call so both sides of
the vector space stay consistent.
- appendTextToVector passes the query text through the quoted `v=`
local param rather than as trailing text after the local params
block. Trailing text is only unambiguous for a single word - a
multi-word query gets combined with other filters into one larger
boolean query string, and the outer parser can split on the
whitespace and try to resolve the extra words against Solr's default
search field, which RODA doesn't define. Verified live against a real
embedding server with a 12-word natural-language query correctly
ranking the semantically relevant document first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01263fsB7QYevDFgwAY8okcq
dosubot
Bot
added
size:L
Without SOLR_MODULES: language-models, Solr never loads the knn_text_to_vector query parser this PR's TextToVectorFilterParameter depends on - semantic search would silently fail with an "unknown query parser" error in both the dev and production-like standalone deploys. Also adds a minimal docker-compose-vector-test.yaml (just ZooKeeper + Solr + PostgreSQL) for quickly verifying the embedding model registration and schema wiring without the full dev stack. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01263fsB7QYevDFgwAY8okcq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
knn_vectorSolr field type and query-time vectorization via Solr'slanguage-modelsmodule, so free-text queries can be matched by meaning rather than only by keyword overlap.TextToVectorFilterParameter(field,query,model,topK) - composes with every otherFilterParametertype via the existing AND/OR/nesting mechanism.embedding_vectorfield (knn_vector, not stored) and avectorized_bflag (defaultfalse) that an external enrichment service polls to find documents still awaiting vectorization. RODA itself never populatesembedding_vector- the second-pass enrichment is a deliberately separate service, since synchronous embedding calls on every index write would be far too slow for ingest.SolrBootstrapUtilsregisters the embedding model with Solr'stext-to-vector-model-storefrom newcore.index.embedding.*config (enabled flag, base URL, model name, dimensions, Solr model name, default top-K) - an OpenAI-compatible/embeddingsendpoint contract, so any external enrichment service stays consistent with query-time vectorization.appendTextToVectorpasses the query text through the quotedv=local param rather than as trailing text after the local params block, fixing a bug where any multi-word query broke with400 undefined field _text_(trailing text is only unambiguous for a single word once combined with other filters into one larger boolean query string).Test plan
mvn -pl roda-common/roda-common-data,roda-core/roda-core -am install -DskipTests)knn_vectorfield, andTextToVectorFilterParameterall confirmed working, including a 12-word natural-language query correctly ranking the semantically relevant document first (proving the multi-word fix)SolrUtilsTestunit tests included for the new filter parameter (not re-run against the full TestNG integration harness in this session - relying on the live end-to-end verification above plus a clean compile)Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com