Add cuvs-bench-elastic: HTTP backend for Elasticsearch GPU vector search by afourniernv · Pull Request #1907 · NVIDIA/cuvs

afourniernv · 2026-03-10T22:31:01Z

Introduce cuvs-bench-elastic as an optional plugin for cuvs-bench that provides an Elasticsearch backend. The backend communicates with Elasticsearch via HTTP and supports HNSW indexing with optional GPU acceleration when using the Elasticsearch GPU image (cuVS-accelerated vector search).

Add cuvs_bench_elastic package with backend and config loader entry points
Extend cuvs_bench registry and search spaces for pluggable backends
Add elastic and integration optional dependencies to cuvs-bench
Add modularization tests and integration test scaffolding (disabled until CI has ES GPU image, cuVS libs, and GPU runner)

copy-pr-bot · 2026-03-10T22:31:05Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Introduce cuvs-bench-elastic as an optional plugin for cuvs-bench that provides an Elasticsearch backend. The backend communicates with Elasticsearch via HTTP and supports HNSW indexing with optional GPU acceleration when using the Elasticsearch GPU image (cuVS-accelerated vector search). - Add cuvs_bench_elastic package with backend and config loader entry points - Extend cuvs_bench registry and search spaces for pluggable backends - Add elastic and integration optional dependencies to cuvs-bench - Add modularization tests and integration test scaffolding (disabled until CI has ES GPU image, cuVS libs, and GPU runner) Signed-off-by: Alex Fournier <afournier@nvidia.com>

Use single-doc format (_index, _id, vector_field) instead of two-part NDJSON (index action + source) so ES accepts the bulk request. Signed-off-by: Alex Fournier <afournier@nvidia.com>

Expose ELASTIC constant and convenience wrappers for build-only, search-only, or full benchmark runs. Signed-off-by: Alex Fournier <afournier@nvidia.com>

- Document run_build, run_search, run_benchmark convenience API - Document ELASTIC constant and orchestrator usage - Add username/password support in config loader (converts to basic_auth) Signed-off-by: Alex Fournier <afournier@nvidia.com>

- New `cuvs_bench_elastic` package with HTTP backend for Elasticsearch GPU vector search (HNSW, int8_hnsw, int4_hnsw, bbq_hnsw index types) - Supports `pip install cuvs-bench[elastic]` without a separate PyPI publish: `cuvs_bench` bundles the plugin via setuptools packages.find - Plugin registers via entry points (`cuvs_bench.backends` / `cuvs_bench.config_loaders`) — no changes to core cuvs-bench required - `ElasticConfigLoader` reads shared `datasets.yaml` from cuvs_bench and `elastic.yaml` from the plugin config; supports sweep and tune modes - `build()` checks index existence before file validation so `force=False` returns immediately without requiring the base file on disk - Removed testcontainers-based integration tests; added unit tests for pre-flight failure, force=False skip, dry-run, helper functions - `elasticsearch` client is an optional dep (`cuvs-bench[elastic]` extra)

The separate cuvs_bench_elastic package required bundling via packages.find and complicated the build. Instead, keep the backend inside cuvs_bench and use entry points pointing back into the same package so the backend only registers when elasticsearch is installed. - git mv backend.py to cuvs_bench/backends/elasticsearch.py - git mv elastic.yaml to cuvs_bench/config/algos/ - Fix imports to relative paths - Fix _get_elastic_config_path() to use ../config from backends/ - Update pyproject.toml: entry points -> cuvs_bench.backends.elasticsearch:register - Remove packages.find (no longer needed) - Remove cuvs_bench_elastic/ package entirely DX unchanged: pip install cuvs-bench[elastic] One package, one publish pipeline. Signed-off-by: Alex Fournier <afournier@nvidia.com>

…ch.py Restores the high-level API that was previously in cuvs_bench_elastic/__init__.py so existing demo scripts continue to work after the module consolidation. Signed-off-by: Alex Fournier <afournier@nvidia.com>

cjnolet · 2026-04-16T18:08:24Z

+                    "Install with: pip install cuvs-bench[elastic]"
+                ) from e
+            host = self.config.get("host", "localhost")
+            port = self.config.get("port", 9200)


Just want to confirm- are there any other protocols to interact with Elasticsearch or is HTTP/Rest pretty much it? Mostly asking to make sure it's the best (fastest) way to comunicate so we aren't adding additional overheads in the mix.

Confirmed. For Elasticsearch the supported client path here is the official HTTP/REST interface; the backend does not add another protocol layer on top of that. We re-ran the backend live against the Brev deployment after the recent fixes. Leaving this open since it is more an interface tradeoff than a code bug.

cjnolet · 2026-04-16T18:14:47Z

+            index_type = build_params.get("type", _DEFAULT_INDEX_TYPE)
+            m = build_params.get("m", _DEFAULT_M)
+            ef_construction = build_params.get(
+                "ef_construction", _DEFAULT_EF_CONSTRUCTION


We should decouple the index type from the backend type. User should be able to specify the backend and the algorithm they want to run in that backend (or maybe the two are coupled, but the backend should not assume the algorithm).

So for example, what if the index type is not supported? What happens when a user wants to test diskbbq against hnsw/cagra? We should decouple these from the beginning, ideally with separate functions for each (for e.g. _parse_cagra_params(), _parse_hnsw_params())

It's also important that we throw a proper (descriptive) error when the backend doesn't support an index type. For example, if someone passes in "SCaNN" to the Elasticsearch backend, they should get a message to the effect of "Received params for SCaNN index type, but Elasticsearch backend does not support this index type. Please check configuration."

Partially addressed. The backend now validates supported algorithm names, index types, and similarities up front with descriptive errors instead of failing later in Elasticsearch. It still models the benchmark family as elastic_* plus build-param type, so this is better than before but not a fully generic backend/index abstraction yet.

cjnolet · 2026-04-16T18:52:58Z

+# SPDX-License-Identifier: Apache-2.0
+#
+"""
+Smoke tests for cuvs-bench modularization (optional deps, entry points, lazy loading).


It's great to see improved test coverage!

cjnolet · 2026-04-16T18:55:13Z

@afourniernv this is a really big PR so I'm reviewing in stages. Thank you for bearing with me. I finally had some time to look through the implementation for comprehensively. Mostly minor, but important things.

# Conflicts: # python/cuvs_bench/cuvs_bench/orchestrator/orchestrator.py # python/cuvs_bench/cuvs_bench/tests/test_registry.py

jnke2016 · 2026-06-25T10:01:54Z

+            dataset=dataset, dataset_path=dataset_path, **kwargs
+        )
+
+    def _discover_algo_groups(


Parameter expansion looks good. _discover_algo_groups() just returns the algo groups and _build_benchmark_configs() receives the already-expanded build_combos / search_combos from the base loader. The framework handles the expansion, same as OpenSearch.

jnke2016 · 2026-06-25T12:00:26Z

+                build_params[k] = v
+
+        try:
+            client = self._get_client()


Is the reason for connecting lazily through _get_client() inside build()/search(), rather than in initialize(), to avoid reaching a live server on a dry_run? The orchestrator always calls initialize(), even for a dry run, while build()/search() return early on dry_run before touching the client, so the lazy path seems to keep dry runs from needing a server.

Yes, that is the reason. initialize() intentionally does not create the Elasticsearch client because the orchestrator calls initialize() even for dry_run; keeping connection setup inside _get_client() lets build() / search() return early on dry runs without requiring a live cluster. Real build/search paths still call _check_network_available() and then create the client lazily before issuing requests.

jnke2016 · 2026-06-25T12:32:28Z

+        index_name = self.config.get("index_name", "cuvs_bench_vectors")
+        idx = indexes[0] if indexes else None
+        build_params = dict(idx.build_param or {}) if idx else {}
+        for k, v in self.config.items():


This loop is redundant in the normal flow. build_params already copies idx.build_param (line 223), and the loader spreads that same dict into backend_config via **build_param (line 818), so every _BUILD_PARAM_KEY in self.config is already in build_params. The remaining entries in self.config are connection keys (host, port, …) that the loop's _BUILD_PARAM_KEYS filter skips. So it never adds anything and can be dropped, leaving idx.build_param as the single source.

The only case where the loop does something is the unusual one where indexes is empty (idx is None, so build_params starts as {}) — but that doesn't happen in the orchestrator path, since each config carries exactly one IndexConfig.

Addressed in 1b81f44. build() now takes its build params directly from the IndexConfig it was given, which is the normal orchestrator path and matches the OpenSearch backend pattern instead of redundantly backfilling the same keys from self.config.

jnke2016 · 2026-06-25T12:59:11Z

+# ── Convenience API ───────────────────────────────────────────────────────────
+
+
+def run_build(


The module-level run_build / run_search / run_benchmark helpers duplicate the entry point already provided by BenchmarkOrchestrator and the CLI, they just call orchestrator.run_benchmark(...) as below:

run_build(...)

# Python API

BenchmarkOrchestrator(backend_type="elastic").run_benchmark( build=True, search=False, dataset=..., dataset_path=..., algorithms=..., host=..., port=..., force=force, )

# CLI

python -m cuvs_bench.run --build \ --dataset <name> --dataset-path <path> --algorithms <group> \ --backend-config <elastic-config>

run_search(...)

# Python API

BenchmarkOrchestrator(backend_type="elastic").run_benchmark( build=False, search=True, dataset=..., dataset_path=..., algorithms=..., host=..., port=..., )

# CLI

python -m cuvs_bench.run --search \ --dataset <name> --dataset-path <path> --algorithms <group> \ --backend-config <elastic-config>

run_benchmark(...)

# Python API

BenchmarkOrchestrator(backend_type="elastic").run_benchmark( build=build, search=search, dataset=..., dataset_path=..., algorithms=..., host=..., port=..., force=force, )

# CLI

python -m cuvs_bench.run --build --search \ --dataset <name> --dataset-path <path> --algorithms <group> \ --backend-config <elastic-config>

Addressed in fcc13f5. The elastic-only run_build / run_search / run_benchmark wrappers are removed so the backend relies on the same orchestrator and CLI entrypoints as the rest of cuvs-bench, matching the OpenSearch backend structure.

@singhmanas1

Would still like to refine a bit but pushing up for early feedback cc @singhmanas1 @janakivamaraju @afourniernv xref NVIDIA#1907 which does something similar but for Elastic EDIT: Okay, this is now ready for review (cc @jnke2016) Authors: - James Bourbeau (https://github.com/jrbourbeau) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Joseph Nke (https://github.com/jnke2016) - Corey J. Nolet (https://github.com/cjnolet) - James Lamb (https://github.com/jameslamb) URL: NVIDIA#2012

jnke2016 · 2026-07-16T11:23:55Z

+    "number_of_replicas",
+    "vector_field",
+)
+_SEARCH_PARAM_KEYS = ("num_candidates", "vector_field")


This is not referenced anywhere anymore I think so it can be removed

Addressed in 4df7c9c. Removed the now-unused Elastic parameter-key constants; the backend now takes build/search params directly from the config-loader/index config flow.

jnke2016 · 2026-07-16T11:24:27Z

+    )
+
+
+def _get_elastic_config_path() -> str:


This is not referenced anywhere I think so it can be removed

Addressed in 4df7c9c. Removed the unused _get_elastic_config_path() helper; ElasticConfigLoader uses _get_cuvs_bench_config_path() directly.

jnke2016 · 2026-07-16T11:32:11Z

 from .config_loaders import DatasetConfig


+def _should_compute_recall(result: SearchResult) -> bool:


Now that recall is computed by the orchestrator for every backend, no backend sets recall_is_authoritative, so the flag branch in _should_compute_recall is only exercised by a unit test (test_modularization.py), not by any real backend. Now that recall is computed by the orchestrator for every backend, the flag no longer has a caller. You may want to simplify _should_compute_recall to success and neighbors.size > 0 (the empty-neighbors check already covers the C++ subprocess case) and drop the corresponding test.

Addressed in 4df7c9c. _should_compute_recall() now only checks result.success and result.neighbors.size > 0, and the obsolete authoritative-recall unit test was removed. No backend uses or sets recall_is_authoritative anymore.

jnke2016

Thanks @afourniernv for addressing the feedback, the PR looks good to me. Just a few minor cleanup requests

afourniernv · 2026-07-21T18:43:48Z

Hi @cjnolet, this PR has been caught up with latest main and the latest review comments have been addressed. Could you take a look and help with final review/merge. Requested per @jnke2016

github-project-automation Bot added this to Unstructured Data Processing Mar 10, 2026

cjnolet assigned afourniernv Mar 10, 2026

cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Mar 10, 2026

cjnolet moved this to In Progress in Unstructured Data Processing Mar 10, 2026

jnke2016 reviewed Mar 11, 2026

View reviewed changes

Comment thread python/cuvs_bench/cuvs_bench/backends/registry.py Outdated

afourniernv added 3 commits March 20, 2026 09:39

Fix bulk indexing format for Elasticsearch

80c104f

Use single-doc format (_index, _id, vector_field) instead of two-part NDJSON (index action + source) so ES accepts the bulk request. Signed-off-by: Alex Fournier <afournier@nvidia.com>

Add run_build, run_search, run_benchmark API for elastic backend

7b25521

Expose ELASTIC constant and convenience wrappers for build-only, search-only, or full benchmark runs. Signed-off-by: Alex Fournier <afournier@nvidia.com>

afourniernv force-pushed the fea-1856-cuvs-lucene-backend branch from ec0b3e3 to 7b25521 Compare March 20, 2026 16:45

cjnolet reviewed Mar 25, 2026

View reviewed changes

Comment thread python/cuvs_bench/cuvs_bench/tests/integration/conftest.py Outdated

cjnolet reviewed Mar 25, 2026

View reviewed changes

Comment thread python/cuvs_bench/cuvs_bench/backends/elasticsearch.py

cjnolet reviewed Apr 9, 2026

View reviewed changes

Comment thread python/cuvs_bench_elastic/pyproject.toml Outdated

jrbourbeau mentioned this pull request Apr 10, 2026

[REVIEW] Add OpenSearch backend to cuvs-bench #2012

Merged

afourniernv added 2 commits April 12, 2026 18:27

Revert unrelated change to get_dataset/__main__.py

087289d

afourniernv force-pushed the fea-1856-cuvs-lucene-backend branch from dca9c6f to 087289d Compare April 13, 2026 01:46

afourniernv added 2 commits April 13, 2026 13:49

cjnolet reviewed Apr 16, 2026

View reviewed changes

Comment thread python/cuvs_bench/cuvs_bench/backends/elasticsearch.py Outdated

cjnolet reviewed Apr 16, 2026

View reviewed changes

Comment thread python/cuvs_bench/cuvs_bench/backends/elasticsearch.py Outdated

cjnolet reviewed Apr 16, 2026

View reviewed changes

jnke2016 reviewed Apr 27, 2026

View reviewed changes

Comment thread python/cuvs_bench/cuvs_bench/backends/elasticsearch.py Outdated

jnke2016 reviewed Apr 27, 2026

View reviewed changes

Comment thread python/cuvs_bench/cuvs_bench/backends/elasticsearch.py Outdated

Merge branch 'main' into pr1907-reconcile

94a9199

# Conflicts: # python/cuvs_bench/cuvs_bench/orchestrator/orchestrator.py # python/cuvs_bench/cuvs_bench/tests/test_registry.py

jnke2016 reviewed Jun 25, 2026

View reviewed changes

josephine-wolf-oberholtzer added this to Unstructured Data Processing Jul 1, 2026

josephine-wolf-oberholtzer moved this to In Progress in Unstructured Data Processing Jul 1, 2026

afourniernv added 2 commits July 13, 2026 07:37

Merge remote-tracking branch 'origin/main' into pr1907-reconcile

075a8a8

Remove unused elastic vector loader wrappers

a0a4b67

afourniernv requested review from a team as code owners July 13, 2026 13:50

afourniernv added 10 commits July 13, 2026 09:55

Remove redundant elastic config loader override

fe34fd4

Use elastic index config as build param source

1b81f44

Remove elastic benchmark convenience wrappers

fcc13f5

Fix elastic test registration fallback

cb5b0fd

Align elastic benchmark dependency metadata

8e03291

Normalize elastic algorithm variants in loader

da8aafa

Load split plugin entry points across registry groups

3f80d6b

Route elastic recall through orchestrator

e30bdd1

Clean up elastic loader wrapper tests

df0cb52

Fix elastic pytest regressions

30d85e6

afourniernv requested a review from jnke2016 July 15, 2026 21:45

jnke2016 reviewed Jul 16, 2026

View reviewed changes

jnke2016 approved these changes Jul 16, 2026

View reviewed changes

afourniernv added 3 commits July 20, 2026 08:18

Merge remote-tracking branch 'origin/main' into pr1907-reconcile

edae65e

Address final elastic review nits

4df7c9c

Merge remote-tracking branch 'origin/main' into pr1907-reconcile

1c1a384

		# ── Convenience API ───────────────────────────────────────────────────────────


		def run_build(

		from .config_loaders import DatasetConfig


		def _should_compute_recall(result: SearchResult) -> bool:

Uh oh!

Conversation

afourniernv commented Mar 10, 2026

Uh oh!

copy-pr-bot Bot commented Mar 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjnolet commented Apr 16, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnke2016 left a comment

Choose a reason for hiding this comment

Uh oh!

afourniernv commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

afourniernv commented Jul 21, 2026 •

edited

Loading