[feat](eval_performance): support filtered HGraph workloads in AutoTune

[jac0626]

Change Type

• Bug fix
• New feature
• Improvement/Refactor
• Documentation
• CI/Build/Infra

Linked Issue

• Fixes: [feat](eval_performance): support filtered HGraph workloads in AutoTune #2735

What Changed

• Add per-query FilterPtr and invalid BitsetPtr workload inputs to typed HGraph TuneSearch and TuneIndex requests.
• Route filtered workloads through the shared evaluation core while preserving HGraph's native bitset search path.
• Validate filter shape, mutual exclusion, supported index scope, and unsupported extra-info filtering.
• Add separate FilterPtr and BitsetPtr examples, focused regression tests, and English/Chinese user documentation.

Test Evidence

• make fmt
• make lint
• make test
• make cov, run tests, and collect coverage
• Other (describe below)

Test details:

clang-format-15 --dry-run --Werror <all changed C++ files>
cmake --build build --target autotune_test eval_dataset_test \
  327_feature_autotune_existing_index \
  330_feature_autotune_existing_index_bitset -j$(nproc)

./build/tools/autotune/autotune_test \
  "AutoTune evaluates per-query filters and bitsets for HGraph"
  All tests passed (29 assertions in 1 test case)

./build/tools/eval/eval_dataset_test \
  "EvalDataset builds a query-only view for id recall"
  All tests passed (21 assertions in 1 test case)

./build/tools/eval/eval_dataset_test \
  "EvaluateSearch validates inputs and propagates search errors"
  All tests passed (23 assertions in 1 test case)

./build/examples/cpp/327_feature_autotune_existing_index
  recall_at_k: 1.0; first neighbor id: 1000

./build/examples/cpp/330_feature_autotune_existing_index_bitset
  recall_at_k: 1.0; first neighbor id: 1000

Compatibility Impact

• API/ABI compatibility: Additive fields in the build-tree AutoTune API under tools/; no installed VSAG SDK API is changed.
• Behavior changes: Typed HGraph tuning requests may now evaluate per-query filters. Filtered requests for other index types fail validation explicitly.

Performance and Concurrency Impact

• Performance impact: The unfiltered evaluation path is unchanged. Filtered trials use the existing native HGraph filter/bitset search overloads.
• Concurrency/thread-safety impact: The same per-query filters or bitsets may be read by concurrent search workers and therefore must remain immutable and thread-safe during tuning.

Documentation Impact

• No docs update needed
• Updated docs:
  • README.md
  • DEVELOPMENT.md
  • CONTRIBUTING.md
  • Other: docs/docs/{en,zh}/src/resources/autotune.md

Risk and Rollback

• Risk level: medium
• Rollback plan: Revert commit 92dfeaa4; unfiltered AutoTune behavior is otherwise unchanged.

Checklist

• I have linked the relevant issue (required for kind/bug and kind/feature; see "Linked Issue" above)
• I have added/updated tests for new behavior or bug fixes
• I have considered API compatibility impact
• I have updated docs if behavior/workflow changed
• My commit messages follow project conventions (Conventional Commits, optional [skip ci] prefix)

[vsag-bot]

/label status/waiting-for-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao

[mergify]

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

• label~=^kind/

🟢 Require version label

• label~=^version/

🟢 Require linked issue for feature/bug PRs

• body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

[Copilot]

Pull request overview

This PR extends the eval_performance AutoTune typed API to support filtered HGraph workloads by accepting per-query FilterPtr or exclusion BitsetPtr inputs, routing them through the shared evaluation pipeline while keeping HGraph’s native filtered-search overloads, and documenting the new capability.

Changes:

• Extend typed AutoTune workload/request plumbing to carry per-query FilterPtr / BitsetPtr into EvalDataset, validation, and evaluation.
• Update search evaluation to apply per-query filters/bitsets, and restrict filtered workloads to HGraph (including rejecting use_extra_info_filter=true).
• Add regression tests, new bitset example, and English/Chinese documentation updates.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tools/eval/eval_dataset.h	Adds per-query filter/bitset storage + accessors and filtered-query accounting.
tools/eval/eval_dataset.cpp	Validates filter/bitset inputs and wires them into dataset construction.
tools/eval/eval_dataset_test.cpp	Adds coverage for per-query filters/bitsets and input validation.
tools/eval/case/search_eval_case.cpp	Applies per-query filter/bitset during KNN evaluation via HGraph overloads.
tools/autotune/autotune.h	Extends Workload with query_filters and query_invalid_bitsets.
tools/autotune/autotune.cpp	Propagates filtered workload data into eval datasets; enforces HGraph-only for filtered cases; reports filtered count.
tools/autotune/autotune_internal.h	Tracks filtered-workload presence in request context.
tools/autotune/autotune_candidate.cpp	Rejects extra-info filtering when tuning filtered workloads.
tools/autotune/autotune_test.cpp	Adds end-to-end tuning tests for per-query filters/bitsets and validation failures.
examples/cpp/CMakeLists.txt	Registers a new bitset-based AutoTune example target.
examples/cpp/327_feature_autotune_existing_index.cpp	Extends existing example to demonstrate per-query FilterPtr tuning and usage.
examples/cpp/330_feature_autotune_existing_index_bitset.cpp	New example demonstrating per-query exclusion bitsets in AutoTune + final search.
docs/docs/zh/src/resources/autotune.md	Documents filtered typed HGraph workloads and links both examples (FilterPtr + bitset).
docs/docs/en/src/resources/autotune.md	Documents filtered typed HGraph workloads and links both examples (FilterPtr + bitset).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jac0626]

/retest

[LHT129]

[suggestion] The bitset recall test verifies recall_avg == 1.0 correctly, but after setting config.enable_recall = false; config.enable_qps = true; on this line, no corresponding EvaluateSearch call is made for the bitset_dataset. The test immediately falls through to the ef_search=0 error-path test using the original dataset (filter-based). This means the bitset path is never exercised under a QPS-only configuration.

If the intent was to also verify the bitset workload works correctly under QPS measurement, add a call like:

const auto bitset_qps = vsag::eval::EvaluateSearch(index, bitset_dataset, config);
REQUIRE(bitset_qps.contains("qps"));

Otherwise, remove the config.enable_qps = true; line to avoid leaving dead state that suggests an incomplete test.

[LHT129]

[note] The index_request is constructed via fixture.Request(index_workspace.Get()) which initializes workload with the unfiltered ground_truth and a hardcoded top_k=3. The test then immediately overwrites ground_truth with filtered_ground_truth and query_invalid_bitsets. This works because TOP_K (3) coincidentally matches the Request() default, but the dependency on the unfiltered ground truth being set first and then overwritten is fragile.

Consider adding a dedicated factory method (e.g. RequestFiltered) or constructing the IndexRequest directly with the filtered workload fields to make the intent explicit and avoid relying on the unfiltered defaults.

[LHT129]

[note] CheckValid performs a linear scan (std::find) over valid_ids_ on every invocation. For the test fixture this is fine since valid_ids_ is small (size = TOP_K = 3), but if this filter class is ever copied into production code or used with larger allow-lists, the O(n) per-ID check would become a hot-path bottleneck. Consider adding a comment noting this is test-only, or using an unordered_set for the allow-list lookup to make the intent clearer.

[LHT129]

Thanks for this PR. The implementation is clean and well-structured — the filter/bitset validation, mutual exclusion checks, and index-type gating are all correctly placed. The test coverage is thorough with both positive and negative paths.

I left 3 inline comments:

1. [suggestion] tools/eval/eval_dataset_test.cpp:556 — The bitset QPS test appears incomplete; config.enable_qps = true is set but never exercised for the bitset dataset.

2. [note] tools/autotune/autotune_test.cpp:1023 — The filtered index_request construction reuses fixture.Request() which sets unfiltered defaults that are immediately overwritten. Consider a dedicated factory to make the intent clearer.

3. [note] tools/autotune/autotune_test.cpp:320 — CountingAllowListFilter::CheckValid uses std::find (O(n)). Fine for the test fixture but worth a comment noting it is test-only.

Overall the PR is in good shape. The core logic (validation, routing, candidate gating) is correct and the documentation updates are thorough.

[LHT129]

[suggestion] GetQueryFilter and GetQueryInvalidBitset access the member vectors via operator[] without bounds checking on query_id. While validate_query_filter_inputs guarantees the vector size equals the query count at construction time, a future code path that constructs an EvalDataset without going through the factory methods could bypass this invariant. Consider adding an assertion (e.g. assert(query_id < query_filters_.size())) or using .at() for bounds-checked access in debug builds.

Current:

return query_filters_.empty() ? no_filter : query_filters_[query_id];

Suggested:

if (query_filters_.empty()) return no_filter;
assert(query_id < query_filters_.size());
return query_filters_[query_id];

(Same for GetQueryInvalidBitset.)

[LHT129]

[note] The extra_info_request test at line 1014 only exercises the use_extra_info_filter rejection path when query_invalid_bitsets is set (inherited from search_request after the bitset test above). The rejection logic in autotune_candidate.cpp:423 does not differentiate between filter types, so the behavior is identical for both paths, but testing the filter path as well would improve coverage symmetry.

Consider adding a similar test with query_filters set instead of query_invalid_bitsets, or adding a brief comment noting that the rejection is type-agnostic.

[LHT129]

[suggestion] The use_extra_info_filter rejection for filtered workloads is validated inside generate_candidates (autotune_candidate.cpp:422-426), which runs during the "candidate_generation" stage. For TuneSearch with a fixed parameter space, this means the error is reported as a candidate_generation failure rather than a validation failure.

In contrast, ParseRequest(SearchRequest) already validates that the index type is HGraph when filters are present (autotune.cpp:581-582), and ParseRequest(IndexRequest) validates index type at parse time (autotune.cpp:558).

Consider adding a use_extra_info_filter check inside ParseRequest(SearchRequest) as well, so the error is caught during the "validation" stage and reported consistently. The check in generate_candidates can remain as a defense-in-depth measure for the TuneIndex path where the parameter space is expanded from ranges.

[LHT129]

[note] The uses_extra_info_filter check inside generate_candidates (autotune_candidate.cpp:422) throws inside the expand callback. For TuneIndex with parameter ranges, this means the exception is thrown during cartesian-product expansion. If the parameter space contains multiple combinations and use_extra_info_filter appears only in some of them, the exception won't trigger until that specific combination is expanded — earlier combinations without the flag will have already been added to the candidate list before the throw. The partial candidates are discarded when the exception propagates (caught by run_tuning_locked), so correctness is preserved, but the error message could be confusing if a user has a mixed parameter space where only some combinations set the flag.

This is consistent with how other candidate-generation errors work (e.g. duplicate candidates are silently skipped), so no change is required. Just noting the behavior for future reference.