Skip to content

Extend RegexExtractEqQuery to fast JSON subfields - #6848

Merged
Mallets merged 3 commits into
mainfrom
mallets/calcfields-optimization-check
Sep 30, 2026
Merged

Mallets merged 3 commits into
mainfrom
mallets/calcfields-optimization-check

Conversation

@Mallets

@Mallets Mallets commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

EQ(REGEXP_EXTRACT(field, pattern, i), "literal") on a fast JSON subfield, including names that resolve to _dynamic such as custom.programName, used to fall back to the JIT predicate, which runs the regex once per document. It is now evaluated once per distinct value: the scorer walks the subfield's string column dictionary with the FST prefilter, checks each accepted value exactly, and selects documents by their first value, matching the JIT semantics. The cost goes from one regex evaluation per document to one per distinct value, plus one ordinal comparison per document.

The scorer opens the column with fast_fields().str(name) and the JIT with dynamic_column_handles(name). Both use the same name and resolve it the same way, including the _dynamic fallback. The JIT only accepts Str columns as string inputs, so both paths read the same column even when the path also holds other types.

The postings shortcut stays disabled for JSON subfields. Their terms share the JSON field's inverted index with every other path and type, so term ordinals are not column value ordinals. No postings prefilter is added to warmup for them.

This PR also renames terms_are_fast_field_values to can_use_postings. The schema only guarantees that terms are a subset of the fast-field values (the raw tokenizer may drop long values), and the scorer checks per segment that both dictionaries have the same size before using postings. The old name claimed the stronger property.

How was this PR tested?

  • test_regex_extract_eq_matches_json_subfield and test_regex_extract_eq_matches_dynamic_field (production shape custom.programName, capture 3) cover arrays (only the first value counts), a number stored at the same path, the same value under another path, and nested and missing paths. Each asserts that the optimized query matches the JIT count and adds no postings prefilter.
  • test_calc_field_regex_extract_eq_warmup_on_dynamic_field checks that warmup loads the custom.programName fast column and no postings automaton.
  • cargo nextest run -p quickwit-query regex_extract calc_field and cargo nextest run -p quickwit-doc-mapper calc_field.

Follow-up

#6849 removes the per-document scan for JSON subfields: it walks the JSON field's terms under the path prefix, uses their postings when the matching terms and matching column values have the same count, and warms the path-prefixed prefilter.

Mallets and others added 2 commits September 30, 2026 10:47
EQ(REGEXP_EXTRACT(...)) on a fast JSON subfield, including dynamic fields,
is now evaluated once per distinct value by walking the subfield's string
column dictionary. The postings shortcut stays disabled for JSON subfields:
their terms share the JSON field's inverted index with every other path, so
term ordinals are not column value ordinals.

Rename `terms_are_fast_field_values` to `can_use_postings`: the schema only
guarantees that terms are a subset of the fast-field values, and the scorer
checks per segment whether both dictionaries are equal before using postings.

Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
@Mallets
Mallets marked this pull request as ready for review September 30, 2026 09:27
@Mallets
Mallets requested a review from a team as a code owner September 30, 2026 09:27
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-30T09:33:32.364539Z 8bf1eab Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Mallets
Mallets added this pull request to stack #6853 September 30, 2026 12:46
@Mallets
Mallets added this pull request to the merge queue Sep 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 30, 2026
@Mallets
Mallets added this pull request to the merge queue Sep 30, 2026
Merged via the queue into main with commit 847550a Sep 30, 2026
15 checks passed
@Mallets
Mallets deleted the mallets/calcfields-optimization-check branch September 30, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants