Repository navigation
Extend RegexExtractEqQuery to fast JSON subfields - #6848
Merged
Merged
Conversation
EQ(REGEXP_EXTRACT(...)) on a fast JSON subfield, including dynamic fields, is now evaluated once per distinct value by walking the subfield's string column dictionary. The postings shortcut stays disabled for JSON subfields: their terms share the JSON field's inverted index with every other path, so term ordinals are not column value ordinals. Rename `terms_are_fast_field_values` to `can_use_postings`: the schema only guarantees that terms are a subset of the fast-field values, and the scorer checks per segment whether both dictionaries are equal before using postings. Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
Mallets
marked this pull request as ready for review
September 30, 2026 09:27
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Mallets
added this pull request to stack #6853
September 30, 2026 12:46
fulmicoton
approved these changes
Sep 30, 2026
fulmicoton-dd
approved these changes
Sep 30, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
EQ(REGEXP_EXTRACT(field, pattern, i), "literal")on a fast JSON subfield, including names that resolve to_dynamicsuch ascustom.programName, used to fall back to the JIT predicate, which runs the regex once per document. It is now evaluated once per distinct value: the scorer walks the subfield's string column dictionary with the FST prefilter, checks each accepted value exactly, and selects documents by their first value, matching the JIT semantics. The cost goes from one regex evaluation per document to one per distinct value, plus one ordinal comparison per document.The scorer opens the column with
fast_fields().str(name)and the JIT withdynamic_column_handles(name). Both use the same name and resolve it the same way, including the_dynamicfallback. The JIT only acceptsStrcolumns as string inputs, so both paths read the same column even when the path also holds other types.The postings shortcut stays disabled for JSON subfields. Their terms share the JSON field's inverted index with every other path and type, so term ordinals are not column value ordinals. No postings prefilter is added to warmup for them.
This PR also renames
terms_are_fast_field_valuestocan_use_postings. The schema only guarantees that terms are a subset of the fast-field values (the raw tokenizer may drop long values), and the scorer checks per segment that both dictionaries have the same size before using postings. The old name claimed the stronger property.How was this PR tested?
test_regex_extract_eq_matches_json_subfieldandtest_regex_extract_eq_matches_dynamic_field(production shapecustom.programName, capture 3) cover arrays (only the first value counts), a number stored at the same path, the same value under another path, and nested and missing paths. Each asserts that the optimized query matches the JIT count and adds no postings prefilter.test_calc_field_regex_extract_eq_warmup_on_dynamic_fieldchecks that warmup loads thecustom.programNamefast column and no postings automaton.cargo nextest run -p quickwit-query regex_extract calc_fieldandcargo nextest run -p quickwit-doc-mapper calc_field.Follow-up
#6849 removes the per-document scan for JSON subfields: it walks the JSON field's terms under the path prefix, uses their postings when the matching terms and matching column values have the same count, and warms the path-prefixed prefilter.