Repository navigation
Evaluate EQ(REGEXP_EXTRACT) once per distinct value with RegexExtractEqQuery - #6819
Conversation
Lower a narrow calculated-field equality over REGEXP_EXTRACT into an inverted-index RegexQuery for fast, indexed, raw string fields, so search can avoid per-document JIT evaluation while preserving warmup.
Honor optional ^/$ by wrapping missing sides with .* and treat EQ as commutative so literal/extract order does not block the RegexQuery path.
Intersect an over-approximating RegexQuery with the original CalcField predicate so unanchored log-style extracts stay semantically exact while the FST path narrows candidates.
When the extract pattern has both ^ and $, the FST RegexQuery alone is equivalent, so drop the CalcField intersection; keep prefilter∧JIT otherwise.
Drop the Exact rewrite for fully anchored patterns: leftmost extract semantics are not always equivalent to an FST whole-term match, so keep CalcField as the exact filter and add a Criterion microbench.
| if let Some(regex_query) = calc_field_query.try_prefilter_regex_query(self.schema) { | ||
| return Ok(Some( | ||
| BoolQuery { | ||
| filter: vec![regex_query.into(), QueryAst::CalcField(calc_field_query)], |
There was a problem hiding this comment.
do we need to keep the Calcfield query at all?
There was a problem hiding this comment.
EQ(REGEXP_EXTRACT(field, pattern, 1), "L") is true only when the leftmost match of pattern yields capture group 1 equal to L. That is stricter than the term somehow contains prefix + L + suffix (later occurrences, longer captures, etc. can make containment true while extract+EQ is false).
So, RegexQuery acts as a pre-filter that over-estimates (it could match a non-first value) and then the JIT checks whether it's exactly the first match. JIT checks have been superseded by a dedicated scorer and posting list check in e15e411.
Replace manual parenthesis scanning in substitute_single_capture with a regex-syntax AST walk, and add a prefilter_only path to the bench to separate FST cost from JIT cost.
- Require the capture group to be a direct part of the top-level concatenation. - Wrap unanchored sides with (?s:.*) so multi-line values are not dropped. - Reject fields whose fast field is normalized. - Keep patterns whose rewrite needs look-arounds on the JIT path.
…IT semantics Replace the term-dictionary rewrite with RegexExtractEqQuery, which walks the fast-field dictionary with the FST prefilter, checks the capture on each candidate value, and selects documents by the ordinal of their first value. This matches the JIT predicate on multi-valued documents, on values longer than MAX_TOKEN_LEN, and on normalized fast fields. Documents are collected with range scans when the matching ordinals form a few runs, with block decoding on full single-valued columns, and with a per-document first-value loop otherwise. Drop the regexp_extract_eq benchmark.
Drop the range-scan and block-decoding strategies: collect matching documents with a single loop over first-value ordinals. Move the RegexExtractEqQuery end-to-end tests next to the query, document that column errors are returned rather than treated as an absent column, and use `regex_extract_eq` naming consistently.
When the field is also indexed with the raw tokenizer and every matching fast-field value is shorter than the raw tokenizer's length limit, visit only the documents in the postings of the matching terms, still keeping a document only when its first fast-field value matches. Otherwise fall back to the first-value loop over every document. Warm the field's term dictionary and postings with the prefilter regex. On 10M-document low-cardinality corpora this takes 1.2-1.5ms instead of 19ms. When the prefilter accepts most of a high-cardinality dictionary, walking the term dictionary a second time doubles the cost.
…RACT) A document of a full or optional column holds only the value of the term whose postings contain it, and that value matches.
Main moved tantivy to e229de6, which left two tantivy-common versions and broke BitSetDocSet::from(BitSet) in RegexExtractEqQuery.
Mallets
changed the title
Avoid byte-offset string splicing and keep prefilters correct under multiline and verbose flags.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 98d99cc4d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// contain the same values; callers still warm them for every raw-compatible segment. | ||
| /// | ||
| /// Returns `None` whenever the expression shape or field is not eligible. | ||
| pub fn try_prefilter_regex_query(&self, schema: &TantivySchema) -> Option<RegexQuery> { |
There was a problem hiding this comment.
with the following code, I think we end up compiling regex twice per split.
Once for the warmup, to identify what needs to be downloaded.
Once for building the query ast.
Before, it was done only once, because the regex compilation in the calculated field evaluation is done at jitexpr compilation and we have a jitexpr cache.
There was a problem hiding this comment.
You are right.
5643314 adds a fst and value expression cache following the LruCache approach in jitexpr.
3 participants
Summary
Calculated-field predicates such as
EQ(REGEXP_EXTRACT(field, pattern, capture_index), literal)currently run a JIT regex expression for every document. This PR addsRegexExtractEqQuery, which preserves JIT semantics while evaluating the regex once per distinct string value.The optimization applies to non-JSON string fast fields when the expression can produce a safe FST prefilter. It accepts either
EQargument order, every explicit capture index, and the two-argumentREGEXP_EXTRACTform, which defaults to capture index 0 (the whole match).Capture isolation
regex-syntaxrewrites the pattern before it reaches the existing fast path:(?:...)groups;Rust regexes have no backreferences, so anonymizing unrelated captures does not change which values match. The exact matcher then uses
captures().get(1)on the rewritten pattern, equivalent to the requested capture in the original pattern.Patterns without a safe FST prefilter, or with a selected capture nested in an alternation or repetition, remain on the JIT path.
Document selection
The scorer builds matching value ordinals with the FST prefilter and the exact regex check. A document matches only when its first fast-field value matches, like the JIT's
load_str_input. A matching later value in a multivalued field does not make the document match.The query builder registers the prefilter automaton for the postings path, so term-dictionary blocks and postings are warmed before the synchronous scorer reads them.
Benchmarks
Criterion bench on in-memory single-segment indexes, field
messagewithSTRING | FAST, comparing the JIT alone (current behavior), the inexact FST prefilter alone, the FST prefilter ∧ JIT andRegexExtractEqQuery. Hit counts were cross-checked against prefilter ∧ JIT for every corpus.RegexExtractEqQuerystatus="info"/"warn"/"error", extract =infostatus="info"/"warn"/"error", extract =errorRegexExtractEqQueryis exact and runs at the cost of the inexact prefilter alone: a single dictionary walk plus the postings of matching terms. The benchmark is not part of this PR.Test plan
make fmtandcargo clippy --tests.REGEXP_EXTRACTequality through the full root-to-leaf search path.