Evaluate EQ(REGEXP_EXTRACT) once per distinct value with RegexExtractEqQuery (#6819)

* Rewrite EQ(REGEXP_EXTRACT) predicates to RegexQuery when eligible

Lower a narrow calculated-field equality over REGEXP_EXTRACT into an
inverted-index RegexQuery for fast, indexed, raw string fields, so
search can avoid per-document JIT evaluation while preserving warmup.

* Allow unanchored EQ(REGEXP_EXTRACT) rewrite with swapped EQ args

Honor optional ^/$ by wrapping missing sides with .* and treat EQ as
commutative so literal/extract order does not block the RegexQuery path.

* Prefilter EQ(REGEXP_EXTRACT) with FST RegexQuery, keep exact JIT

Intersect an over-approximating RegexQuery with the original CalcField
predicate so unanchored log-style extracts stay semantically exact while
the FST path narrows candidates.

* Skip JIT for fully anchored EQ(REGEXP_EXTRACT) rewrites

When the extract pattern has both ^ and $, the FST RegexQuery alone is
equivalent, so drop the CalcField intersection; keep prefilter∧JIT otherwise.

* Always prefilter EQ(REGEXP_EXTRACT) with FST RegexQuery

Drop the Exact rewrite for fully anchored patterns: leftmost extract
semantics are not always equivalent to an FST whole-term match, so keep
CalcField as the exact filter and add a Criterion microbench.

* Parse REGEXP_EXTRACT pattern with regex-syntax for prefilter rewrite

Replace manual parenthesis scanning in substitute_single_capture with a
regex-syntax AST walk, and add a prefilter_only path to the bench to
separate FST cost from JIT cost.

* Fix soundness of EQ(REGEXP_EXTRACT) FST prefilter

- Require the capture group to be a direct part of the top-level concatenation.
- Wrap unanchored sides with (?s:.*) so multi-line values are not dropped.
- Reject fields whose fast field is normalized.
- Keep patterns whose rewrite needs look-arounds on the JIT path.

* Evaluate EQ(REGEXP_EXTRACT) on the fast-field dictionary with exact JIT semantics

Replace the term-dictionary rewrite with RegexExtractEqQuery, which walks the
fast-field dictionary with the FST prefilter, checks the capture on each
candidate value, and selects documents by the ordinal of their first value.
This matches the JIT predicate on multi-valued documents, on values longer
than MAX_TOKEN_LEN, and on normalized fast fields.

Documents are collected with range scans when the matching ordinals form a
few runs, with block decoding on full single-valued columns, and with a
per-document first-value loop otherwise.

Drop the regexp_extract_eq benchmark.

* Simplify RegexExtractEqQuery to a first-value document loop

Drop the range-scan and block-decoding strategies: collect matching
documents with a single loop over first-value ordinals. Move the
RegexExtractEqQuery end-to-end tests next to the query, document that
column errors are returned rather than treated as an absent column, and
use `regex_extract_eq` naming consistently.

* Read postings for EQ(REGEXP_EXTRACT) on raw-indexed fields

When the field is also indexed with the raw tokenizer and every matching
fast-field value is shorter than the raw tokenizer's length limit, visit
only the documents in the postings of the matching terms, still keeping a
document only when its first fast-field value matches. Otherwise fall
back to the first-value loop over every document.

Warm the field's term dictionary and postings with the prefilter regex.

On 10M-document low-cardinality corpora this takes 1.2-1.5ms instead of
19ms. When the prefilter accepts most of a high-cardinality dictionary,
walking the term dictionary a second time doubles the cost.

* Skip the first-value check on single-valued postings of EQ(REGEXP_EXTRACT)

A document of a full or optional column holds only the value of the term
whose postings contain it, and that value matches.

* Keep tantivy-common on the same rev as tantivy

Main moved tantivy to e229de6, which left two tantivy-common versions and
broke BitSetDocSet::from(BitSet) in RegexExtractEqQuery.

* Walk only the term dictionary when it holds every fast-field value

* Support any capture index in REGEXP_EXTRACT equality

* Clarify regex extract query documentation

* Clarify and simplify regex extract planning

* Rewrite REGEXP_EXTRACT capture isolation via regex-syntax AST

Avoid byte-offset string splicing and keep prefilters correct under multiline and verbose flags.

* Simplify verbose-flag wrapping in capture substitution

* Cache REGEXP_EXTRACT query regexes

* Remove unused search FST dependency

Author: Mallets

quickwit/Cargo.lock

@@ -386,6 +386,8 @@ tantivy = { git = "https://github.com/quickwit-oss/tantivy/", rev = "e229de6db74
   "zstd-compression",
   "columnar-zstd-compression",
 ] }
+# Must stay on the same rev as `tantivy`.
+tantivy-common = { git = "https://github.com/quickwit-oss/tantivy/", rev = "e229de6db748c6b21747a99a18220446e76104e1" }
 tantivy-fst = "0.5"
 
 # This is actually not used directly the goal is to fix the version

quickwit/Cargo.toml

@@ -380,6 +380,18 @@ impl<'a, 'b: 'a> QueryAstVisitor<'a> for ExtractPrefixTermRanges<'b> {
         );
         Ok(())
     }
+
+    fn visit_calc_field(&mut self, calc_field_query: &'a CalcFieldQuery) -> Result<(), Self::Err> {
+        let Some(regex_query) = calc_field_query.try_prefilter_regex_query(self.schema) else {
+            return Ok(());
+        };
+        let resolved = regex_query.to_resolved(self.schema, Some(self.tokenizer_manager))?;
+        self.add_automaton(
+            resolved.field,
+            Automaton::Regex(resolved.json_path, resolved.regex),
+        );
+        Ok(())
+    }
 }
 
 type TermRangeWarmupInfo = HashMap<Field, HashMap<TermRange, PositionNeeded>>;
@@ -400,7 +412,7 @@ fn extract_prefix_term_ranges_and_automaton(
 
 #[cfg(test)]
 mod test {
-    use std::collections::HashSet;
+    use std::collections::{HashMap, HashSet};
     use std::ops::Bound;
 
     use quickwit_common::shared_consts::FIELD_PRESENCE_FIELD_NAME;
@@ -413,11 +425,14 @@ mod test {
     };
     use tantivy::Term;
     use tantivy::jitexpr::ast::deserialize;
-    use tantivy::schema::{DateOptions, DateTimePrecision, FAST, INDEXED, STORED, Schema, TEXT};
+    use tantivy::schema::{
+        DateOptions, DateTimePrecision, FAST, Field, INDEXED, STORED, STRING, Schema, TEXT,
+    };
 
     use super::{ExtractPrefixTermRanges, build_query};
     use crate::{
-        DYNAMIC_FIELD_NAME, FastFieldWarmupInfo, SOURCE_FIELD_NAME, TermRange, WarmupInfo,
+        Automaton, DYNAMIC_FIELD_NAME, FastFieldWarmupInfo, SOURCE_FIELD_NAME, TermRange,
+        WarmupInfo,
     };
 
     fn calc_field(expression: &str) -> QueryAst {
@@ -488,6 +503,64 @@ mod test {
         );
     }
 
+    #[test]
+    fn test_calc_field_regex_extract_eq_warmup() {
+        let mut schema_builder = Schema::builder();
+        let service = schema_builder.add_text_field("service", STRING | FAST);
+        schema_builder.add_text_field("tokenized", TEXT | FAST);
+        schema_builder.add_text_field("fast_only", FAST);
+        let schema = schema_builder.build();
+        let context = BuildTantivyAstContext::for_test(&schema);
+
+        // The predicate always reads the fast field. It may also read postings of terms accepted
+        // by the prefilter when the field uses the raw tokenizer for both indexing and fast fields.
+        for (expression, fast_field, expected_prefilter) in [
+            (
+                r#"(EQ (REGEXP_EXTRACT service "^svc-([a-z]+)-prod$" 1u64) "api")"#,
+                "service",
+                Some("svc-api-prod"),
+            ),
+            (
+                r#"(EQ (REGEXP_EXTRACT service "svc-([a-z]+)-prod" 1u64) "api")"#,
+                "service",
+                Some("(?s:.*)svc-api-prod(?s:.*)"),
+            ),
+            (
+                r#"(EQ (REGEXP_EXTRACT tokenized "^svc-([a-z]+)-prod$" 1u64) "api")"#,
+                "tokenized",
+                None,
+            ),
+            (
+                r#"(EQ (REGEXP_EXTRACT fast_only "^svc-([a-z]+)-prod$" 1u64) "api")"#,
+                "fast_only",
+                None,
+            ),
+            (
+                r#"(EQ (REGEXP_EXTRACT service "(?m)^svc-([a-z]+)" 1u64) "api")"#,
+                "service",
+                None,
+            ),
+        ] {
+            let (_, warmup) = build_query(calc_field(expression), &context, None).unwrap();
+            assert_eq!(
+                warmup.fast_fields,
+                expected_fast_fields(&[fast_field]),
+                "{expression}"
+            );
+            let expected_automatons: HashMap<Field, HashSet<Automaton>> = expected_prefilter
+                .map(|regex| {
+                    let automaton = Automaton::Regex(None, regex.to_string());
+                    (service, HashSet::from([automaton]))
+                })
+                .into_iter()
+                .collect();
+            assert_eq!(
+                warmup.automatons_grouped_by_field, expected_automatons,
+                "{expression}"
+            );
+        }
+    }
+
     fn full_text_query_for_warmup(field: &str, tokenizer: Option<&str>) -> QueryAst {
         quickwit_query::query_ast::FullTextQuery {
             field: field.to_string(),

quickwit/quickwit-doc-mapper/src/query_builder.rs

@@ -15,11 +15,14 @@ anyhow = { workspace = true }
 base64 = { workspace = true }
 bitpacking = { workspace = true }
 hex = { workspace = true }
+lru = { workspace = true }
 regex = { workspace = true }
+regex-syntax = { workspace = true }
 serde = { workspace = true }
 serde_json = { workspace = true }
 serde_with = { workspace = true }
 tantivy = { workspace = true }
+tantivy-common = { workspace = true }
 unicode-segmentation = { workspace = true }
 tantivy-fst = { workspace = true }
 tracing = { workspace = true }

quickwit/quickwit-query/Cargo.toml

@@ -12,7 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use criterion::{Criterion, Throughput, black_box, criterion_group, criterion_main};
+use std::hint::black_box;
+
+use criterion::{Criterion, Throughput, criterion_group, criterion_main};
 use quickwit_query::CodeTokenizer;
 use tantivy::tokenizer::{RegexTokenizer, TextAnalyzer, Token, TokenStream};