[Enhancement] Fulltext index BloomFilter pushdown pre-filter by ck89119 · Pull Request #23833 · matrixorigin/matrixone

ck89119 · 2026-03-10T07:26:03Z

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

Implement BloomFilter pushdown for fulltext index tables, similar to the existing IVF index pushdown.

When a fulltext search query has additional filter predicates (e.g. WHERE match(...) against(...) AND category = 'tech'), the filtered PKs are collected into a BloomFilter and pushed down to the fulltext index table reader to skip irrelevant rows early.

current applyJoinFullTextIndices's query plan：

Project
  -> Sort (score DESC)
    -> Join (src.pk = ft_alias.doc_id)
      -> Table Scan on src (with filter)
      -> Table Function fulltext_index_scan

When there is a filter, the doc_id returned by fulltext_index_scan can be massively selected for filters, resulting in wastes significant I/O.

optimized query plan(with filter)

Project
  -> Sort (score DESC, LIMIT)
    -> Join INNER (src.pk = ft_alias.doc_id)  [runtime filter: IN-list -> scanNode]
      -> Table Scan on src (with filter)
      -> Join INNER (ft_alias.doc_id = second.pk)  [runtime filter: BloomFilter -> ft_func]
        -> Table Function fulltext_index_scan
        -> Project (pk only)
          -> Table Scan on src (with filter, copy)

Key changes:

plan (apply_indices_fulltext.go): applyJoinFullTextIndices builds RuntimeFilterBuildList with BloomFilter for the inner join between fulltext results and the source table.
compile (scope.go, remoterun.go): Pass BloomFilter from context to FilterHint when the target table is a fulltext index table (TableType == "fulltext").
reader (reader.go): tryUpdateColumns uses doc_id column for BF filtering on fulltext tables, independent of pkPos since internal SQL does not include __mo_fake_pk_col.
pk_filter_mem (pk_filter_mem.go): NewMemPKFilter sets BFSeqNum using doc_id column for fulltext tables as fallback when __mo_index_pri_col is absent.
build_ddl (build_ddl.go): Fix fulltext index table relkind from SystemIndexRel ("i") to FullTextIndex_TblType ("fulltext") so the TableType check matches at runtime.

Bug fixes included:

relkind mismatch: buildFullTextIndexTable wrote "i" to mo_tables.relkind instead of "fulltext", causing BF context propagation to be skipped.
pkPos gate: BF filtering logic was inside if pkPos != -1 block, but fulltext internal SQL never includes the PK column.
BFSeqNum not set: NewMemPKFilter only looked for __mo_index_pri_col to set BFSeqNum; fulltext tables use doc_id instead.

Tests:

Unit tests for NewMemPKFilter with fulltext table BFSeqNum
Unit tests for tryUpdateColumns with fulltext BF filtering
BVT: test/distributed/cases/fulltext/fulltext_pushdown.sql with 7 scenarios

@xxx

…rigin#23832) Implement BloomFilter pushdown for fulltext index tables, similar to the existing IVF index pushdown. When a fulltext search query has additional filter predicates (e.g. WHERE match(...) against(...) AND category = 'tech'), the filtered PKs are collected into a BloomFilter and pushed down to the fulltext index table reader to skip irrelevant rows early. Key changes: - plan: applyJoinFullTextIndices builds RuntimeFilterBuildList with BF - compile: scope.go/remoterun.go pass BF from context to FilterHint - reader: tryUpdateColumns uses doc_id column for BF filtering on fulltext tables (independent of pkPos since internal SQL doesn't include __mo_fake_pk_col) - pk_filter_mem: NewMemPKFilter sets BFSeqNum using doc_id column for fulltext tables (fallback when __mo_index_pri_col is absent) - build_ddl: fix fulltext index table relkind from SystemIndexRel to FullTextIndex_TblType so TableType check matches at runtime Approved by: @xxx Ref matrixorigin#23832

…lkind 1. Apply BloomFilter probe to all fulltext table function nodes instead of only the first one when multiple fulltext_match expressions exist. 2. Add IsFullTextIndexTableType() to handle legacy fulltext index tables whose relkind is still 'i' (SystemIndexRel) instead of 'fulltext', so BF pushdown works for pre-upgrade tables as well.

mergify · 2026-03-23T07:19:56Z

Merge Queue Status

✅ Entered queue — 2026-03-23 07:19 UTC · Rule: main
✅ Checks started · in-place · dashboard
🟠 Running checks
⏳ Merge · ETA: 2026-03-23 08:15 UTC 🚀

Required conditions to merge

Required conditions to stay in the queue

ck89119 requested review from XuPeng-SH, aunjgr, fengttt, heni02 and ouyuanning as code owners March 10, 2026 07:26

ck89119 had a problem deploying to ci March 10, 2026 07:26 — with GitHub Actions Error

ck89119 temporarily deployed to ci March 10, 2026 07:26 — with GitHub Actions Inactive

ck89119 had a problem deploying to ci March 10, 2026 07:26 — with GitHub Actions Error

ck89119 temporarily deployed to ci March 10, 2026 07:26 — with GitHub Actions Inactive

matrix-meow added the size/L Denotes a PR that changes [500,999] lines label Mar 10, 2026

Merge branch 'main' into fulltext-pushdown-23832

7fe2143

mergify bot added kind/feature kind/enhancement kind/test-ci labels Mar 10, 2026

mergify bot had a problem deploying to ci March 10, 2026 07:27 Failure

mergify bot had a problem deploying to ci March 10, 2026 07:27 Error

mergify bot temporarily deployed to ci March 10, 2026 07:27 Inactive

mergify bot had a problem deploying to ci March 10, 2026 07:27 Error

mergify bot had a problem deploying to ci March 10, 2026 07:27 Failure

mergify bot temporarily deployed to ci March 10, 2026 07:27 Inactive

ck89119 had a problem deploying to ci March 13, 2026 09:09 — with GitHub Actions Error

ck89119 temporarily deployed to ci March 13, 2026 09:09 — with GitHub Actions Inactive

ck89119 had a problem deploying to ci March 13, 2026 09:09 — with GitHub Actions Error

Merge branch 'main' into fulltext-pushdown-23832

33aa532

ck89119 temporarily deployed to ci March 13, 2026 09:11 — with GitHub Actions Inactive

ck89119 temporarily deployed to ci March 13, 2026 09:12 — with GitHub Actions Inactive

ck89119 requested a review from cpegeric March 13, 2026 09:52

cpegeric approved these changes Mar 13, 2026

View reviewed changes

heni02 approved these changes Mar 13, 2026

View reviewed changes

aunjgr approved these changes Mar 17, 2026

View reviewed changes

ck89119 temporarily deployed to ci March 19, 2026 06:49 — with GitHub Actions Inactive

Merge branch 'main' into fulltext-pushdown-23832

8c109c4

XuPeng-SH approved these changes Mar 23, 2026

View reviewed changes

Merge branch 'main' into fulltext-pushdown-23832

ad9e720

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Fulltext index BloomFilter pushdown pre-filter#23833

[Enhancement] Fulltext index BloomFilter pushdown pre-filter#23833
ck89119 wants to merge 8 commits intomatrixorigin:mainfrom
ck89119:fulltext-pushdown-23832

ck89119 commented Mar 10, 2026 •

edited

Loading

Uh oh!

mergify bot commented Mar 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

ck89119 commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

Key changes:

Bug fixes included:

Tests:

Uh oh!

mergify bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ck89119 commented Mar 10, 2026 •

edited

Loading

mergify bot commented Mar 23, 2026 •

edited

Loading