Skip to content

Add search_after cursor paging to the native search API - #6830

Open
oleksandr-zhyhalo wants to merge 1 commit into
quickwit-oss:mainfrom
rootprint:native-search-after-cursors
Open

oleksandr-zhyhalo wants to merge 1 commit into
quickwit-oss:mainfrom
rootprint:native-search-after-cursors

Conversation

@oleksandr-zhyhalo

Copy link
Copy Markdown

Description

Closes #3967. Related: #1544 (paging past the 10,000 start_offset limit) and #4078 (loading the documents around a hit with two search_after queries).

The native search API accepts a search_after cursor, in the GET query string or the POST body, and returns cursors: one per hit, in the same order as hits, like snippets. To get the next page, pass the last entry of cursors with the same query and sort_by. To page backward from any hit, pass its cursor and invert the sort_by orders (ts and -ts swap, -_doc inverts the default order).

A cursor holds the hit's PartialHit (sort values, split ID, segment, doc ID), encoded as protobuf and URL-safe base64 without padding, about 50 characters. SearchAfterCursor implements Display/FromStr, like ScrollKeyAndStartOffset. Two choices differ from the sketch in #3967:

  • cursors has one entry per hit. A parallel array keeps the response backward compatible, and loading the context of an arbitrary hit (Add a query to gets surroudnings documents #4078) needs that hit's cursor.
  • Protobuf encoding keeps the cursor at a third of the JSON size, and a GET query string carries it without escaping. The format stays opaque, so a later change touches SearchAfterCursor alone.

Root search now validates search_after for all APIs and resolves the type-validation TODO in validate_sort_by_fields_and_search_after:

  • Root rejects a sort value with no sort field at its position, a value for _doc/_shard_doc, and a non-float value for _score. The leaf search used to panic on these and fail every split with a 500. You could trigger it by reusing a cursor with another sort_by, or with an Elasticsearch search_after: [1] on sort: _score.
  • Without sort_by, Quickwit sorts hits by _doc, so root rejects a search_after without a split ID, as it does for an explicit _doc sort.
  • Invalid datetime search_after values return a 400 instead of a 500.

The docs cover two limitations. Quickwit changes hit addresses when it merges splits, so a merge between two requests can make a page skip or repeat the hits tied on the sort value, and any hit when you page without sort_by. The scroll API stays the snapshot option. Root also rejects the cursor of a hit without a value for a sort field, through the existing sort value count check. Supporting those cursors needs a change in how the leaf compares missing values, which I left for a follow-up.

The REST client response model gains cursors, empty when an older server omits it. The PR also updates the docs and the changelog.

How was this PR tested?

  • Unit tests in quickwit-search (223), quickwit-serve (168), and quickwit-rest-client (12). New tests cover the cursor round trip, a golden encoding, one cursor per hit, posivalidation, the precision guard, the REST parameter over GET and POST, and the ut cursors.
  • New REST scenario qw_search_api/0006_search_after.yaml: forward paging through a three-hit tie, backward paging with ts/-ts and -_doc, the default sort, a reused cursor, invalid cursors, and the precision guard.
  • I ran the qw_search_api and es_compatibility REST suites against a local build: all 58 files pass, with no leaf panics in the server log.
  • make fmt, cargo clippy --tests -- -D warnings on the changed crates, and -tests`.

@oleksandr-zhyhalo
oleksandr-zhyhalo requested a review from a team as a code owner September 28, 2026 16:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

search_after on quickwit API

1 participant