Repository navigation
Add search_after cursor paging to the native search API - #6830
Open
oleksandr-zhyhalo wants to merge 1 commit into
Open
oleksandr-zhyhalo wants to merge 1 commit into
oleksandr-zhyhalo wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #3967. Related: #1544 (paging past the 10,000
start_offsetlimit) and #4078 (loading the documents around a hit with twosearch_afterqueries).The native search API accepts a
search_aftercursor, in the GET query string or the POST body, and returnscursors: one per hit, in the same order ashits, likesnippets. To get the next page, pass the last entry ofcursorswith the samequeryandsort_by. To page backward from any hit, pass its cursor and invert thesort_byorders (tsand-tsswap,-_docinverts the default order).A cursor holds the hit's
PartialHit(sort values, split ID, segment, doc ID), encoded as protobuf and URL-safe base64 without padding, about 50 characters.SearchAfterCursorimplementsDisplay/FromStr, likeScrollKeyAndStartOffset. Two choices differ from the sketch in #3967:cursorshas one entry per hit. A parallel array keeps the response backward compatible, and loading the context of an arbitrary hit (Add a query to gets surroudnings documents #4078) needs that hit's cursor.SearchAfterCursoralone.Root search now validates
search_afterfor all APIs and resolves the type-validation TODO invalidate_sort_by_fields_and_search_after:_doc/_shard_doc, and a non-float value for_score. The leaf search used to panic on these and fail every split with a 500. You could trigger it by reusing a cursor with anothersort_by, or with an Elasticsearchsearch_after: [1]onsort: _score.sort_by, Quickwit sorts hits by_doc, so root rejects asearch_afterwithout a split ID, as it does for an explicit_docsort.search_aftervalues return a 400 instead of a 500.The docs cover two limitations. Quickwit changes hit addresses when it merges splits, so a merge between two requests can make a page skip or repeat the hits tied on the sort value, and any hit when you page without
sort_by. The scroll API stays the snapshot option. Root also rejects the cursor of a hit without a value for a sort field, through the existing sort value count check. Supporting those cursors needs a change in how the leaf compares missing values, which I left for a follow-up.The REST client response model gains
cursors, empty when an older server omits it. The PR also updates the docs and the changelog.How was this PR tested?
quickwit-search(223),quickwit-serve(168), andquickwit-rest-client(12). New tests cover the cursor round trip, a golden encoding, one cursor per hit, posivalidation, the precision guard, the REST parameter over GET and POST, and the utcursors.qw_search_api/0006_search_after.yaml: forward paging through a three-hit tie, backward paging withts/-tsand-_doc, the default sort, a reused cursor, invalid cursors, and the precision guard.qw_search_apiandes_compatibilityREST suites against a local build: all 58 files pass, with no leaf panics in the server log.make fmt,cargo clippy --tests -- -D warningson the changed crates, and -tests`.