Add search_after cursor paging to the native search API

CHANGELOG.md

@@ -9,11 +9,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - Azure Blob Storage: support custom endpoints via `endpoint` and `endpoint_suffix` configuration options for sovereign clouds (#6624)
+- Search API: cursor paging with the `search_after` parameter and the per-hit `cursors` of the response
 
 ### Fixed
 - (Jaeger) Query resource attributes when Jaeger request carries tags
+- Return a 400 for `search_after` sort values that do not match the sort fields (was a 500)
 
 ### Changed
+- (Elasticsearch API) `search_after` on a datetime sort field with a sub-millisecond `fast_precision` requires the `epoch_nanos_int` format
 
 ### Deprecated
 

docs/reference/es_compatible_api.md

@@ -237,6 +237,8 @@ You can pass the `sort` value of the last hit in a subsequent request where othe
 
 This allows you to paginate your results.
 
+If a datetime sort field has a `fast_precision` finer than milliseconds, `search_after` requires the `epoch_nanos_int` format for that field. Quickwit rejects these requests without it: with millisecond sort values, Quickwit would skip the hits within the same millisecond as the last hit.
+
 ### `_msearch`   Multi search API
 
 ```

docs/reference/rest-api.md

@@ -64,6 +64,7 @@ POST api/v1/<index id>/search
 | `start_timestamp` | `i64`      | If set, restrict search to documents with a `timestamp >= start_timestamp`, taking advantage of potential time pruning opportunities. The value must be in seconds. | |
 | `end_timestamp`   | `i64`      | If set, restrict search to documents with a `timestamp < end_timestamp`, taking advantage of potential time pruning opportunities. The value must be in seconds.    | |
 | `start_offset`    | `Integer`  | Number of documents to skip | `0` |
+| `search_after`    | `String`   | Cursor from the `cursors` of a previous response. The search returns the hits that come after that hit in the `sort_by` order, i.e. the next page. See [paging with cursors](#paging-with-cursors). | |
 | `max_hits`        | `Integer`  | Maximum number of hits to return (by default 20) | `20` |
 | `search_field`    | `[String]` | Fields to search on if no field name is specified in the query. Comma-separated list, e.g. "field1,field2"  | index_config.search_settings.default_search_fields |
 | `snippet_fields`  | `[String]` | Fields to extract snippet on. Comma-separated list, e.g. "field1,field2"  | |
@@ -82,9 +83,18 @@ The response is a JSON object, and the content type is `application/json; charse
 | Field                   | Description                    | Type       |
 | --------------------    | ------------------------------ | :--------: |
 | `hits`                | Results of the query           | `[hit]`    |
+| `cursors`             | Opaque `search_after` cursor of each hit, in the same order as `hits` | `[string]` |
 | `num_hits`            | Total number of matches        | `number`   |
 | `elapsed_time_micros` | Processing time of the query   | `number`   |
 
+#### Paging with cursors
+
+To page past the 10,000 `start_offset` limit, pass the last entry of `cursors` as `search_after` in the next request, with the same `query` and `sort_by`. To page backward, pass a hit's cursor and invert every `sort_by` order: `ts` and `-ts` swap, and `-_doc` inverts the default order.
+
+- Quickwit breaks ties on equal sort values by hit address and changes addresses when it merges splits. A merge between two requests can skip or repeat the tied hits, or any hit without `sort_by`. For a snapshot, use the Elasticsearch-compatible [scroll API](es_compatible_api.md#_searchscroll--scroll-api).
+- Quickwit sorts hits without a value for a sort field last in both directions and rejects their cursors.
+- Quickwit rejects paging on a datetime field whose `fast_precision` is finer than milliseconds, because cursors carry milliseconds.
+
 ### Search multiple indices
 Search APIs that accept `index id` requests path parameter also support multi-target syntax.
 

quickwit/quickwit-cli/src/tool.rs

@@ -565,6 +565,7 @@ pub async fn local_search_cli(args: LocalSearchArgs) -> anyhow::Result<()> {
         sort_by,
         count_all: CountHits::CountAll,
         allow_failed_splits: false,
+        search_after: None,
     };
     let search_request =
         search_request_from_api_request(vec![args.index_id], search_request_query_string)?;

quickwit/quickwit-rest-client/src/models.rs

@@ -81,6 +81,8 @@ impl ApiResponse {
 pub struct SearchResponseRestClient {
     pub num_hits: u64,
     pub hits: Vec<JsonValue>,
+    #[serde(default)]
+    pub cursors: Vec<String>,
     pub snippets: Option<Vec<JsonValue>>,
     pub elapsed_time_micros: u64,
     pub errors: Vec<String>,

quickwit/quickwit-rest-client/src/rest_client.rs

@@ -852,6 +852,7 @@ mod test {
         let expected_search_response = SearchResponseRestClient {
             num_hits: 0,
             hits: Vec::new(),
+            cursors: Vec::new(),
             snippets: None,
             aggregations: None,
             elapsed_time_micros: 100,
@@ -874,6 +875,44 @@ mod test {
         );
     }
 
+    #[tokio::test]
+    async fn test_search_endpoint_with_cursors() {
+        let mock_server = MockServer::start().await;
+        let server_url = Url::parse(&mock_server.uri()).unwrap();
+        let qw_client = QuickwitClientBuilder::new(server_url).build();
+        let search_query_params = SearchRequestQueryString {
+            ..Default::default()
+        };
+        let expected_search_response = SearchResponseRestClient {
+            num_hits: 1,
+            hits: vec![json!({"body": "foo"})],
+            cursors: vec!["EgFhGAEgAlICEAU".to_string()],
+            snippets: None,
+            aggregations: None,
+            elapsed_time_micros: 100,
+            errors: Vec::new(),
+        };
+        Mock::given(method("POST"))
+            .and(path("/api/v1/my-index/search"))
+            .respond_with(ResponseTemplate::new(StatusCode::OK).set_body_json(json!({
+                "num_hits": 1,
+                "hits": [{"body": "foo"}],
+                "cursors": ["EgFhGAEgAlICEAU"],
+                "elapsed_time_micros": 100,
+                "errors": []
+            })))
+            .up_to_n_times(1)
+            .mount(&mock_server)
+            .await;
+        assert_eq!(
+            qw_client
+                .search("my-index", search_query_params)
+                .await
+                .unwrap(),
+            expected_search_response
+        );
+    }
+
     fn get_ndjson_filepath(ndjson_dataset_filename: &str) -> String {
         format!(
             "{}/resources/tests/{}",

quickwit/quickwit-search/src/lib.rs

@@ -91,7 +91,7 @@ pub use crate::root::{
 };
 pub use crate::search_job_placer::{Job, SearchJobPlacer};
 pub use crate::search_response_rest::{
-    AggregationResults, SearchPlanResponseRest, SearchResponseRest,
+    AggregationResults, SearchAfterCursor, SearchPlanResponseRest, SearchResponseRest,
 };
 pub use crate::service::{MockSearchService, SearchService, SearchServiceImpl};