Sort index by field

docs/configuration/index-config.md

@@ -594,6 +594,7 @@ This section describes indexing settings for a given index.
 | ------------- | ------------- | ------------- |
 | `commit_timeout_secs`      | Maximum number of seconds before committing a split since its creation.   | `60` |
 | `split_num_docs_target` | Target number of docs per split.   | `10000000` |
+| `sort_fields` | Physical document order within splits. An empty list disables sorting; currently at most one raw text fast field is supported. | `[]` |
 | `merge_policy` | Describes the strategy used to trigger split merge operations for logs/traces (see [Merge policies](#merge-policies) section below). |
 | `parquet_merge_policy` | Describes the merge policy for Parquet (metrics/sketches) splits (see [Parquet merge policy](#parquet-merge-policy) section below). |
 | `parquet_indexing` | Parquet-specific indexing settings: sort schema, window duration (see [Parquet indexing settings](#parquet-indexing-settings) section below). |
@@ -609,6 +610,33 @@ When decommissioning definitively an indexer node that received data through the
 
 :::
 
+### Physical document sorting
+
+To sort newly written splits by service, configure a single-valued raw text fast field:
+
+```yaml
+doc_mapping:
+  field_mappings:
+    - name: service
+      type: text
+      tokenizer: raw
+      fast: true
+  tag_fields: [service]
+
+indexing_settings:
+  sort_fields: [service]
+```
+
+`sort_fields` accepts a list of mapped field names but currently rejects more than one entry.
+A bare name sorts ascending; prefix it with `-` for descending order, for example
+`sort_fields: [-service]`. An explicit `+` also means ascending. Nested paths and escaped dots
+use the same syntax as other field references. Empty names and repeated order prefixes are
+rejected. The field must be indexed with the `raw` tokenizer and stored as a fast field.
+Arrays, numeric fields, and unmapped JSON paths are not supported. Strings are compared by their
+UTF-8 bytes. Missing and null values sort
+first for `asc` and last for `desc`; an empty string is a distinct value. Equal values retain all
+documents. Physical ordering does not change query result ordering.
+
 ### Merge policies
 
 Quickwit makes it possible to define the strategy used to decide which splits should be merged together and when.

docs/internals/sorting.md

@@ -12,6 +12,13 @@ to use for sorting. Sorting is Descending by default. The sorting order can be r
 a field name with a hyphen `-`.
 The special value `_score` means sorting by score, it is also Descending by default.
 
+The prefix parser is shared with `indexing_settings.sort_fields` through
+`quickwit_common::parse_sort_fields`, which returns field names and orders.
+Each caller supplies its default and reverse orders: descending by default for search,
+ascending by default for indexing.
+A `+` keeps that default and a `-` reverses it. Each caller handles its own list format
+and field validation.
+
 In case of equality between two documents, the GlobalDocId, composed of (SplitId, SegmentId, DocId)
 is used as a tie breaker. It is used to sort in the same order as the first field being sorted by.
 This means it is in Descending order by default.

docs/internals/split-format.md

@@ -12,6 +12,24 @@ It contains the field name, type and capabilities.
 - a versioned protobuf entry named `split_recovery_metadata` containing the immutable split
   metadata and direct lineage needed to reconstruct its metastore record.
 
+Splits may declare `sort_fields` in their metastore metadata and recovery entry. Each entry
+records a field name, order, and type, for example
+`{"field": "service", "order": "asc", "type": "text"}`. The type is resolved from the writer's
+schema, even for an all-null field; it is independent of the physical column encoding. A nonempty
+declaration with a missing or unknown type is rejected, including during recovery.
+
+In Rust, `IndexingSortSchema` represents the ordered list used for merge compatibility, with
+`SortFieldMetadata` describing each field. It serializes directly as the `sort_fields` array;
+the wrapper does not add a level to the stored JSON format.
+
+The list is empty for legacy and unsorted splits; currently writers support one scalar raw text
+field, with ascending or descending byte ordering. Missing values sort first ascending and last
+descending. The declaration must agree with both the sort setting and schema in Tantivy's
+`meta.json`. Merge planners group by the complete sort schema, including type, and merge execution
+rejects mismatches rather than adopting the current index configuration as the order of older
+data. Splits must also have the same document-mapping UID to merge; matching sort declarations
+do not remove that constraint.
+
 The split file data layout looks like this:
 - concatenation all of the files in the split
 - a footer

quickwit/quickwit-common/src/lib.rs

@@ -40,6 +40,7 @@ pub mod retry;
 pub mod ring_buffer;
 pub mod runtimes;
 pub mod shared_consts;
+mod sort_fields;
 pub mod sorted_iter;
 pub mod stream_utils;
 pub mod temp_dir;
@@ -66,6 +67,7 @@ pub use kill_switch::KillSwitch;
 pub use metrics_specific::*;
 pub use path_hasher::PathHasher;
 pub use progress::{Progress, ProtectedZoneGuard};
+pub use sort_fields::parse_sort_fields;
 pub use stream_utils::{BoxStream, ServiceStream};
 
 /// Returns true at compile time. This function is mostly used with serde to initialize boolean

quickwit/quickwit-common/src/sort_fields.rs

@@ -0,0 +1,41 @@
+// Copyright 2021-Present Datadog, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/// Parses field names with an optional direction prefix. Bare names and `+` use
+/// `default_order`; `-` uses `reverse_order`. Indexing uses ascending by default, while the
+/// REST search API historically uses descending.
+///
+/// Callers handle list delimiters, whitespace, empty entries, and field validation.
+/// Only one leading sign is consumed; the rest of each field name is preserved verbatim.
+pub fn parse_sort_fields<Order: Copy>(
+    fields: impl IntoIterator<Item = impl AsRef<str>>,
+    default_order: Order,
+    reverse_order: Order,
+) -> Vec<(String, Order)> {
+    fields
+        .into_iter()
+        .map(|expression| {
+            let expression = expression.as_ref();
+            let (field, order) = if let Some(field) = expression.strip_prefix('-') {
+                (field, reverse_order)
+            } else {
+                (
+                    expression.strip_prefix('+').unwrap_or(expression),
+                    default_order,
+                )
+            };
+            (field.to_string(), order)
+        })
+        .collect()
+}

quickwit/quickwit-compaction/src/planner/compaction_state.rs

@@ -36,6 +36,7 @@ pub struct CompactionPartitionKey {
     pub source_id: SourceId,
     pub partition_id: u64,
     pub doc_mapping_uid: DocMappingUid,
+    pub sort_fields: quickwit_metastore::IndexingSortSchema,
 }
 
 impl CompactionPartitionKey {
@@ -45,6 +46,7 @@ impl CompactionPartitionKey {
             source_id: split.source_id.clone(),
             partition_id: split.partition_id,
             doc_mapping_uid: split.doc_mapping_uid,
+            sort_fields: split.sort_fields.clone(),
         }
     }
 }
@@ -319,6 +321,56 @@ mod tests {
         assert_eq!(state.partition_keys().len(), 1);
     }
 
+    #[test]
+    fn test_sort_fields_separate_compaction_groups() {
+        use quickwit_metastore::SortFieldMetadata;
+        use quickwit_proto::search::{SortFieldType, SortOrder};
+
+        let index_uid = IndexUid::for_test("test-index", 0);
+        let mut state = CompactionState::default();
+
+        for (group, sort_fields) in [
+            Vec::new(),
+            vec![SortFieldMetadata {
+                field: "service".to_string(),
+                order: SortOrder::Asc,
+                field_type: SortFieldType::Text,
+            }],
+            vec![SortFieldMetadata {
+                field: "service".to_string(),
+                order: SortOrder::Desc,
+                field_type: SortFieldType::Text,
+            }],
+            vec![SortFieldMetadata {
+                field: "service".to_string(),
+                order: SortOrder::Asc,
+                field_type: SortFieldType::I64,
+            }],
+        ]
+        .into_iter()
+        .enumerate()
+        {
+            for i in 0..2 {
+                let mut split = test_split(&format!("split-{group}-{i}"), &index_uid);
+                split.sort_fields.fields = sort_fields.clone();
+                state.track_split(split);
+            }
+        }
+        let keys = state.partition_keys();
+        assert_eq!(keys.len(), 4);
+
+        for key in keys {
+            state.plan_partition(&key, &*test_merge_policy());
+        }
+        assert_eq!(state.pending_operations.len(), 4);
+
+        for pending in state.pending_operations.iter() {
+            let splits = &pending.operation.splits;
+            assert_eq!(splits.len(), 2);
+            assert_eq!(splits[0].sort_fields, splits[1].sort_fields);
+        }
+    }
+
     #[test]
     fn test_plan_partition_moves_splits_to_in_flight() {
         let index_uid = IndexUid::for_test("test-index", 0);