Repository navigation
Sort index by field - #6833
Draft
guilload wants to merge 1 commit into
Draft
Sort index by field#6833guilload wants to merge 1 commit into
guilload wants to merge 1 commit into
Conversation
guilload
commented
Sep 28, 2026
Comment on lines
+624
to
+634
| tag_fields: [service] | ||
|
|
||
| indexing_settings: | ||
| sort_fields: [service] | ||
| ``` | ||
|
|
||
| `sort_fields` accepts a list of mapped field names but currently rejects more than one entry. | ||
| A bare name sorts ascending; prefix it with `-` for descending order, for example | ||
| `sort_fields: [-service]`. An explicit `+` also means ascending. Nested paths and escaped dots | ||
| use the same syntax as other field references. Empty names and repeated direction prefixes are | ||
| rejected. The field must be indexed with the `raw` tokenizer and use a raw fast-field normalizer. |
Member
Author
There was a problem hiding this comment.
tag_fields should have been part of indexing_settings :/
| first for `asc` and last for `desc`; an empty string is a distinct value. Equal values retain all | ||
| documents. Physical ordering does not change query result ordering. | ||
|
|
||
| Generation-zero splits are sorted during indexing. Compaction preserves the input order, |
| and are compacted separately. This setting does not repartition splits or select service-specific | ||
| output boundaries. | ||
|
|
||
| Physical field sorting cannot be combined with node-level document fingerprint clustering; |
| a field name with a hyphen `-`. | ||
| The special value `_score` means sorting by score, it is also Descending by default. | ||
|
|
||
| The prefix parser is shared with `indexing_settings.sort_fields` through |
Member
Author
There was a problem hiding this comment.
Can the parse be moved to quickwit-common?
| metadata and direct lineage needed to reconstruct its metastore record. | ||
|
|
||
| Splits may declare `sort_fields` in their metastore metadata and recovery entry. Each entry | ||
| records a field name, direction, and logical comparison type, for example |
Member
Author
There was a problem hiding this comment.
Suggested change
| records a field name, direction, and logical comparison type, for example | |
| records a field name, order, and type, for example |
| // but this will require work on tantivy. | ||
| let indexed_split = IndexedSplit { | ||
| split_attrs: SplitAttrs { | ||
| split_id: "test-split".into(), |
Member
Author
There was a problem hiding this comment.
Suggested change
| split_id: "test-split".into(), | |
| split_id: "test-split".to_string(), |
| .send_message(PackagedSplitBatch::new( | ||
| vec![PackagedSplit { | ||
| split_attrs: SplitAttrs { | ||
| split_id: "test-split".into(), |
Member
Author
There was a problem hiding this comment.
Suggested change
| split_id: "test-split".into(), | |
| split_id: "test-split".to_string(), |
| let split_scratch_directory_2 = TempDirectory::for_test(); | ||
| let packaged_split_1 = PackagedSplit { | ||
| split_attrs: SplitAttrs { | ||
| split_id: "test-split-1".into(), |
Member
Author
There was a problem hiding this comment.
Suggested change
| split_id: "test-split-1".into(), | |
| split_id: "test-split-1".to_string(), |
Explicit conversions via to_string() are better than implicit conversations via into. Apply this suggestion everywhere.
| index_uid: pipeline_id.index_uid, | ||
| source_id: pipeline_id.source_id, | ||
| doc_mapping_uid, | ||
| sort_fields: Vec::new(), |
Member
Author
There was a problem hiding this comment.
sort_fields should be known by know why can't we pass it directly and remove the garbage tantivy_sort_by_field, resolve_sort_fields, etc.?
| /// Declaring a type here does not enable indexing configuration support for that type. | ||
| #[derive(Clone, Copy, Debug, Eq, PartialEq, Hash, Serialize, Deserialize, utoipa::ToSchema)] | ||
| #[serde(rename_all = "lowercase")] | ||
| pub enum SortValueType { |
Member
Author
There was a problem hiding this comment.
Can we rationalize with what already exists in quickwit-search (SortFieldType, etc.) so we can remove SortValueType, SplitSortFieldType, etc.
guilload
force-pushed
the
guilload/sorted-splits-1
branch
from
September 28, 2026 21:51
0119e60 to
998682c
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introduce ability to sort an index by a single field.
How was this PR tested?
Describe how you tested this PR.