test: add regression test for scalar index training sort spill under bounded memory pool - #8128
Merged
westonpace merged 1 commit intoAug 3, 2026
Conversation
…bounded memory pool Training a BTREE index sorts the column through DataFusion's external sort against a bounded FairSpillPool. Before the 40MB-capped sort spill reservation sizing (lance-format#7675), the un-spillable merge phase could exhaust the pool on large string columns (ResourcesExhausted from ExternalSorterMerge), failing index creation unless LANCE_BYPASS_SPILLING was set. Add a test that forces many spilled sort runs under a 4MiB pool to keep this path covered.
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Gate recommendation: approve. This focused end-to-end test guards a real bounded-pool failure at the BTREE training boundary: the workload exceeds the configured pool, exercises the value-sort path with spilling enabled, and passes under the pool-relative reservation fix. A lower-level sort test could expose spill metrics more directly, but it would not cover the index-training wiring this regression protects.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
westonpace
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Training a BTREE scalar index sorts the whole column through DataFusion's external sort against a bounded
FairSpillPool. Before #7675 sized the sort spill reservation to the pool ((pool / 3).min(40MiB)), the un-spillable merge phase requested DataFusion's default 10MB reservation, which could exceed the entire pool on small-pool / large-column configurations. Index creation then failed withResourcesExhaustedfromExternalSorterMerge(surfaced throughlance-datafusion/src/chunker.rs), and the only workaround was bypassing spilling entirely viaLANCE_BYPASS_SPILLING(unbounded memory).This adds a regression test that trains a BTREE index over
36MiB of 64-byte strings with1 and passes from 18381f3 (#7675) onward, on both DataFusion 53 and 54. Runs in ~7s (debug).LANCE_MEM_POOL_SIZE=4MiB, forcing many spilled sort runs. Verified against history: the test fails with the exactResourcesExhausted(ExternalSorterMerge)error at 18381f3The test lives in its own single-test integration binary because the training path resolves the pool size only from the process-global
LANCE_MEM_POOL_SIZE; the test also asserts the resolved options (pool size, spilling enabled) so it fails loudly instead of passing vacuously if that plumbing changes.