Repository navigation
Conversation
BuildFromSortedFiles builds an index from any number of sorted files (spec §3.9). It splits the blocks into stripes of consecutive blocks holding about 32K to 256K keys each. One goroutine gathers each stripe's run out of every file, through aligned buffers that a pool of read goroutines fills ahead of it, and each worker takes whole stripes: it sorts a stripe's records by block with a counting sort, solves its blocks and hands them to the existing writer, which commits them in block order. Nothing merges, memory is about 256 MiB whatever the number of keys or files, and the index is byte-identical to a SortedBuilder's for the same keys. A build whose output is one of its inputs fails before creating it. The parallel pipeline now starts with the worker it is given (startPipeline), runWorker's per-block work is shared with the stripe workers (loadBlock, blockSolver.build, sendResult), and its drain refuses to finalize once the pipeline's context is canceled, so a producer that stops early never leaves a short index. newBuildConfig folds the options once, and WithPayload records that it was given, so the build can reject one that disagrees with the files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjQed1VX9uGU9PWpAjBiz7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Building one index from many sorted files used to mean merging them into a single stream and feeding it to a
SortedBuilder. The merge is serial, and so is the one goroutine that callsAddKey. In stellar-rpc, where a window index covers up to 90 billion keys in 1,000 files, the merge capped the build at about 29M keys/s, so a full window took about an hour. A single feeder caps any design of that shape at about 70M keys/s.The build doesn't need a merge. streamhash only needs each block's keys together and the blocks in order, and the order inside a block doesn't change the index. A sorted file holds any key range as one contiguous run. So
BuildFromSortedFilesmakes the stripe, a run of consecutive blocks holding about a hundred thousand keys, the unit of work:AddKeywould have fed them.The index is byte-identical to one built by adding the same keys to a
SortedBuilderin block order. Equivalence tests pin that across both algorithms, payload and fingerprint sizes, worker counts, stripe sizes and file layouts.The options are
NewSortedBuilder's, andWithWorkersis the only one that tunes speed. The payload size comes from the files. The build refuses a file that is truncated or out of order, a duplicate key, mismatched payload sizes, and an output path that is one of its inputs. A failed or canceled build removes its partial output. Memory is fixed at about 256 MiB, half read buffers and half stripes in flight, plus each worker's block state, however many keys or files there are.Measured on a 32-vCPU machine with local NVMe, 1.2B keys with a 3-byte payload and a 1-byte fingerprint, two builds each:
At 24 workers, reads and index writes together reach the disk's limit. The existing builders are unchanged: sorted
AddKeybuilds of 100M keys ran at 77.8M keys/s before and 78.1M after, and unsorted ones at 39.3M and 39.4M.Tuning the merge instead (two-way nodes, a loser tree) measured 20 to 38% faster and stays serial. Exposing prefix bits so callers can feed blocks themselves hands the caller streamhash's internals and keeps one feeder.
🤖 Generated with Claude Code