Zero-copy anticompaction via verbatim compression-chunk copy - #5015
Open
clohfink wants to merge 1 commit into
Open
Zero-copy anticompaction via verbatim compression-chunk copy#5015clohfink wants to merge 1 commit into
clohfink wants to merge 1 commit into
Conversation
Anticompact an sstable by splitting it -- copying its compression chunks verbatim and rebuilding every other component from an Index.db-only pass -- instead of rewriting every row, when its full/transient/unrepaired partitions form contiguous token runs. Interleaved ranges, which is what vnodes produce, fall back to the existing rewrite. Off by default (zero_copy_anticompaction_enabled). Where the filesystem can share extents (xfs -m reflink=1, btrfs) the child's Data.db is reflinked from the parent's rather than copied, so a split writes no data blocks and needs no additional disk space. Because a verbatim copy has no CompactionController, this path retains the droppable tombstones and shadowed data a rewrite would have purged, and the children inherit the parent's per-sstable statistics rather than recomputing them. Retention only, never data loss. A child that does not begin on a chunk boundary carries a dead prefix of unindexed bytes, so Scrubber and Verifier now seek to the first index position rather than requiring it to be zero, MmappedRegions seeds segment placement at the first chunk's offset rather than at 0, and CassandraOutgoingFile.contained measures the live span so such a child stays eligible for entire-sstable streaming. New: ZeroCopySSTableSplitter, AntiCompactionRunPlanner, Reflink. Config: zero_copy_anticompaction_enabled, zero_copy_split_reflink_enabled, zero_copy_split_digest_enabled. Metric: BytesZeroCopyAnticompaction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note on PR size: 78% documentation and tests
Anticompact an sstable by splitting it copying its compression chunks verbatim and rebuilding every other component from an Index.db-only pass -- instead of rewriting every row, when its full/transient/unrepaired partitions form contiguous token runs. Interleaved ranges, which is what vnodes produce, fall back to the existing rewrite. Off by default
(zero_copy_anticompaction_enabled).
Where the filesystem can share extents (xfs -m reflink=1, btrfs) the child's Data.db is reflinked from the parent's rather than copied, so a split writes no data blocks and needs no additional disk space.
Because a verbatim copy has no CompactionController, this path retains the droppable tombstones and shadowed data a rewrite would have purged, and the children inherit the parent's per-sstable statistics rather than recomputing them. Retention only, never data loss.
A child that does not begin on a chunk boundary carries a dead prefix of unindexed bytes, so Scrubber and Verifier now seek to the first index position rather than requiring it to be zero, MmappedRegions seeds segment placement at the first chunk's offset rather than at 0, and CassandraOutgoingFile.contained measures the live span so such a child stays eligible for entire-sstable streaming.
New: ZeroCopySSTableSplitter, AntiCompactionRunPlanner, Reflink.
Config: zero_copy_anticompaction_enabled, zero_copy_split_reflink_enabled,
zero_copy_split_digest_enabled. Metric: BytesZeroCopyAnticompaction.