Honest inventory of what segment-buffer does, by status. Code is the source of
truth; this file tracks reality, not aspirations.
| Status | Meaning |
|---|---|
| FULLY_FUNCTIONAL | Code present and working (tests pass, or exercised in prod). |
| PARTIALLY_FUNCTIONAL | Ships but has documented gaps or edge-case limitations. |
| PLANNED | Designed or discussed; no code yet. |
| WORTH_CONSIDERING | Raw idea, not yet designed. |
Versioning note. Items marked (v0.5.0) shipped in the v0.5.0 release tag; (v0.5.4) shipped in v0.5.4; (v0.5.5) shipped in v0.5.5; (v0.5.6) shipped in v0.5.6; (v0.5.7) shipped in v0.5.7; (v0.6.0) shipped in v0.6.0. The current release is v0.6.0.
| Capability | Status | Notes |
|---|---|---|
Durable bounded queue (SegmentBuffer<T>) |
FULLY_FUNCTIONAL | Generic over T: Serialize + DeserializeOwned + Clone + Send. Proven on 597M+ events in monitor365. |
Append with sequence-number assignment (append) |
FULLY_FUNCTIONAL | Sequence numbers computed atomically inside the mutex (concurrency bug fixed; see CHANGELOG). |
Batch append (append_all) |
FULLY_FUNCTIONAL | Single-lock batch primitive added in v0.4.1; assigns contiguous seqs under one lock acquisition. |
Batch + interval auto-flush (flush) |
FULLY_FUNCTIONAL | Configurable via FlushPolicy (v0.4.0): Batch, Interval, BatchOrInterval, BatchOrIntervalMin, Manual. BatchOrIntervalMin suppresses tiny segments during low-throughput periods via a min_batch threshold + max_interval safety valve. Clone + Eq + Hash — usable as a per-policy metrics-cache / shard-dispatch key. |
Config builder (SegmentConfig::builder()) |
FULLY_FUNCTIONAL | Fluent builder added in v0.4.0; required because the struct is #[non_exhaustive]. Clone since v0.5.0 (cipher moved to Arc). |
Owned-item iterator (iter_from → SegmentIter) |
FULLY_FUNCTIONAL | (v0.5.0) iter_from(start, limit) returns (seq, item) pairs; standard Iterator combinators work. seq is derived from the segment's actual start / pending-window base, so it remains correct even when start_seq falls inside a deleted segment. for_each_from stays for the zero-copy lending path. |
Range read across disk + memory (read_from) |
FULLY_FUNCTIONAL | Merges on-disk segments with the in-memory tail, in ascending sequence order. |
Lending iterator (for_each_from) |
FULLY_FUNCTIONAL | Callback consumption without an owned Vec<T>; on par with read_from on the in-memory tail since the panic-free re-entrancy fix (mutex never held across callback; one bounded clone). |
Ack-based segment deletion (delete_acked) |
PARTIALLY_FUNCTIONAL | Removes flushed segment files only; unflushed in-memory items remain until flushed (documented, count stays honest via head_seq clamp). |
Backlog size (pending_count, len, is_empty) |
FULLY_FUNCTIONAL | next_seq - head_seq; honest even when acks race unflushed items. len/is_empty added as standard-collection aliases in 0.2.0. |
Atomic snapshot (stats → BufferStats) |
FULLY_FUNCTIONAL | Single-lock snapshot of pending/latest/head/next seq + disk bytes + live segment_count + pressure. segment_count added in v0.5.5 (non-breaking, #[non_exhaustive]). Base snapshot added in 0.2.0. |
Recovery report (open_with_report) |
FULLY_FUNCTIONAL | Returns RecoveryReport (segment_count, head_seq, next_seq, disk_bytes, removed_tmp_files). Added in v0.4.0. |
Disk-byte resync (sync_disk_bytes) |
FULLY_FUNCTIONAL | Re-stats the directory to correct drift from external file manipulation. Added in v0.4.1. |
Segment size distribution (segment_size_stats) |
FULLY_FUNCTIONAL | On-demand count / min / max / mean / p50 / p90 byte-size distribution of on-disk segment files for tuning FlushPolicy::Batch(N). O(n_segments) scan outside the mutex (like sync_disk_bytes), nearest-rank percentiles. (v0.5.5) (non-breaking; SegmentSizeStats is #[non_exhaustive]). |
Accessors (path, config) |
FULLY_FUNCTIONAL | Expose the directory path and the opened config without Debug-parsing. Added in v0.4.1. |
Backpressure metrics (store_pressure, is_overloaded) |
FULLY_FUNCTIONAL | Ratio of approx_disk_bytes to max_size_bytes. Admission policy is caller-defined. |
unflushed Vec capacity recycling |
FULLY_FUNCTIONAL | flush() reserves the previous batch's capacity on the fresh empty unflushed, avoiding ~log2(N) reallocs on the next batch. No API change. |
Single-process lock (flock at open) |
FULLY_FUNCTIONAL | (v0.5.0) Exclusive flock on <dir>/.segment-buffer.lock via fs4::FileExt::try_lock. Fail-fast SegmentError::Locked on contention. Released on Drop. |
Compile-time Send + Sync of SegmentBuffer<T> |
FULLY_FUNCTIONAL | Static assertion in lib.rs — adding a non-thread-safe field fails the build. Added in 0.2.0. |
| Capability | Status | Notes |
|---|---|---|
| zstd + CBOR segment files | FULLY_FUNCTIONAL | seg_{start:012}_{end:012}.zst, configurable compression_level (1-22). |
Format envelope (SBF1 magic + version) |
FULLY_FUNCTIONAL | 8-byte header; forward-evolvable without breaking legacy readers. |
| Envelope hardening | FULLY_FUNCTIONAL | Reserved-bytes-zero check drops legacy encrypted false-positive rate from 2⁻³² to 2⁻⁵⁶. Added in 0.2.0. |
| Legacy file compatibility (unencrypted) | FULLY_FUNCTIONAL | Pre-envelope (monitor365) files auto-detected; zero migration. Covered by legacy_envelopeless_file_still_reads. |
| Legacy file compatibility (encrypted) | FULLY_FUNCTIONAL | Covered by legacy_encrypted_file_without_envelope_still_reads in 0.2.0 (previously untested). |
| Filename-based crash recovery | FULLY_FUNCTIONAL | No WAL, no metadata DB. open() scans filenames to rebuild head_seq/next_seq. |
| Atomic write (tmp → fsync → rename) | FULLY_FUNCTIONAL | A crash never leaves a partial segment; .tmp debris is cleaned on open(). |
Configurable durability (DurabilityPolicy) |
FULLY_FUNCTIONAL | (v0.5.0) Maximal (fsync file + dir after rename), Segment (fsync file only), Throughput (no fsync; default since v0.6.0, cloud is durable). Copy + Eq + Hash — usable directly as a HashMap/HashSet key. |
| Mutex-never-held-across-I/O invariant | FULLY_FUNCTIONAL | flush() drops the lock before write_segment; recover() drops it across the fs::metadata loop (fixed in 0.2.0). |
Scan cache (scan_segments) |
FULLY_FUNCTIONAL | Directory scan result cached; invalidated by every on-disk mutation. Added in v0.4.0. |
| Scan-cache mtime probe | FULLY_FUNCTIONAL | (v0.5.0) Capability-probed at open(); on capable fs, scan_segments invalidates the cache when the directory mtime moves. |
| Atomic disk-byte tracking | FULLY_FUNCTIONAL | approx_disk_bytes is AtomicU64; flush() no longer re-acquires the mutex to bump it. Added in v0.4.0. |
| Crash-recovery limitation | PARTIALLY_FUNCTIONAL | Unflushed in-memory items are lost on crash (by design — durability requires flush). |
Pluggable I/O (SegmentStore trait) |
PARTIALLY_FUNCTIONAL | (v0.5.0) Trait + RealStore impl shipped; reachable externally only under the loom feature via open_with_store. A second production impl (S3, in-memory) is deferred until a concrete consumer exists. |
| Capability | Status | Notes |
|---|---|---|
SegmentCipher trait (pluggable AEAD) |
FULLY_FUNCTIONAL | Always available; bring any Send + Sync + Debug encrypt/decrypt impl. |
AesGcmCipher (AES-256-GCM, random 12-byte nonce) |
FULLY_FUNCTIONAL | Behind the encryption feature. Byte-compatible with monitor365's segment format. |
XChaCha20Poly1305Cipher (24-byte nonce) |
FULLY_FUNCTIONAL | (v0.5.0) Behind the encryption feature. Installed for new buffers by SegmentConfigBuilder::recommended_cipher(key). Constant-time in software, no AES-NI dep. |
SegmentCipher stored as Arc<dyn … + Send + Sync> |
FULLY_FUNCTIONAL | (v0.5.0) Makes SegmentConfig + builder Clone (was Box pre-v0.5.0; breaking). |
Opaque CipherError with source() chaining |
FULLY_FUNCTIONAL | with_source preserves the underlying AEAD error. Added in 0.2.0 (breaking). |
| Capability | Status | Notes |
|---|---|---|
MPMC via parking_lot::Mutex |
FULLY_FUNCTIONAL | Multiple writers and readers; mutex never held across file I/O. |
| Panic-free re-entrancy | FULLY_FUNCTIONAL | (v0.5.5) for_each_from and iter_from never hold the mutex across the user callback, so re-entrant calls are safe and no public API path panics. Supersedes the v0.4.1 re-entrancy guard. |
tracing instrumentation (debug / info) |
FULLY_FUNCTIONAL | Flush, delete, and recovery events are logged. |
| Typed errors with segment-path context | FULLY_FUNCTIONAL | Cbor/Cipher/Integrity carry the offending file path + phase. Io carries IoSite (Dir/Segment(PathBuf)/Unknown) since v0.5.0 (was Option<PathBuf>; breaking). |
SegmentError::Locked { path } |
FULLY_FUNCTIONAL | (v0.5.0) Distinct from Io; callers pattern-match lock contention without sniffing strings. |
| Criterion benchmarks | FULLY_FUNCTIONAL | 16 targets: append, read_from, read_vs_for_each, delete_acked, recover, stats, append_all, durability_policy, segment_size_stats, cipher, concurrent_append, iter_from, flush, concurrent_read, mixed_read_write, append_realistic. (v0.5.7; expanded post-v0.6.0) |
CI matrix (ubuntu/macos × stable/1.86, -D warnings) |
FULLY_FUNCTIONAL | Dedicated MSRV (1.86) verification job. |
Nightly benchmark CI (bench-nightly.yml) |
FULLY_FUNCTIONAL | cargo bench --features encryption at 04:00 UTC on master; criterion baseline cached between runs for regression comparison; target/criterion/ uploaded as an artifact. No automatic alerting — regressions appear in the Actions log only. |
| Code duplication gate (jscpd) | FULLY_FUNCTIONAL | CI job + scripts/check-duplication.sh (verify-gate gate #19): 2% threshold over src/*.rs, ~1% baseline (2 intentional clones: segment encode signatures, AEAD output assembly). Config .jscpd.json; --exitCode v3 brokenness worked around via jq. |
| Strict Clippy lint architecture (v0.5.5) | FULLY_FUNCTIONAL | [lints.clippy] in Cargo.toml denies pedantic + nursery + restriction lints (as_conversions, arithmetic_side_effects, unwrap_used, expect_used, indexing_slicing, string_slice, panic, exit, todo, unreachable) across all targets. Library code is fully clippy-clean under the entire strict set — provably panic-free. Test/bench/example modules carry #![allow(...)] overrides. namtao-inspired. |
| Capability | Status | Notes |
|---|---|---|
| Unit tests (141) + property tests (38) | FULLY_FUNCTIONAL | CRUD, partial reads, limits, crash recovery, concurrency, error paths, envelope detection, encrypted-legacy read, append_all batch semantics, panic-free re-entrancy, FlushPolicy/BatchOrIntervalMin decision logic + edge cases, segment_size_stats distribution, percentile edge cases, live segment_count lifecycle, scan-cache TOCTOU regression, cipher constructor equivalence, flock contention. Two MPMC stress tests prove read_from never corrupts under concurrent delete_acked or flush. See Test coverage detail below for the full category breakdown. |
| Doc tests (39) | FULLY_FUNCTIONAL | Every public method has a runnable example. Count is the cargo test --features encryption doctest-binary total. |
Property tests (proptest, 38 properties) |
FULLY_FUNCTIONAL | Filename bijection, payload bijection, envelope identity, encrypted roundtrip with varied key, corrupted-segment/recovery analogues, FlushPolicy::Manual never auto-flushes, read_from limit monotonicity, delete_acked pending_count monotone non-increasing, for_each_from ↔ read_from equivalence, sync_disk_bytes reconciliation, segment_size_stats directory cross-check (count/min/max/mean + nearest-rank percentiles vs an independent integer implementation), parametrized nearest-rank percentile for all pct in 0..=100, append_all boundary, BatchOrIntervalMin decision formula, unwrap_envelope boundary at ENVELOPE_LEN, segment_count matches disk across arbitrary append/flush/delete sequences, plus ten consistency-model tests: three deterministic (surviving-items-after-delete, disk-memory-split, gap-closes-after-flush), four single-mutation concurrent (read_from delete-acked race, read_from flush race with completeness assertion, for_each_from under concurrent flush, for_each_from under concurrent delete_acked), two dual-mutation concurrent (read_from vs deleter + flusher, iter_from vs deleter + flusher), and one segment_count vs disk across flush/delete ops, plus publish_disk_stats correctness (approx_disk_bytes + segment_count vs directory truth after sync and recover), delete_acked idempotency under concurrent append (overlapping ack ranges, no double-count), compute_store_pressure (zero-max, clamping, monotonicity, saturation), and percentile_of_sorted edge cases (empty, all-equal, no-interpolation, p0/p100 boundaries). |
Fuzz targets (cargo-fuzz) |
FULLY_FUNCTIONAL | Scaffold in fuzz/; 7 targets (fuzz_corrupted_read, fuzz_recovery, fuzz_parse_filename, fuzz_envelope, fuzz_append_all, fuzz_flush_policy, fuzz_for_each_from). Verified locally via Nix devShells.fuzz (nightly + libfuzzer-sys). Nightly scheduled CI workflow added in v0.4.1; fuzz_flush_policy added in v0.5.5; fuzz_for_each_from added in v0.5.6. Proptest analogues run on every cargo test as additional coverage. |
| Loom concurrency verification | FULLY_FUNCTIONAL | 14 tests in tests/loom.rs. Covers the in-memory hot path (append / append_all / pending_count / latest_sequence / stats snapshot), the delete_acked + append interleaving (4 tests exhaustively enumerating every two-thread schedule via a loom-aware MockStore), the scan-cache populate path under read_from (2 tests exercising flush/delete_acked racing the cache-populate interleaving), segment_count self-healing under concurrent flush + delete (proves the atomic counter never panics and is recalibrated by sync_disk_bytes after a momentary wrap), AND for_each_from Phase 2 snapshot + iter_from under concurrent append (2 tests proving the snapshot-then-release-lock pattern and the materialising iterator are safe under every interleaving). flush (the byte-level encode pipeline) and recover still covered statistically by the stress test concurrency_4_writers_1_reader_10k_events in src/tests.rs. |
| Allocation-count regression guard | FULLY_FUNCTIONAL | tests/alloc_guard.rs: a counting allocator asserting fixed heap-allocation budgets on the hot paths (warm append: 1, read_from in-memory: 3, stats: 1, append+flush: 32). Machine-independent — catches tail-latency regressions (extra clones, Vec growth) without CI hardware variance. |
The 141 unit tests + 38 property tests cover:
Unit tests (src/tests.rs):
- CRUD lifecycle (open, append, flush, read_from, delete_acked, close/reopen)
- Partial reads, limit enforcement, boundary conditions
- Crash recovery (filename parsing, tmp cleanup, head_seq/next_seq rebuild)
- Concurrency stress (4-writer/1-reader 10k events, MPMC boundary, high-concurrency segment_count)
- Error paths (corrupted zstd, truncated segments, Io failures, integrity errors)
- Envelope detection (SBF1 magic, legacy auto-detect, false-positive resistance)
- Encrypted-legacy read compatibility
append_allbatch semantics (gap-free sequences, concurrent safety)FlushPolicyvariants:Batch(N),Manual,BatchOrIntervalMindecision logic + edge casessegment_size_statsdistribution (empty/single/multi-segment, nearest-rank percentiles)- Live
segment_countlifecycle (flush increment, delete decrement, recover/sync recalibrate) - Scan-cache TOCTOU deterministic regression (Barrier-based interleaving)
- Cipher constructor equivalence, XChaCha20 roundtrip
flockcontention (single-process enforcement)format_bytes_humanedge cases (0, 1023, 1024, 1025, u64::MAX)- Compression-level default regression guard
Property tests (src/property_tests.rs):
- Filename/payload/envelope bijections
- Encrypted roundtrip with varied keys
- Corrupted-segment/recovery fuzz analogues
read_fromlimit monotonicity, surviving-items correctness after deletefor_each_from↔read_fromequivalencesync_disk_bytesreconciliationsegment_size_statsdirectory cross-check vs independent implementation- Nearest-rank percentile parametrized (all
pct in 0..=100) append_allboundary,delete_ackedidempotency under concurrent append- Consistency-model: 3 deterministic + 4 single-mutation concurrent + 2 dual-mutation concurrent + 1 segment_count invariant
publish_disk_statscorrectness,compute_store_pressure(zero-max, clamping, saturation)
| Capability | Status | Notes |
|---|---|---|
| cargo-deny config | FULLY_FUNCTIONAL | deny.toml: advisories + licenses + bans + sources. Added v0.4.0. |
| cargo-audit in CI | FULLY_FUNCTIONAL | Dedicated supply-chain job in CI. Added v0.4.1. |
| cargo-deny in CI | FULLY_FUNCTIONAL | Same supply-chain job. Added v0.4.1. |
cargo supply-chain publishers report |
FULLY_FUNCTIONAL | (v0.5.0) Weekly informational workflow surfacing crates.io publisher attribution. Non-gating. |
| Dependabot + Renovate | FULLY_FUNCTIONAL | Belt-and-braces dep update configs. Added v0.4.1. Dependabot auto-merge enabled v0.5.0. |
| Nix flake CI | FULLY_FUNCTIONAL | .github/workflows/nix.yml: flake check + build + test + fmt on ubuntu + macOS. |
| Signed commits | FULLY_FUNCTIONAL | (v0.5.0) git verify-commit HEAD succeeds via gpg.ssh.allowedSignersFile. |
| Nightly fuzz CI | FULLY_FUNCTIONAL | .github/workflows/fuzz.yml: scheduled cargo-fuzz runs. v0.4.1. |
| Flake.lock auto-update | FULLY_FUNCTIONAL | .github/workflows/update-flake-lock.yml: weekly PR. v0.4.1. |
| Capability | Status | Notes |
|---|---|---|
Performance tuning guide (docs/PERFORMANCE.md § Tuning) |
FULLY_FUNCTIONAL | Impact-ordered guide to the four config-only Tier 0 levers: DurabilityPolicy, FlushPolicy + append_all, compression_level, for_each_from. |
examples/basic_usage.rs |
FULLY_FUNCTIONAL | Minimum append/read/delete lifecycle cycle. |
examples/cloud_sync.rs |
FULLY_FUNCTIONAL | Full at-least-once drain loop with retry under transient failures. |
examples/cloud_sync_disk_full.rs |
FULLY_FUNCTIONAL | Drain loop that pushes backpressure to the producer when store_pressure() exceeds a threshold. |
examples/idempotent_server.rs |
FULLY_FUNCTIONAL | Server-side (producer_id, seq) dedup pattern that makes at-least-once effectively-once. |
examples/crash_recovery.rs |
FULLY_FUNCTIONAL | Flushed segments survive a simulated crash; unflushed don't; open_with_report prints the recovery scan. |
examples/backpressure.rs |
FULLY_FUNCTIONAL | The canonical pattern for translating store_pressure() into an admission decision. |
examples/background_flush.rs |
FULLY_FUNCTIONAL | Recommended pattern for p99-sensitive producers: FlushPolicy::Manual + caller-owned timer thread. Demonstrates shutdown-flag + final-sync-before-exit. |
examples/mpmc.rs |
FULLY_FUNCTIONAL | Multi-producer / multi-consumer sharing via Arc<SegmentBuffer<T>>. |
examples/hotpath_profile.rs |
FULLY_FUNCTIONAL | Latency-histogram harness for the append hot path. |
examples/scaling.rs |
FULLY_FUNCTIONAL | End-to-end 1M–100M lifecycle throughput benchmark. |
examples/encrypted.rs (v0.5.0, --features encryption) |
FULLY_FUNCTIONAL | AES-256-GCM and XChaCha20-Poly1305 ciphers end-to-end. |
examples/bring_your_own_cipher.rs (v0.5.0, --features encryption) |
FULLY_FUNCTIONAL | Implementing the SegmentCipher trait for a custom cipher. |
examples/batch_or_interval_min.rs (v0.5.4) |
FULLY_FUNCTIONAL | Demonstrates BatchOrIntervalMin tiny-segment suppression: burst phase (auto-flush at batch_size) + drip phase (items stay in memory below min_batch). |
examples/segment_tuning.rs (v0.5.5) |
FULLY_FUNCTIONAL | Demonstrates segment_size_stats() used to adjust FlushPolicy::Batch(N) based on observed p50/max. Three-phase tuning loop (baseline → sweep → recommendation). |
See ROADMAP.md for long-term direction (async I/O, envelope v2 with streaming deserialise + Blake3 checksum + compression negotiation, a second SegmentStore impl, streaming cipher) and TODO_LIST.md for short/mid-term work.