Skip to content

Latest commit

 

History

History
168 lines (145 loc) · 38.3 KB

FEATURES.md

File metadata and controls

168 lines (145 loc) · 38.3 KB

Features

Honest inventory of what segment-buffer does, by status. Code is the source of truth; this file tracks reality, not aspirations.

Status Meaning
FULLY_FUNCTIONAL Code present and working (tests pass, or exercised in prod).
PARTIALLY_FUNCTIONAL Ships but has documented gaps or edge-case limitations.
PLANNED Designed or discussed; no code yet.
WORTH_CONSIDERING Raw idea, not yet designed.

Versioning note. Items marked (v0.5.0) shipped in the v0.5.0 release tag; (v0.5.4) shipped in v0.5.4; (v0.5.5) shipped in v0.5.5; (v0.5.6) shipped in v0.5.6; (v0.5.7) shipped in v0.5.7; (v0.6.0) shipped in v0.6.0. The current release is v0.6.0.

Core queue

Capability Status Notes
Durable bounded queue (SegmentBuffer<T>) FULLY_FUNCTIONAL Generic over T: Serialize + DeserializeOwned + Clone + Send. Proven on 597M+ events in monitor365.
Append with sequence-number assignment (append) FULLY_FUNCTIONAL Sequence numbers computed atomically inside the mutex (concurrency bug fixed; see CHANGELOG).
Batch append (append_all) FULLY_FUNCTIONAL Single-lock batch primitive added in v0.4.1; assigns contiguous seqs under one lock acquisition.
Batch + interval auto-flush (flush) FULLY_FUNCTIONAL Configurable via FlushPolicy (v0.4.0): Batch, Interval, BatchOrInterval, BatchOrIntervalMin, Manual. BatchOrIntervalMin suppresses tiny segments during low-throughput periods via a min_batch threshold + max_interval safety valve. Clone + Eq + Hash — usable as a per-policy metrics-cache / shard-dispatch key.
Config builder (SegmentConfig::builder()) FULLY_FUNCTIONAL Fluent builder added in v0.4.0; required because the struct is #[non_exhaustive]. Clone since v0.5.0 (cipher moved to Arc).
Owned-item iterator (iter_from → SegmentIter) FULLY_FUNCTIONAL (v0.5.0) iter_from(start, limit) returns (seq, item) pairs; standard Iterator combinators work. seq is derived from the segment's actual start / pending-window base, so it remains correct even when start_seq falls inside a deleted segment. for_each_from stays for the zero-copy lending path.
Range read across disk + memory (read_from) FULLY_FUNCTIONAL Merges on-disk segments with the in-memory tail, in ascending sequence order.
Lending iterator (for_each_from) FULLY_FUNCTIONAL Callback consumption without an owned Vec<T>; on par with read_from on the in-memory tail since the panic-free re-entrancy fix (mutex never held across callback; one bounded clone).
Ack-based segment deletion (delete_acked) PARTIALLY_FUNCTIONAL Removes flushed segment files only; unflushed in-memory items remain until flushed (documented, count stays honest via head_seq clamp).
Backlog size (pending_count, len, is_empty) FULLY_FUNCTIONAL next_seq - head_seq; honest even when acks race unflushed items. len/is_empty added as standard-collection aliases in 0.2.0.
Atomic snapshot (stats → BufferStats) FULLY_FUNCTIONAL Single-lock snapshot of pending/latest/head/next seq + disk bytes + live segment_count + pressure. segment_count added in v0.5.5 (non-breaking, #[non_exhaustive]). Base snapshot added in 0.2.0.
Recovery report (open_with_report) FULLY_FUNCTIONAL Returns RecoveryReport (segment_count, head_seq, next_seq, disk_bytes, removed_tmp_files). Added in v0.4.0.
Disk-byte resync (sync_disk_bytes) FULLY_FUNCTIONAL Re-stats the directory to correct drift from external file manipulation. Added in v0.4.1.
Segment size distribution (segment_size_stats) FULLY_FUNCTIONAL On-demand count / min / max / mean / p50 / p90 byte-size distribution of on-disk segment files for tuning FlushPolicy::Batch(N). O(n_segments) scan outside the mutex (like sync_disk_bytes), nearest-rank percentiles. (v0.5.5) (non-breaking; SegmentSizeStats is #[non_exhaustive]).
Accessors (path, config) FULLY_FUNCTIONAL Expose the directory path and the opened config without Debug-parsing. Added in v0.4.1.
Backpressure metrics (store_pressure, is_overloaded) FULLY_FUNCTIONAL Ratio of approx_disk_bytes to max_size_bytes. Admission policy is caller-defined.
unflushed Vec capacity recycling FULLY_FUNCTIONAL flush() reserves the previous batch's capacity on the fresh empty unflushed, avoiding ~log2(N) reallocs on the next batch. No API change.
Single-process lock (flock at open) FULLY_FUNCTIONAL (v0.5.0) Exclusive flock on <dir>/.segment-buffer.lock via fs4::FileExt::try_lock. Fail-fast SegmentError::Locked on contention. Released on Drop.
Compile-time Send + Sync of SegmentBuffer<T> FULLY_FUNCTIONAL Static assertion in lib.rs — adding a non-thread-safe field fails the build. Added in 0.2.0.

Storage format

Capability Status Notes
zstd + CBOR segment files FULLY_FUNCTIONAL seg_{start:012}_{end:012}.zst, configurable compression_level (1-22).
Format envelope (SBF1 magic + version) FULLY_FUNCTIONAL 8-byte header; forward-evolvable without breaking legacy readers.
Envelope hardening FULLY_FUNCTIONAL Reserved-bytes-zero check drops legacy encrypted false-positive rate from 2⁻³² to 2⁻⁵⁶. Added in 0.2.0.
Legacy file compatibility (unencrypted) FULLY_FUNCTIONAL Pre-envelope (monitor365) files auto-detected; zero migration. Covered by legacy_envelopeless_file_still_reads.
Legacy file compatibility (encrypted) FULLY_FUNCTIONAL Covered by legacy_encrypted_file_without_envelope_still_reads in 0.2.0 (previously untested).
Filename-based crash recovery FULLY_FUNCTIONAL No WAL, no metadata DB. open() scans filenames to rebuild head_seq/next_seq.
Atomic write (tmp → fsync → rename) FULLY_FUNCTIONAL A crash never leaves a partial segment; .tmp debris is cleaned on open().
Configurable durability (DurabilityPolicy) FULLY_FUNCTIONAL (v0.5.0) Maximal (fsync file + dir after rename), Segment (fsync file only), Throughput (no fsync; default since v0.6.0, cloud is durable). Copy + Eq + Hash — usable directly as a HashMap/HashSet key.
Mutex-never-held-across-I/O invariant FULLY_FUNCTIONAL flush() drops the lock before write_segment; recover() drops it across the fs::metadata loop (fixed in 0.2.0).
Scan cache (scan_segments) FULLY_FUNCTIONAL Directory scan result cached; invalidated by every on-disk mutation. Added in v0.4.0.
Scan-cache mtime probe FULLY_FUNCTIONAL (v0.5.0) Capability-probed at open(); on capable fs, scan_segments invalidates the cache when the directory mtime moves.
Atomic disk-byte tracking FULLY_FUNCTIONAL approx_disk_bytes is AtomicU64; flush() no longer re-acquires the mutex to bump it. Added in v0.4.0.
Crash-recovery limitation PARTIALLY_FUNCTIONAL Unflushed in-memory items are lost on crash (by design — durability requires flush).
Pluggable I/O (SegmentStore trait) PARTIALLY_FUNCTIONAL (v0.5.0) Trait + RealStore impl shipped; reachable externally only under the loom feature via open_with_store. A second production impl (S3, in-memory) is deferred until a concrete consumer exists.

Encryption

Capability Status Notes
SegmentCipher trait (pluggable AEAD) FULLY_FUNCTIONAL Always available; bring any Send + Sync + Debug encrypt/decrypt impl.
AesGcmCipher (AES-256-GCM, random 12-byte nonce) FULLY_FUNCTIONAL Behind the encryption feature. Byte-compatible with monitor365's segment format.
XChaCha20Poly1305Cipher (24-byte nonce) FULLY_FUNCTIONAL (v0.5.0) Behind the encryption feature. Installed for new buffers by SegmentConfigBuilder::recommended_cipher(key). Constant-time in software, no AES-NI dep.
SegmentCipher stored as Arc<dyn … + Send + Sync> FULLY_FUNCTIONAL (v0.5.0) Makes SegmentConfig + builder Clone (was Box pre-v0.5.0; breaking).
Opaque CipherError with source() chaining FULLY_FUNCTIONAL with_source preserves the underlying AEAD error. Added in 0.2.0 (breaking).

Concurrency & operations

Capability Status Notes
MPMC via parking_lot::Mutex FULLY_FUNCTIONAL Multiple writers and readers; mutex never held across file I/O.
Panic-free re-entrancy FULLY_FUNCTIONAL (v0.5.5) for_each_from and iter_from never hold the mutex across the user callback, so re-entrant calls are safe and no public API path panics. Supersedes the v0.4.1 re-entrancy guard.
tracing instrumentation (debug / info) FULLY_FUNCTIONAL Flush, delete, and recovery events are logged.
Typed errors with segment-path context FULLY_FUNCTIONAL Cbor/Cipher/Integrity carry the offending file path + phase. Io carries IoSite (Dir/Segment(PathBuf)/Unknown) since v0.5.0 (was Option<PathBuf>; breaking).
SegmentError::Locked { path } FULLY_FUNCTIONAL (v0.5.0) Distinct from Io; callers pattern-match lock contention without sniffing strings.
Criterion benchmarks FULLY_FUNCTIONAL 16 targets: append, read_from, read_vs_for_each, delete_acked, recover, stats, append_all, durability_policy, segment_size_stats, cipher, concurrent_append, iter_from, flush, concurrent_read, mixed_read_write, append_realistic. (v0.5.7; expanded post-v0.6.0)
CI matrix (ubuntu/macos × stable/1.86, -D warnings) FULLY_FUNCTIONAL Dedicated MSRV (1.86) verification job.
Nightly benchmark CI (bench-nightly.yml) FULLY_FUNCTIONAL cargo bench --features encryption at 04:00 UTC on master; criterion baseline cached between runs for regression comparison; target/criterion/ uploaded as an artifact. No automatic alerting — regressions appear in the Actions log only.
Code duplication gate (jscpd) FULLY_FUNCTIONAL CI job + scripts/check-duplication.sh (verify-gate gate #19): 2% threshold over src/*.rs, ~1% baseline (2 intentional clones: segment encode signatures, AEAD output assembly). Config .jscpd.json; --exitCode v3 brokenness worked around via jq.
Strict Clippy lint architecture (v0.5.5) FULLY_FUNCTIONAL [lints.clippy] in Cargo.toml denies pedantic + nursery + restriction lints (as_conversions, arithmetic_side_effects, unwrap_used, expect_used, indexing_slicing, string_slice, panic, exit, todo, unreachable) across all targets. Library code is fully clippy-clean under the entire strict set — provably panic-free. Test/bench/example modules carry #![allow(...)] overrides. namtao-inspired.

Testing & trust

Capability Status Notes
Unit tests (141) + property tests (38) FULLY_FUNCTIONAL CRUD, partial reads, limits, crash recovery, concurrency, error paths, envelope detection, encrypted-legacy read, append_all batch semantics, panic-free re-entrancy, FlushPolicy/BatchOrIntervalMin decision logic + edge cases, segment_size_stats distribution, percentile edge cases, live segment_count lifecycle, scan-cache TOCTOU regression, cipher constructor equivalence, flock contention. Two MPMC stress tests prove read_from never corrupts under concurrent delete_acked or flush. See Test coverage detail below for the full category breakdown.
Doc tests (39) FULLY_FUNCTIONAL Every public method has a runnable example. Count is the cargo test --features encryption doctest-binary total.
Property tests (proptest, 38 properties) FULLY_FUNCTIONAL Filename bijection, payload bijection, envelope identity, encrypted roundtrip with varied key, corrupted-segment/recovery analogues, FlushPolicy::Manual never auto-flushes, read_from limit monotonicity, delete_acked pending_count monotone non-increasing, for_each_from ↔ read_from equivalence, sync_disk_bytes reconciliation, segment_size_stats directory cross-check (count/min/max/mean + nearest-rank percentiles vs an independent integer implementation), parametrized nearest-rank percentile for all pct in 0..=100, append_all boundary, BatchOrIntervalMin decision formula, unwrap_envelope boundary at ENVELOPE_LEN, segment_count matches disk across arbitrary append/flush/delete sequences, plus ten consistency-model tests: three deterministic (surviving-items-after-delete, disk-memory-split, gap-closes-after-flush), four single-mutation concurrent (read_from delete-acked race, read_from flush race with completeness assertion, for_each_from under concurrent flush, for_each_from under concurrent delete_acked), two dual-mutation concurrent (read_from vs deleter + flusher, iter_from vs deleter + flusher), and one segment_count vs disk across flush/delete ops, plus publish_disk_stats correctness (approx_disk_bytes + segment_count vs directory truth after sync and recover), delete_acked idempotency under concurrent append (overlapping ack ranges, no double-count), compute_store_pressure (zero-max, clamping, monotonicity, saturation), and percentile_of_sorted edge cases (empty, all-equal, no-interpolation, p0/p100 boundaries).
Fuzz targets (cargo-fuzz) FULLY_FUNCTIONAL Scaffold in fuzz/; 7 targets (fuzz_corrupted_read, fuzz_recovery, fuzz_parse_filename, fuzz_envelope, fuzz_append_all, fuzz_flush_policy, fuzz_for_each_from). Verified locally via Nix devShells.fuzz (nightly + libfuzzer-sys). Nightly scheduled CI workflow added in v0.4.1; fuzz_flush_policy added in v0.5.5; fuzz_for_each_from added in v0.5.6. Proptest analogues run on every cargo test as additional coverage.
Loom concurrency verification FULLY_FUNCTIONAL 14 tests in tests/loom.rs. Covers the in-memory hot path (append / append_all / pending_count / latest_sequence / stats snapshot), the delete_acked + append interleaving (4 tests exhaustively enumerating every two-thread schedule via a loom-aware MockStore), the scan-cache populate path under read_from (2 tests exercising flush/delete_acked racing the cache-populate interleaving), segment_count self-healing under concurrent flush + delete (proves the atomic counter never panics and is recalibrated by sync_disk_bytes after a momentary wrap), AND for_each_from Phase 2 snapshot + iter_from under concurrent append (2 tests proving the snapshot-then-release-lock pattern and the materialising iterator are safe under every interleaving). flush (the byte-level encode pipeline) and recover still covered statistically by the stress test concurrency_4_writers_1_reader_10k_events in src/tests.rs.
Allocation-count regression guard FULLY_FUNCTIONAL tests/alloc_guard.rs: a counting allocator asserting fixed heap-allocation budgets on the hot paths (warm append: 1, read_from in-memory: 3, stats: 1, append+flush: 32). Machine-independent — catches tail-latency regressions (extra clones, Vec growth) without CI hardware variance.

Test coverage detail

The 141 unit tests + 38 property tests cover:

Unit tests (src/tests.rs):

  • CRUD lifecycle (open, append, flush, read_from, delete_acked, close/reopen)
  • Partial reads, limit enforcement, boundary conditions
  • Crash recovery (filename parsing, tmp cleanup, head_seq/next_seq rebuild)
  • Concurrency stress (4-writer/1-reader 10k events, MPMC boundary, high-concurrency segment_count)
  • Error paths (corrupted zstd, truncated segments, Io failures, integrity errors)
  • Envelope detection (SBF1 magic, legacy auto-detect, false-positive resistance)
  • Encrypted-legacy read compatibility
  • append_all batch semantics (gap-free sequences, concurrent safety)
  • FlushPolicy variants: Batch(N), Manual, BatchOrIntervalMin decision logic + edge cases
  • segment_size_stats distribution (empty/single/multi-segment, nearest-rank percentiles)
  • Live segment_count lifecycle (flush increment, delete decrement, recover/sync recalibrate)
  • Scan-cache TOCTOU deterministic regression (Barrier-based interleaving)
  • Cipher constructor equivalence, XChaCha20 roundtrip
  • flock contention (single-process enforcement)
  • format_bytes_human edge cases (0, 1023, 1024, 1025, u64::MAX)
  • Compression-level default regression guard

Property tests (src/property_tests.rs):

  • Filename/payload/envelope bijections
  • Encrypted roundtrip with varied keys
  • Corrupted-segment/recovery fuzz analogues
  • read_from limit monotonicity, surviving-items correctness after delete
  • for_each_from ↔ read_from equivalence
  • sync_disk_bytes reconciliation
  • segment_size_stats directory cross-check vs independent implementation
  • Nearest-rank percentile parametrized (all pct in 0..=100)
  • append_all boundary, delete_acked idempotency under concurrent append
  • Consistency-model: 3 deterministic + 4 single-mutation concurrent + 2 dual-mutation concurrent + 1 segment_count invariant
  • publish_disk_stats correctness, compute_store_pressure (zero-max, clamping, saturation)

Supply chain & ops

Capability Status Notes
cargo-deny config FULLY_FUNCTIONAL deny.toml: advisories + licenses + bans + sources. Added v0.4.0.
cargo-audit in CI FULLY_FUNCTIONAL Dedicated supply-chain job in CI. Added v0.4.1.
cargo-deny in CI FULLY_FUNCTIONAL Same supply-chain job. Added v0.4.1.
cargo supply-chain publishers report FULLY_FUNCTIONAL (v0.5.0) Weekly informational workflow surfacing crates.io publisher attribution. Non-gating.
Dependabot + Renovate FULLY_FUNCTIONAL Belt-and-braces dep update configs. Added v0.4.1. Dependabot auto-merge enabled v0.5.0.
Nix flake CI FULLY_FUNCTIONAL .github/workflows/nix.yml: flake check + build + test + fmt on ubuntu + macOS.
Signed commits FULLY_FUNCTIONAL (v0.5.0) git verify-commit HEAD succeeds via gpg.ssh.allowedSignersFile.
Nightly fuzz CI FULLY_FUNCTIONAL .github/workflows/fuzz.yml: scheduled cargo-fuzz runs. v0.4.1.
Flake.lock auto-update FULLY_FUNCTIONAL .github/workflows/update-flake-lock.yml: weekly PR. v0.4.1.

Documentation & examples

Capability Status Notes
Performance tuning guide (docs/PERFORMANCE.md § Tuning) FULLY_FUNCTIONAL Impact-ordered guide to the four config-only Tier 0 levers: DurabilityPolicy, FlushPolicy + append_all, compression_level, for_each_from.
examples/basic_usage.rs FULLY_FUNCTIONAL Minimum append/read/delete lifecycle cycle.
examples/cloud_sync.rs FULLY_FUNCTIONAL Full at-least-once drain loop with retry under transient failures.
examples/cloud_sync_disk_full.rs FULLY_FUNCTIONAL Drain loop that pushes backpressure to the producer when store_pressure() exceeds a threshold.
examples/idempotent_server.rs FULLY_FUNCTIONAL Server-side (producer_id, seq) dedup pattern that makes at-least-once effectively-once.
examples/crash_recovery.rs FULLY_FUNCTIONAL Flushed segments survive a simulated crash; unflushed don't; open_with_report prints the recovery scan.
examples/backpressure.rs FULLY_FUNCTIONAL The canonical pattern for translating store_pressure() into an admission decision.
examples/background_flush.rs FULLY_FUNCTIONAL Recommended pattern for p99-sensitive producers: FlushPolicy::Manual + caller-owned timer thread. Demonstrates shutdown-flag + final-sync-before-exit.
examples/mpmc.rs FULLY_FUNCTIONAL Multi-producer / multi-consumer sharing via Arc<SegmentBuffer<T>>.
examples/hotpath_profile.rs FULLY_FUNCTIONAL Latency-histogram harness for the append hot path.
examples/scaling.rs FULLY_FUNCTIONAL End-to-end 1M–100M lifecycle throughput benchmark.
examples/encrypted.rs (v0.5.0, --features encryption) FULLY_FUNCTIONAL AES-256-GCM and XChaCha20-Poly1305 ciphers end-to-end.
examples/bring_your_own_cipher.rs (v0.5.0, --features encryption) FULLY_FUNCTIONAL Implementing the SegmentCipher trait for a custom cipher.
examples/batch_or_interval_min.rs (v0.5.4) FULLY_FUNCTIONAL Demonstrates BatchOrIntervalMin tiny-segment suppression: burst phase (auto-flush at batch_size) + drip phase (items stay in memory below min_batch).
examples/segment_tuning.rs (v0.5.5) FULLY_FUNCTIONAL Demonstrates segment_size_stats() used to adjust FlushPolicy::Batch(N) based on observed p50/max. Three-phase tuning loop (baseline → sweep → recommendation).

Planned / worth considering

See ROADMAP.md for long-term direction (async I/O, envelope v2 with streaming deserialise + Blake3 checksum + compression negotiation, a second SegmentStore impl, streaming cipher) and TODO_LIST.md for short/mid-term work.