Skip to content

Blueprint for chaos#109

Open
gloskull wants to merge 2 commits into
VeriNode-Labs:mainfrom
gloskull:Blueprint-for-Chaos
Open

Blueprint for chaos#109
gloskull wants to merge 2 commits into
VeriNode-Labs:mainfrom
gloskull:Blueprint-for-Chaos

Conversation

@gloskull

Copy link
Copy Markdown

Motivation

  • Provide a code-backed, single source of truth for staging chaos experiments so off-chain orchestrators and runbooks share the same safety thresholds.
  • Define system-wide safety gates (latency, availability, error-budget, healthy zones, security approvals) and guarded blue-green/canary rollout phases for safe staging experiments.
  • Document the staging experiment scope and operational runbook so runs can be audited and automatically gated.

Description

  • Add a new module src/chaos.rs that defines constants, ServiceSurface, FaultKind, RolloutPhase, ChaosExperiment, ChaosHealthSnapshot, a canonical STAGING_CHAOS_EXPERIMENTS catalog, and next_rollout_phase and safety-gate helpers.
  • Register the new module from the crate root by adding pub mod chaos; in src/lib.rs so the catalog and helpers are available to off-chain tooling.
  • Add human-facing documentation at docs/chaos-engineering-staging.md describing architecture, scope, safety gates, monitoring/alerting, blue-green/canary flow, and a runbook.
  • Include unit tests in src/chaos.rs that validate catalog coverage, safety-gate behavior, and rollout progression.

Testing

  • Ran cargo test chaos which executed the chaos module unit tests catalog_covers_every_service_surface_with_safe_blast_radius, safety_gate_enforces_latency_availability_security_and_canary_burn, and rollout_phase_advances_through_blue_green_and_canary, and all passed.
  • The run also executed the existing mempool chaos test chaos_50k_transactions_block_matches_top_fee_per_gas_subject_to_gas_limit and it passed.
  • Overall the automated test run completed with the chaos-related tests passing and no test failures.

Closes #80

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blueprint for Chaos Engineering Testing in Staging

1 participant