notears

Safe chaos engineering + detection validation for homelab k3s / Prometheus stacks.

Inspired by Netflix Chaos Monkey - with guardrails so you (ideally) shed no tears.

Designed to pair with small-homelab-boi (Docker “VMs” + Ansible + k3s + kube-prometheus-stack).

Mode	Job
Chaos	Randomly or deliberately stop containers, restart services, disable interfaces, break DNS, or isolate hosts/CIDRs inside a test window - and always restore.
Detect	Run those scenarios and check whether Prometheus saw the signal and Alertmanager fired the expected alert.

Default posture: dry-run. Mutations require --live plus a confirmation phrase.

Safety (defaults)

• No --live / no --confirm → zero mutations
• protect_control_plane: true + deny-lists: leaves shb-master / host master alone
• k3s, ssh, sshd denied as restart targets
• Fault duration clamped (min / max_duration_seconds)
• Restore in finally (including Ctrl+C) + retry + host-side watchdogs
• notears recover for emergency healing (also gated by --live)

Compatibility with small-homelab-boi

Lab	notears
Containers shb-worker1 / shb-worker2	kill_container targets
SSH ports 2222 / 2223	service / DNS / network chaos
Prometheus localhost:30090	PromQL expectations
Alertmanager localhost:30093	alert expectations
SSH key ../small-homelab-boi/docker/ssh/id_ed25519	default config.example.yaml path

Clone both repos as siblings:

~/Projects/
  small-homelab-boi/
  notears/

Install

git clone https://github.com/thearrowoftime/notears.git
cd notears
python -m venv .venv
# Windows: .\.venv\Scripts\Activate.ps1
# Unix:    source .venv/bin/activate
pip install -e ".[dev]"
cp config.example.yaml config.yaml

Edit config.yaml if your lab path or NodePorts differ.

Quick start

notears -c config.yaml doctor
notears -c config.yaml detect list
notears -c config.yaml chaos once
notears -c config.yaml chaos once -a kill_container -t shb-worker2
notears -c config.yaml detect run

# LIVE - actually mutates the lab
notears -c config.yaml --live --confirm "I UNDERSTAND THIS WILL BREAK THINGS" \
  chaos once -a restart_service -t worker1:k3s-agent

Chaos actions

Action	Effect	Restore
kill_container	docker stop	docker start (+ watchdog)
restart_service	systemctl restart over SSH	ensure active
disable_interface	ip link set … down	up (+ nohup failsafe)
break_dns	overwrite /etc/resolv.conf	restore backup
isolate_host	iptables DROP to host/CIDR	flush chain

Detection scenarios

YAML files under scenarios/. Each declares a chaos action plus Prometheus / Alertmanager expectations. Tune promql / alertname to your real kube-prometheus-stack rules - scenarios should fail if monitoring does not see what you claim.

Layout

notears/
├── notears/           # Python package
│   ├── cli.py
│   ├── chaos/         # chaos engine + actions
│   └── detection/     # validation + probes
├── scenarios/
├── tests/
├── config.example.yaml
└── reports/