AB2D-7374 Crash/fault injection and observable recovery for pause-resume prototype - #1861
AB2D-7374 Crash/fault injection and observable recovery for pause-resume prototype#1861Sadibhatla wants to merge 3 commits into
Conversation
…resume prototype Introduce a small config-driven CrashInjector that can halt the worker at a chosen pipeline point (process, read, write, assemble) so recovery can be exercised in a deployed container, replacing the one hardcoded crash hook in EobItemProcessor. Wire it into the reader, writer, and assembler as well. Add greppable log markers for the crash/recovery lifecycle (CRASH-INJECTION ARMED/firing, and SOFT RESUME / HARD RECOVERY in PrototypeJobRecovery) so a deployed run can be validated from the logs. Tests: CrashInjectorTest for the decision logic, and a new reader-crash recovery case (crashDuringReadingRecovers) filling the one uncovered stage. Add docs/prototype-crash-testing.md runbook for crashing a deployed worker (env-var injector or AWS FIS / ecs stop-task) and what to verify.
Removed unnecessary comments from CrashInjector.java.
| } | ||
|
|
||
| long fenceToken = jobLease.bump(jobUuid, owner); | ||
| log.info("RECOVERY for job {}: HARD RECOVERY on new token {} - a crashed or stale worker was fenced, " |
There was a problem hiding this comment.
Technically acquire gets run every time a job gets picked up, including the first time. So this will log a hard recovery on job pickup. Honestly, we should probably just tighten up the fresh/soft/hard acquisition because fresh/soft are bleeding into each other a little bit.
Hard/soft recovery is already logged, these logs (this hard recovery one and the soft resume one above it) could be edits to the existing ones in the processor.
|
|
||
| @Override | ||
| public CoverageSummary read() { | ||
| crashInjector.maybeCrash("read"); |
There was a problem hiding this comment.
Would be good to make the injection point names read/write/process/assembly an enum.
| this.crashProbability = crashProbability; | ||
| if (crashProbability > 0) { | ||
| // Loud on purpose: if you see this in a deployed worker's logs, it is going to crash itself. | ||
| log.warn("CRASH-INJECTION ARMED at '{}' with probability {} - this worker will halt itself, " |
There was a problem hiding this comment.
Going off the enum idea, this would be a good place to validate that the property matches a value in the enum
| in-process. This runbook is for the other half: proving a **deployed** worker container recovers from a | ||
| real crash without corrupting or duplicating output. | ||
|
|
||
| Crash injection is off by default and is only meant for `dev`/`test`. **Never arm it in prod.** |
There was a problem hiding this comment.
The crash injector should check execution.env and refuse to arm if it's prod/sandbox.
|
|
||
| @Override | ||
| public void write(@NonNull Chunk<? extends SerializedEobs> chunk) throws Exception { | ||
| crashInjector.maybeCrash("write"); |
There was a problem hiding this comment.
The crash is synchronous so this always crashes before we start writing. We'd want to crash mid-write.
2 participants
🎫 Ticket
https://jira.cms.gov/browse/AB2D-7374
🛠 Changes
Adds crash/fault injection and observable recovery to the pause-resume prototype worker so recovery pathways can be validated in a deployed environment.
CrashInjector- a small, config-driven component that halts the worker at a chosen pipeline point.
Runbook— docs/prototype-crash-testing.md: how to crash a deployed worker (env-var injector or AWS FIS / 'ecs stop-task'), what log markers to watch, and what to verify after recovery.
Default behavior is unchanged: 'crash-probability=0' means every injection point is a no-op unless explicitly armed for testing.
ℹ️ Context
We want confidence that a worker can safely restart a job in a deployed environment without data corruption or faulty output (including crashes mid-file-write). Unit and integration tests already cover many recovery cases in-process; this adds a way to exercise recovery on a real, deployed container and to follow the crash → recovery → clean-completion story in the logs.
No new dependencies, security controls, or data flows are affected. Crash injection is off by default and intended for dev/test only.
🧪 Validation
mvn -pl worker -am test — full build green on JDK 25.
CrashInjectorTest: 3/3 pass (arming off by default, fires only at the configured point, case-insensitive).
PrototypeCrashPointIntegrationTest: 5/5 pass, including the new crashDuringReadingRecovers; full prototype recovery suite 17/17, every scenario recovering to SUCCESSFUL with every beneficiary delivered exactly once.