AB2D-7379 AB2D-7425 Add pause/resume liveness check

[bennavapbc]

🎫 Ticket

• https://jira.cms.gov/browse/AB2D-7379
• https://jira.cms.gov/browse/AB2D-7425

🛠 Changes

Add liveness check for PrototypeJobLeaseRenewer

ℹ️ Context

PrototypeJobLeaseRenewer renews a lease periodically (every 20 seconds by default). It's possible for a worker to be dead and/or not making any progress in which case we don't want to continue renewing the lease.

An in-memory heartbeat is used to check for liveness, and if the worker appears to be alive, its lease will continue to be renewed by PrototypeJobLeaseRenewer. If too much time elapses without a heartbeat (whether it be from a SpringBatch listener or called manually before creating the aggregated table), its lease will not be renewed.

🧪 Validation

Added integration test with two test cases

[Copilot]

🟡 Changes recommended

The new liveness config keys don’t bind to PrototypeProperties as written and there’s a real race in lease untracking that can drop fresh heartbeats during renewal.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a liveness/heartbeat mechanism to PrototypeJobLeaseRenewer so a worker only continues renewing a job lease when the job is actively making progress, reducing the risk of “zombie” workers holding leases indefinitely.

Changes:

• Introduces heartbeat event/context types and tracks heartbeat state per (jobUuid, fenceToken) while renewing leases on a schedule.
• Emits heartbeats at key execution points (before aggregated table creation, after chunk writes, before assembling output).
• Adds new prototype configuration properties intended to control max allowed time between heartbeats for each stage.

File summaries

File	Description
worker/src/main/resources/application.properties	Adds new liveness threshold configuration values for prototype lease renewal.
worker/src/main/java/gov/cms/ab2d/worker/processor/prototype/PrototypeProperties.java	Adds new configuration fields for heartbeat/liveness thresholds.
worker/src/main/java/gov/cms/ab2d/worker/processor/prototype/PrototypeJobProcessorImpl.java	Posts heartbeats at key job lifecycle points and after item writes; updates untrack signature.
worker/src/main/java/gov/cms/ab2d/worker/processor/prototype/lease/PrototypeJobLeaseToken.java	Adds a typed key for tracking heartbeat state per job + fence token.
worker/src/main/java/gov/cms/ab2d/worker/processor/prototype/lease/PrototypeJobLeaseRenewer.java	Implements heartbeat tracking and conditional lease renewal based on liveness thresholds.
worker/src/main/java/gov/cms/ab2d/worker/processor/prototype/lease/heartbeat/HeartbeatEvent.java	Defines the heartbeat event types used for liveness checks.
worker/src/main/java/gov/cms/ab2d/worker/processor/prototype/lease/heartbeat/HeartbeatContext.java	Stores last heartbeat timestamp, event, and computed expiry threshold.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.