AB2D-7352 Add ECS Instance Status Alerts

[smirnovaae]

🎫 Ticket

https://jira.cms.gov/browse/AB2D-7352

🛠 Changes

Added ECS alerts

ℹ️ Context

If an ECS task, service, or instance starts failing silently, we do not currently have a reliable alerting mechanism to notify the team. This creates a risk that failures could go unnoticed until someone checks logs manually or a downstream issue is reported.

🧪 Validation

[Sadibhatla]

@smirnovaae , Looks like one of the checks failed . Could you please check .
cc : @gfreeman-navapbc

[bennavapbc]

Do we want a separate Slack channel for dev/test/sandbox? I'd hate to add more noise to our current alerts channel

[Sadibhatla]

Do we want a separate Slack channel for dev/test/sandbox? I'd hate to add more noise to our current alerts channel

Yes, Grant was referring to it too .. Will need to get a ticket created for it.
@BharathL95

[Copilot]

Pull request overview

Adds CloudWatch/EventBridge-based alerting for ECS task/service failures so silent ECS issues surface in the shared ab2d-<env>-cloudwatch-alarms SNS → Slack pipeline.

Changes:

• Introduces EventBridge rules + CloudWatch Logs metric filters/alarms for ECS task stops, deployment failures, running-below-desired, and scheduled task failures.
• Centralizes API ALB health alarm notifications onto the shared CloudWatch alarms SNS topic (removing the per-service SNS topic/subscriptions).
• Adjusts 10-core alarm publishing/subscription wiring for the shared alarms topic.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ops/services/60-monitors/ecs.tf	Adds EventBridge capture + log metric filters + CloudWatch alarms for ECS failure/health conditions.
ops/services/60-monitors/data.tf	Adds ECS cluster/service lookups and a log resource policy for EventBridge → CloudWatch Logs delivery.
ops/services/30-api/main.tf	Removes per-service SNS topic/subscription and routes API “healthy host” alarm to shared alarms topic.
ops/services/30-api/cloudwatch.tf	Removes Splunk email subscription tied to the API CloudWatch alarms topic.
ops/services/10-core/main.tf	Updates alarm action wiring and the alarms-topic Slack subscription endpoint reference.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[smirnovaae]

@smirnovaae , Looks like one of the checks failed . Could you please check . cc : @gfreeman-navapbc

Now all checks passed

[smirnovaae]

Do we want a separate Slack channel for dev/test/sandbox? I'd hate to add more noise to our current alerts channel

I added filter when only prod sends alert to slack
this might be temporary until we have a separate Slack channel

[mianava]

Are any of these already covered by Datadog ECS alerts?

[smirnovaae]

Are any of these already covered by Datadog ECS alerts?

Datadog only defines two ECS monitors, both utilization-based: CPU High and Memory High. No overlap with anything here.