Shrink ECS cluster to one t3a.medium by ale210 · Pull Request #263 · hackforla/incubator

ale210 · 2026-10-10T18:15:53Z

For #255. Shrinks the ECS cluster to one t3a.medium, now that every service uses bridge networking (#254) and nothing needs an ENI.

platform/compute.tf: instance_type m5.large → t3a.medium; desired_capacity 2 → 1 (in ignore_changes, so this only keeps the code honest). max_size = 2 stays as the slot for replacing an instance without an outage.
modules/container/main.tf: the two spread placement strategies become one binpack on memory.
Comments that reasoned about ENI slots or m5.large now reason about memory on one host, in main.tf, compute.tf and iam.tf. The service lifecycle comment above aws_ecs_service's lifecycle block still describes ENIs and needs the same rewording.

Expected plan: launch template updated in place (mints version 10, t3a.medium, metadata_options unchanged), and all ten ECS services updated in place for ordered_placement_strategy. Nothing replaced. The ASG should show no change. Changing the placement strategy may start a new deployment of each service; production deploys at 100/200 and stays up, non-production at 0/200 may blip briefly.

Merging changes no running instance. The host swap is manual and comes after merge, as set out in #255: drain and terminate one m5.large, launch the t3a.medium, then drain and terminate the other.

Sizing, checked 2026-10-10: the ten tasks reserve 2,816 MiB against an estimated ~3,850 MiB registered on a t3a.medium. Seven-day CPU across both current hosts averages about 7% of a 2-vCPU instance, inside the 20% burst baseline.

…p ci]

github-actions · 2026-10-10T18:16:55Z

Terraform plan in terraform
With backend config files: terraform/prod.backend.tfvars

Plan: 0 to add, 11 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # module.platform.aws_launch_template.ecs will be updated in-place
!~  resource "aws_launch_template" "ecs" {
        id                                   = "lt-0093b22d4b048e101"
!~      instance_type                        = "m5.large" -> "t3a.medium"
!~      latest_version                       = 9 -> (known after apply)
        name                                 = "ecs-incubator-prod"
        tags                                 = {}
#        (17 unchanged attributes hidden)

#        (2 unchanged blocks hidden)
    }

  # module.civic-tech-index.module.backend_prod_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/civic-tech-index-backend-prod"
        name                               = "civic-tech-index-backend-prod"
        tags                               = {
            "project" = "civic-tech-index"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.civic-tech-index.module.backend_stage_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/civic-tech-index-backend-stage"
        name                               = "civic-tech-index-backend-stage"
        tags                               = {
            "project" = "civic-tech-index"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.civic-tech-jobs.module.civic_tech_jobs_fullstack_stage_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/civic-tech-jobs-fullstack-stage"
        name                               = "civic-tech-jobs-fullstack-stage"
        tags                               = {
            "project" = "civic-tech-jobs"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.home-unite-us.module.prod_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/home-unite-us-fullstack-prod"
        name                               = "home-unite-us-fullstack-prod"
        tags                               = {
            "project" = "home-unite-us"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.home-unite-us.module.qa_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/home-unite-us-fullstack-qa"
        name                               = "home-unite-us-fullstack-qa"
        tags                               = {
            "project" = "home-unite-us"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.people-depot.module.backend_dev_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/people-depot-backend-dev"
        name                               = "people-depot-backend-dev"
        tags                               = {
            "project" = "people-depot"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.vrms.module.backend_dev_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/vrms-backend-dev"
        name                               = "vrms-backend-dev"
        tags                               = {
            "project" = "vrms"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.vrms.module.backend_prod_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/vrms-backend-prod"
        name                               = "vrms-backend-prod"
        tags                               = {
            "project" = "vrms"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.vrms.module.frontend_dev_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/vrms-frontend-dev"
        name                               = "vrms-frontend-dev"
        tags                               = {
            "project" = "vrms"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

  # module.vrms.module.frontend_prod_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/vrms-frontend-prod"
        name                               = "vrms-frontend-prod"
        tags                               = {
            "project" = "vrms"
        }
#        (19 unchanged attributes hidden)

!~      ordered_placement_strategy {
!~          field = "attribute:ecs.availability-zone" -> "memory"
!~          type  = "spread" -> "binpack"
        }
-       ordered_placement_strategy {
-           field = "instanceId" -> null
-           type  = "spread" -> null
        }

#        (4 unchanged blocks hidden)
    }

Plan: 0 to add, 11 to change, 0 to destroy.

❌ Error applying plan in Terraform apply (OIDC) #104

For #255. Fixes the failed apply from #263. #263's apply rejected all ten ECS service updates: *"Availability Zone Rebalancing only supports availability zone spread as task placement strategy."* Every service has AZ rebalancing enabled, which allows only AZ-spread placement, so the new `binpack` was refused. The launch template change in the same apply succeeded (version 10, `t3a.medium`). The services were left untouched and still serve normally. This sets `availability_zone_rebalancing = "DISABLED"` on `aws_ecs_service`. The cluster is one host in one Availability Zone, so there is nothing to rebalance across. Comments that cited AZ rebalancing as the reason `deployment_maximum_percent` stays at 200 now give the real reason: production starts its new task before stopping the old one. **Expected plan: 0 to add, 10 to change, 0 to destroy.** Each ECS service is updated in place, with `availability_zone_rebalancing` going to `DISABLED` and the placement strategy going to `binpack` on `memory`. The provider sends both changes in one `UpdateService` call. If ECS still rejects the placement because rebalancing has to be off first, the apply fails the same harmless way as #263's, and the fallback is a PR that changes only the rebalancing setting. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

For #255. Comment-only. The comment explaining why `aws_ecs_service` has no `create_before_destroy` still reasoned about ENI slots on two `m5.large` hosts. Every task is bridge-mode now and the cluster is one `t3a.medium`, so the reason is memory: a duplicate service doubles its reservation on a host with about 1 GB to spare. This was the one stale comment left out of #263. **Expected plan: no changes.**

ale210 and others added 2 commits October 10, 2026 11:15

Shrink ECS cluster to one t3a.medium

5a4ae60

terraform-docs: automated updates to Terraform modules README.md [ski…

b4ea828

…p ci]

ale210 merged commit 17e032a into main Oct 10, 2026

ale210 mentioned this pull request Oct 10, 2026

Disable ECS AZ rebalancing so binpack placement applies #264

Merged

This was referenced Oct 10, 2026

Shrink the ECS cluster to one t3a.medium #255

Open

Fix stale ENI comment on the ECS service lifecycle #265

Merged

Navigation Menu

Repository navigation

Shrink ECS cluster to one t3a.medium - #263

Shrink ECS cluster to one t3a.medium#263
ale210 merged 2 commits into
mainfrom
255-shrink-to-t3a-medium

ale210 commented Oct 10, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ale210 commented Oct 10, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Oct 10, 2026 •

edited

Loading