Skip to content

Move people-depot-backend-dev to bridge networking - #260

Merged
ale210 merged 1 commit into
mainfrom
253-people-depot-dev-bridge
Oct 10, 2026
Merged

ale210 merged 1 commit into
mainfrom
253-people-depot-dev-bridge

Conversation

@ale210

@ale210 ale210 commented Oct 10, 2026

Copy link
Copy Markdown
Member

For #253. This is the pilot PR only; the other five non-production services follow in a second one.

Moves people-depot-backend-dev to bridge networking with network_mode = "bridge". peopledepot-dev.vrms.io will be unreachable for about two minutes during the apply.

Expected plan: 2 to add, 2 to change, 5 to destroy, matching the scratch plan on #252 (#252 (comment)):

  • aws_ecs_service.fargate[0]: update in place. If the plan shows it replaced, do not merge.
  • aws_ecs_task_definition.task: replaced, as a new revision with bridge networking and hostPort 0.
  • aws_lb_target_group.this: people-depot-backend-dev-145 (ip) replaced by people-depot-backend-dev-ab6 (instance), created before the old one is destroyed.
  • aws_lb_listener_rule.static: updated in place to forward to the new group.
  • aws_security_group.container[0] and its two rules: destroyed.

The security group's deletion may be slow while the old awsvpc task's ENI detaches; the provider retries DependencyViolation.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Terraform plan in terraform
With backend config files: terraform/prod.backend.tfvars

Plan: 2 to add, 2 to change, 5 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place
-   destroy
-/+ destroy and then create replacement
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.people-depot.module.backend_dev_service.aws_ecs_service.fargate[0] will be updated in-place
!~  resource "aws_ecs_service" "fargate" {
        id                                 = "arn:aws:ecs:us-west-2:035866691871:service/incubator-prod/people-depot-backend-dev"
        name                               = "people-depot-backend-dev"
        tags                               = {
            "project" = "people-depot"
        }
!~      task_definition                    = "arn:aws:ecs:us-west-2:035866691871:task-definition/people-depot-backend-dev:24" -> (known after apply)
#        (18 unchanged attributes hidden)

-       load_balancer {
-           container_name   = "people-depot-backend-dev" -> null
-           container_port   = 8000 -> null
-           target_group_arn = "arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/people-depot-backend-dev-145/86e5c6761fe5a0b0" -> null
#            (1 unchanged attribute hidden)
        }
+       load_balancer {
+           container_name   = "people-depot-backend-dev"
+           container_port   = 8000
+           target_group_arn = (known after apply)
#            (1 unchanged attribute hidden)
        }

-       network_configuration {
-           assign_public_ip = false -> null
-           security_groups  = [
-               "sg-02ea7fea37446badd",
            ] -> null
-           subnets          = [
-               "subnet-03ed55f60a6c28e72",
-               "subnet-089e80a53e1522e28",
            ] -> null
        }

#        (5 unchanged blocks hidden)
    }

  # module.people-depot.module.backend_dev_service.aws_ecs_task_definition.task must be replaced
-/+ resource "aws_ecs_task_definition" "task" {
!~      arn                      = "arn:aws:ecs:us-west-2:035866691871:task-definition/people-depot-backend-dev:24" -> (known after apply)
!~      arn_without_revision     = "arn:aws:ecs:us-west-2:035866691871:task-definition/people-depot-backend-dev" -> (known after apply)
!~      container_definitions    = (sensitive value) # forces replacement
!~      enable_fault_injection   = false -> (known after apply)
!~      id                       = "************************" -> (known after apply)
!~      network_mode             = "awsvpc" -> "bridge" # forces replacement
!~      revision                 = 24 -> (known after apply)
        tags                     = {
            "project" = "people-depot"
        }
#        (12 unchanged attributes hidden)
    }

  # module.people-depot.module.backend_dev_service.aws_lb_listener_rule.static will be updated in-place
!~  resource "aws_lb_listener_rule" "static" {
        id           = "arn:aws:elasticloadbalancing:us-west-2:035866691871:listener-rule/app/incubator-prod-lb/7451adf77133ef36/390a225766a4daf3/bb0f094e084455bb"
        tags         = {}
#        (5 unchanged attributes hidden)

!~      action {
!~          target_group_arn = "arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/people-depot-backend-dev-145/86e5c6761fe5a0b0" -> (known after apply)
#            (2 unchanged attributes hidden)
        }

#        (2 unchanged blocks hidden)
    }

  # module.people-depot.module.backend_dev_service.aws_lb_target_group.this must be replaced
+/- resource "aws_lb_target_group" "this" {
!~      arn                                = "arn:aws:elasticloadbalancing:us-west-2:035866691871:targetgroup/people-depot-backend-dev-145/86e5c6761fe5a0b0" -> (known after apply)
!~      arn_suffix                         = "targetgroup/people-depot-backend-dev-145/86e5c6761fe5a0b0" -> (known after apply)
+       connection_termination             = (known after apply)
!~      id                                 = "*************************************************************************************************************" -> (known after apply)
!~      ip_address_type                    = "ipv4" -> (known after apply)
!~      load_balancer_arns                 = [
-           "arn:aws:elasticloadbalancing:us-west-2:035866691871:loadbalancer/app/incubator-prod-lb/7451adf77133ef36",
        ] -> (known after apply)
!~      load_balancing_algorithm_type      = "round_robin" -> (known after apply)
!~      load_balancing_anomaly_mitigation  = "off" -> (known after apply)
!~      load_balancing_cross_zone_enabled  = "use_load_balancer_configuration" -> (known after apply)
!~      name                               = "people-depot-backend-dev-145" -> "people-depot-backend-dev-ab6" # forces replacement
+       name_prefix                        = (known after apply)
+       preserve_client_ip                 = (known after apply)
!~      protocol_version                   = "HTTP1" -> (known after apply)
-       tags                               = {} -> null
-       target_control_port                = 0 -> null
!~      target_type                        = "ip" -> "instance" # forces replacement
#        (9 unchanged attributes hidden)

!~      health_check {
!~          timeout             = 5 -> (known after apply)
#            (8 unchanged attributes hidden)
        }

!~      stickiness (known after apply)
-       stickiness {
-           cookie_duration = 86400 -> null
-           enabled         = false -> null
-           type            = "lb_cookie" -> null
#            (1 unchanged attribute hidden)
        }

!~      target_failover (known after apply)
-       target_failover {}

!~      target_group_health (known after apply)
-       target_group_health {
-           dns_failover {
-               minimum_healthy_targets_count      = "1" -> null
-               minimum_healthy_targets_percentage = "off" -> null
            }
-           unhealthy_state_routing {
-               minimum_healthy_targets_count      = 1 -> null
-               minimum_healthy_targets_percentage = "off" -> null
            }
        }

!~      target_health_state {
+           enable_unhealthy_connection_termination = false
+           unhealthy_draining_interval             = 0
        }
    }

  # module.people-depot.module.backend_dev_service.aws_security_group.container[0] will be destroyed
  # (because index [0] is out of range for count)
-   resource "aws_security_group" "container" {
-       arn                    = "arn:aws:ec2:us-west-2:035866691871:security-group/sg-02ea7fea37446badd" -> null
-       description            = "Container people-depot-backend-dev" -> null
-       egress                 = [
-           {
-               cidr_blocks      = [
-                   "0.0.0.0/0",
                ]
-               from_port        = 0
-               ipv6_cidr_blocks = []
-               prefix_list_ids  = []
-               protocol         = "-1"
-               security_groups  = []
-               self             = false
-               to_port          = 0
#                (1 unchanged attribute hidden)
            },
        ] -> null
-       id                     = "sg-02ea7fea37446badd" -> null
-       ingress                = [
-           {
-               cidr_blocks      = [
-                   "10.10.0.0/16",
                ]
-               from_port        = 8000
-               ipv6_cidr_blocks = []
-               prefix_list_ids  = []
-               protocol         = "tcp"
-               security_groups  = []
-               self             = false
-               to_port          = 8000
#                (1 unchanged attribute hidden)
            },
        ] -> null
-       name                   = "ecs-container-people-depot-backend-dev" -> null
-       owner_id               = "035866691871" -> null
-       region                 = "us-west-2" -> null
-       revoke_rules_on_delete = false -> null
-       tags                   = {
-           "Name" = "ecs-container-people-depot-backend-dev"
        } -> null
-       tags_all               = {
-           "Name"       = "ecs-container-people-depot-backend-dev"
-           "managed-by" = "terraform-incubator"
        } -> null
-       vpc_id                 = "vpc-0bec93a4d80243845" -> null
#        (1 unchanged attribute hidden)
    }

  # module.people-depot.module.backend_dev_service.aws_vpc_security_group_egress_rule.allow_all_traffic[0] will be destroyed
  # (because index [0] is out of range for count)
-   resource "aws_vpc_security_group_egress_rule" "allow_all_traffic" {
-       arn                    = "arn:aws:ec2:us-west-2:035866691871:security-group-rule/sgr-03d942deb0807cc0f" -> null
-       cidr_ipv4              = "0.0.0.0/0" -> null
-       id                     = "sgr-03d942deb0807cc0f" -> null
-       ip_protocol            = "-1" -> null
-       region                 = "us-west-2" -> null
-       security_group_id      = "sg-02ea7fea37446badd" -> null
-       security_group_rule_id = "sgr-03d942deb0807cc0f" -> null
-       tags_all               = {
-           "managed-by" = "terraform-incubator"
        } -> null
    }

  # module.people-depot.module.backend_dev_service.aws_vpc_security_group_ingress_rule.container_ingress_port[0] will be destroyed
  # (because index [0] is out of range for count)
-   resource "aws_vpc_security_group_ingress_rule" "container_ingress_port" {
-       arn                    = "arn:aws:ec2:us-west-2:035866691871:security-group-rule/sgr-05275a0fabb94fbe3" -> null
-       cidr_ipv4              = "10.10.0.0/16" -> null
-       from_port              = 8000 -> null
-       id                     = "sgr-05275a0fabb94fbe3" -> null
-       ip_protocol            = "tcp" -> null
-       region                 = "us-west-2" -> null
-       security_group_id      = "sg-02ea7fea37446badd" -> null
-       security_group_rule_id = "sgr-05275a0fabb94fbe3" -> null
-       tags_all               = {
-           "managed-by" = "terraform-incubator"
        } -> null
-       to_port                = 8000 -> null
    }

Plan: 2 to add, 2 to change, 5 to destroy.

✅ Plan applied in Terraform apply (OIDC) #101

@ale210
ale210 merged commit da54eec into main Oct 10, 2026
3 checks passed
ale210 added a commit that referenced this pull request Oct 10, 2026
For #253. This is the second PR, moving the remaining five
non-production services to bridge networking after the
`people-depot-backend-dev` pilot (#260) checked out.

Adds `network_mode = "bridge"` to:

- `civic-tech-index-backend-stage`
- `civic-tech-jobs-fullstack-stage`
- `home-unite-us-fullstack-qa`
- `vrms-backend-dev`
- `vrms-frontend-dev`

**Each service will be unreachable for about two minutes during the
apply.** Each project's team should be told before this merges.

**Expected plan: 10 to add, 10 to change, 25 to destroy.** That is five
copies of the pilot's 2/2/5. For each service: ECS service **updated in
place**, task definition replaced, target group replaced by an
`instance`-type one, listener rule updated in place, and the security
group plus its two rules destroyed. If any ECS service shows as
replaced, do not merge.

`vrms-frontend-dev` has been at 0/1, failing to start, since before this
epic began. Moving it changes nothing for users. It will not pass the
post-merge "1/1 and answers at its hostname" check, and that failure is
not caused by this PR.

Pilot results: the service was updated in place, its target became
healthy on a dynamic host port, and from inside the task an instance
metadata token request times out while the task's own ECS credentials
still work.
ale210 added a commit that referenced this pull request Oct 10, 2026
For #254. Moves the four production services to bridge networking in one
PR:

- `vrms-backend-prod`, `vrms-frontend-prod`
- `civic-tech-index-backend-prod`
- `home-unite-us-fullstack-prod`

**Merge this in a window agreed with all three teams.** It applies as
one run, so all four production sites are down at the same time, for one
to two minutes each, while their new target groups pass health checks.
Users see a `503` from the load balancer during that time. This replaces
the ticket's one-PR-per-project plan.

**Expected plan: 8 to add, 8 to change, 20 to destroy.** That is four
copies of the 2/2/5 seen on the six non-production services (#260,
#261). For each service: ECS service **updated in place**, task
definition replaced, target group replaced by an `instance`-type one,
listener rule updated in place, and the security group plus its two
rules destroyed. If any ECS service shows as replaced, do not merge.

`home-unite-us/environment-prod.tf` also gets a one-space alignment
change on `launch_type`, from `terraform fmt`; that file was fmt-clean
before.

`vrms-backend-prod` is OOM-killed roughly every 30 minutes (#174). A
restart around the move is that bug, not this change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant