Logs not forwarding from GitLab to CloudWatch

[dsotirho-ucsc]

dev

anvildev

GitLab dev

From /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log:

2026-07-08T00:19:38Z E! [outputs.cloudwatchlogs] InvalidParameterException: 1 validation error detected: Value at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+, will not retry the request

It appears the CloudWatch agent isn't setup correctly on GitLab dev & anvildev, specifically in the agent's config the log_group_name fields contains a variable name instead of the actual value.

Compare GitLab dev's /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml:

    [[inputs.logfile.file_config]]
      deployment_environment = ""
      file_path = "/var/log/amazon/ssm/amazon-ssm-agent.log"
      from_beginning = true
      log_group_class = ""
      log_group_name = "${aws_cloudwatch_log_group.gitlab_cwagent.name}"
      log_stream_name = "/var/log/amazon/ssm/amazon-ssm-agent.log"
      pipe = false
      retention_in_days = -1
      service_name = ""

... to GitLab anvilprod's /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml:

    [[inputs.logfile.file_config]]
      deployment_environment = ""
      file_path = "/var/log/amazon/ssm/amazon-ssm-agent.log"
      from_beginning = true
      log_group_class = ""
      log_group_name = "/aws/cwagent/azul-gitlab"
      log_stream_name = "/var/log/amazon/ssm/amazon-ssm-agent.log"
      pipe = false
      retention_in_days = -1
      service_name = ""

[nadove-ucsc]

Regression introduced in 7fc92c0 -- compressing the user_data prevents Terraform from expanding the substitutions, leading to the string literal Daniel observed. There were actually two expansions affected by this:

azul/terraform/gitlab/gitlab.tf.json.template.py

Line 1860 in 3634a58

'--hostname ${aws_route53_record.gitlab.name}',

and the one that caused the log forwarding failure:

azul/terraform/gitlab/gitlab.tf.json.template.py

Line 2173 in 3634a58

'log_group_name': '${aws_cloudwatch_log_group.gitlab_cwagent.name}',

[nadove-ucsc]

There's a straightforward fix. Thus spake Claude:

🤖 🤖 🤖

The fix: both of these Terraform refs turn out to be resolvable statically, in Python, without waiting on Terraform:

• aws_cloudwatch_log_group.gitlab_cwagent.name is a literal Python string already: '/aws/cwagent/azul-gitlab' (gitlab.tf.json.template.py:1257) — not an AWS-assigned attribute.
• aws_route53_record.gitlab.name reduces (via departition(None, '.', f'gitlab.{config.domain_name}')) to f'gitlab.{config.domain_name}' — also fully known to Python at template-generation time.

So rather than routing these through Terraform interpolation (which the compression step now defeats), I'd introduce two small module-level variables next to the others at line ~316-326 (e.g. gitlab_fqdn = f'gitlab.{config.domain_name}' and gitlab_cwagent_log_group = '/aws/cwagent/azul-gitlab'), use them to define the two resources, and substitute the same Python variables
directly into the cloud-init content instead of the ${...} refs. That avoids relying on Terraform to interpolate anything inside compressed data at all, and keeps the two definitions from drifting apart.

The alternative — wrapping the whole cloud-config string in Terraform's base64gzip() function so compression happens at apply-time after interpolation — would work too, but requires careful HCL-string-literal escaping across ~600 lines of generated YAML (backslashes, quotes, literal $ sequences), which is riskier for comparatively little benefit here since both values are
already static.

🤖 🤖 🤖

[dsotirho-ucsc]

For demo, show logs forwarded to the /aws/cwagent/azul-gitlab log group on dev and anvildev