Harden EKS demo configs and manifests

[ryanmaclean]

Summary

This update modernizes and hardens the EKS demo repository so fresh deployments are safer, more reproducible, and easier to operate. The impact is reduced security risk (secret handling, privileged runtime settings, API exposure defaults), fewer deployment surprises, and better day-2 reliability for demo workloads.

Follow-up increments harden the CloudFormation bootstrap path by removing risky defaults, updating an end-of-life Lambda runtime, clarifying legacy ALB ingress behavior, and making nested stack orchestration more deterministic. This increment also improves IaC tooling guidance by prioritizing OpenTofu for a more permissive licensing path while retaining Terraform compatibility.

Changes

• Refreshed and restructured README workflow for provisioning, Datadog install, validation, and cleanup.
• Improved eksctl/script.sh with strict shell mode and explicit cluster/region/version/nodegroup parameters via environment-variable overrides.
• Hardened operator deployment flow:
  • Added strict shell mode and API key presence check.
  • Switched to idempotent secret creation (kubectl ... --dry-run=client | apply).
  • Stopped rendering API keys into manifest files and moved cluster agent API key to secretKeyRef.
  • Applied manifests using script-relative paths.
• Updated operator/cluster-agent.yaml:
  • Removed invalid pod-template metadata.name.
  • Switched image pull policy from Always to IfNotPresent.
  • Replaced inline DD_API_KEY value with Kubernetes secret reference.
• Hardened demo app manifests under storedog/:
  • Removed creationTimestamp/status generated fields from source manifests.
  • Changed DB secret from data to stringData for valid and explicit secret rendering.
  • Removed privileged DB container setting.
  • Added readiness/liveness probes and CPU/memory requests/limits.
  • Disabled Flask debug mode for service deployments.
  • Changed workload image pull policy to IfNotPresent where appropriate.
• Hardened Datadog Helm values:
  • Enabled kubelet TLS verification (DD_KUBELET_TLS_VERIFY=true).
• Modernized Terraform security and compatibility:
  • Added required_providers with version constraints and raised Terraform minimum version.
  • Switched external IP discovery to HTTPS endpoint.
  • Added optional workstation_external_cidr variable override.
  • Explicitly restricted EKS API public access CIDR to workstation CIDR.
  • Updated kubeconfig exec auth to aws eks get-token with client.authentication.k8s.io/v1beta1.
  • Marked sensitive outputs (kubeconfig, config_map_aws_auth) as sensitive.
• CloudFormation bootstrap hardening:
  • Changed ProvisionBastionHost default from Enabled to Disabled.
  • Updated custom resource Lambda runtime from python3.7 to python3.11.
  • Clarified ProvisionALBIngressController as a legacy ALB ingress path in template parameter description.
  • Replaced ineffective Metadata.DependsOn blocks with resource-level DependsOn declarations on nested stacks to enforce deterministic orchestration.
• Updated README CloudFormation guidance:
  • Added required-input clarity for stack creation.
  • Added legacy ALB ingress behavior and phased migration guidance.
  • Switched kubeconfig instructions to region/cluster-name variables and added CloudFormation output lookup command for autogenerated cluster names.
• Updated README IaC guidance to prefer OpenTofu while keeping Terraform compatibility:
  • Renamed Terraform section to OpenTofu (Terraform-compatible).
  • Switched init/apply command examples from terraform to tofu.
  • Added explicit note that terraform can be used in place of tofu.
  • Updated IaC reference link to current AWS provider example path.
• Investigation notes captured in Datadog notebook for session memory:
  • https://app.datadoghq.com/notebook/14457462

Testing

• Ran repo formatter tool:
  • hclfmt applied to Terraform changes (terraform/eks-cluster.tf).
• Ran repo linter tool:
  • tflint passed for modified Terraform files.
• Performed YAML parse validation for updated manifests:
  • operator/cluster-agent.yaml
  • operator/datadog-cluster-agent_service.yaml
  • storedog/advertisements.yaml
  • storedog/db.yaml
  • storedog/discounts.yaml
  • storedog/frontend.yaml
  • helm/values.yaml
• Performed shell syntax validation:
  • bash -n eksctl/script.sh
  • bash -n operator/operator_deploy.sh
• Performed CloudFormation template syntax parse validation:
  • python yaml.compose against cloudformation/amazon-eks-template.yaml
• Verified removal of ineffective metadata dependency patterns:
  • Metadata.DependsOn no longer present in cloudformation/amazon-eks-template.yaml
• Current increment validation:
  • Ran repo Format and Lint tools (README is not auto-processed by configured format/lint tooling in this repo).
  • Manually verified README command/path consistency for OpenTofu and Terraform compatibility guidance.

---

PR by Bits - View session in Datadog

Comment @DataDog to request changes

[datadog-rm]

View session in Datadog

Bits Dev status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

[datadog-rm]

I can only run on private repositories.