[GA blocker] Operator runtime image is built on devcontainer base, not distroless

[WentingWu666666]

Summary

The operator container image is built on mcr.microsoft.com/devcontainers/base:alpine — the devcontainer base image used by VS Code for development environments. It ships with a full shell, apk, git, vim, sudo, ssh, curl, wget, and a few hundred packages the operator binary does not need at runtime.

For a static Go binary (CGO_ENABLED=0 GOEXPERIMENT=nosystemcrypto, already produced by the existing build stage) the appropriate runtime base is a distroless image such as mcr.microsoft.com/cbl-mariner/distroless/base:2.0 (or the newer mcr.microsoft.com/azurelinux/distroless/base:3.0, staying in the same Microsoft image lineage as the builder).

Current state

operator/src/Dockerfile:

FROM mcr.microsoft.com/oss/go/microsoft/golang:1.25-azurelinux3.0 AS builder
...
RUN GOEXPERIMENT=nosystemcrypto CGO_ENABLED=0 GOOS=... GOARCH=... go build -a -o manager cmd/main.go

# Use Microsoft official image from MCR as minimal base image to package the manager binary
FROM mcr.microsoft.com/devcontainers/base:alpine
WORKDIR /
COPY --from=builder /workspace/manager .
RUN addgroup -S manager && adduser -S manager -G manager
USER manager

The "minimal base" comment is misleading — devcontainers/base:alpine is a development environment image, not a minimal production runtime.

Why this matters for GA

Concern	Today	After distroless switch
Runtime base size	~300–500 MB	~5–10 MB
Pre-installed shell	bash, sh, dash	none
Package manager present	yes (apk)	none
Network tools	curl, wget, ssh	none
CVE surface from base packages	hundreds	~3 (ca-certificates, tzdata, libc)
Trivy / Defender scan noise	constant Alpine CVE churn	near-zero
Container-escape primitives at runtime	many	none
kubectl exec -it ... -- sh post-RCE	works (attacker dream)	no shell to exec
runAsNonRoot: true semantics	hacky — Alpine adduser -S manager produces an unverifiable UID, requiring an explicit runAsUser: 100 workaround in PR #382 to even pass kubelet admission	clean — distroless images declare USER 65532:65532 (nonroot) by default and kubelet can verify it

Real-world impact

1. Customer security reviews will flag this. Enterprise procurement / SOC2 / FedRAMP / customer security questionnaires routinely ask for the runtime base image and how the attack surface is minimized. Shipping vim, git, and apk in the operator runtime is hard to defend.
2. CVE noise becomes a support treadmill at GA. Customers running Trivy / Microsoft Defender for Containers / Snyk against AKS clusters with this operator will file CVE tickets weekly. Most will not be exploitable; the support cost of triaging "Alpine CVE-X is in your operator image" repeatedly is large.
3. It is a fixed-cost, one-shot migration. Done before GA, it is a few-line Dockerfile change. Done after GA, it becomes a coordinated rollout that some customers will resist on the grounds of "you changed our image."
4. CNPG already does this. The operator's most important upstream dependency ships on a distroless base, with readOnlyRootFilesystem: true, no shell, UID 10001. Customers comparing the two side-by-side notice.

Suggested fix

Replace the runtime stage in operator/src/Dockerfile with something like:

FROM mcr.microsoft.com/cbl-mariner/distroless/base:2.0
WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532
ENTRYPOINT ["/manager"]

(or mcr.microsoft.com/azurelinux/distroless/base:3.0 to match the Azure Linux lineage used by the builder stage).

Companion changes:

• Remove the adduser / addgroup lines — distroless ships a nonroot user (UID 65532) already.
• Update the operator chart's securityContext.runAsUser from 100 (the Alpine adduser -S workaround introduced in feat(helm): add pod security, resources, and scheduling knobs to chart #382) to 65532.
• Update securityContext.runAsGroup to 65532 for symmetry.
• Apply the same base-image swap to other Dockerfiles in the repo where applicable:
  • operator/cnpg-plugins/sidecar-injector/Dockerfile
  • any future wal-replica Dockerfile when it lands
  • the gateway image is built downstream from documentdb-local so is out of scope here
• Update the security-context documentation under docs/operator-public-documentation/ to reflect the new UID and to advertise the distroless story (helps with the customer-security-questionnaire pain).

Test plan

• Build the new image locally and docker run --rm <image> --help to confirm the binary starts under the new base.
• Verify TLS connections to the API server still work (needs ca-certificates; both cbl-mariner/distroless/base:2.0 and azurelinux/distroless/base:3.0 include them).
• Smoke test in kind: install the chart with the new image, confirm operator pod becomes Ready, create a sample DocumentDB CR, confirm reconciliation succeeds.
• Re-run the existing E2E suite.
• Re-run the existing image scan in CI (if any) and confirm CVE count drops.

Out of scope

• Switching the base for the database / gateway images (they are produced from a different upstream pipeline — see Dockerfile_gateway_deb).
• Pod-spec hardening for the operator container — already done in feat(helm): add pod security, resources, and scheduling knobs to chart #382.
• Runtime-pod (cluster pod) sidecar hardening — tracked in [GA blocker] Runtime cluster pods are not Pod Security Admission "restricted" compliant #387.

References

• Follow-up to Helm chart improvements for GA readiness #381 (umbrella GA chart audit)
• Companion to feat(helm): add pod security, resources, and scheduling knobs to chart #382 (operator container securityContext hardening — became necessary partly because the Alpine base lacked a verifiable nonroot user)
• Companion to [GA blocker] Runtime cluster pods are not Pod Security Admission "restricted" compliant #387 (runtime cluster pod PSA restricted gap on injected sidecars)
• Distroless images
• Azure Linux distroless images

[WentingWu666666]

Follow-up: enable readOnlyRootFilesystem: true as part of this migration

Migrating to distroless makes containerSecurityContext.readOnlyRootFilesystem: true essentially free for the operator, sidecar-injector, and (when its image lands) wal-replica plugin. We should turn it on in the same PR that swaps the base image.

Why couple it to the distroless swap:

• Distroless ships no shell, no package manager, no log rotators, no ~/.bash_history, no apk cache virtually nothing in the image expects a writable rootfs. On the current Alpine devcontainer base, enabling readOnlyRootFilesystem would silently break kubectl exec debugging workflows and any apk/shell behavior that touches /var, /etc, /root. On distroless those concerns don't exist.
• The operator binary is built CGO_ENABLED=0 static it only needs scratch space for Go runtime temp files (typically /tmp) and nothing else.
• PSA restricted recommends but does not require readOnlyRootFilesystem. It's the single biggest hardening field left on the table after PR feat(helm): add pod security, resources, and scheduling knobs to chart #382. It neutralizes the most common post-exploitation step: dropping a binary or persistence script on disk.

What the PR for #388 should include (in addition to the base image swap):

1. Add readOnlyRootFilesystem: true to containerSecurityContext in operator/documentdb-helm-chart/values.yaml for all three components (operator, sidecarInjector, walReplicaPlugin).

2. Add an emptyDir volume named tmp and mount it at /tmp for each Deployment that needs scratch space. CNPG's pattern (calls theirs scratch-data):

   volumes:
     - name: tmp
       emptyDir: {}
   containers:
     - volumeMounts:
         - name: tmp
           mountPath: /tmp

3. Live-test on kind: deploy a few DocumentDB CRs under reconcile load and confirm no read-only file system errors in operator/sidecar-injector logs. If anything else fails, add a scoped mount for that specific path do not widen the writable surface to the whole rootfs.

4. Apply the same readOnlyRootFilesystem: true to operator/cnpg-plugins/sidecar-injector/internal/lifecycle/lifecycle.go (gateway and otel-collector containers) tracked separately in [GA blocker] Runtime cluster pods are not Pod Security Admission "restricted" compliant #387, but the two PRs should land close together so the hardening posture is consistent across chart-shipped and injected pods.

Result after both #388 and this follow-up land: every container shipped by the chart will be PSA restricted compliant and have readOnlyRootFilesystem: true, matching CNPG's operator posture exactly.

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: bug
Project fields suggested: Component build · Priority P1 · Effort S · Status Triage
Confidence: 0.92 (llm)

Reasoning

LLM: GA-blocking security/hardening issue: the operator runtime image uses a bloated devcontainer base instead of a distroless image, dramatically expanding the CVE attack surface and failing security review standards; fix is a small Dockerfile change.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.