Skip to content

fix(router): authenticate worker-image pulls against private registries - #1539

Merged
zbigniewsobiecki merged 3 commits into
devfrom
fix/worker-image-pull-registry-auth
Aug 25, 2026
Merged

zbigniewsobiecki merged 3 commits into
devfrom
fix/worker-image-pull-registry-auth

Conversation

@zbigniewsobiecki

Copy link
Copy Markdown
Member

Problem

Every router-side registry pull — the spawn self-heal in container-manager.ts, worker-image validation, and the Dockerfile-build base refresh — funnels through pullImageOnce, whose docker.pull() carries no credentials: dockerode sends nothing unless given an explicit authconfig, the Docker daemon has no ambient login, and ~/.docker/config.json is a CLI-side concept the Engine API never reads.

Against a private registry every such pull 401s. Verified live 2026-08-20: a weekly host docker image prune --all removed the worker image (nothing references it between ephemeral worker runs), and every subsequent spawn failed instantly with:

Worker spawn failed: Error: (HTTP code 401) unexpected - Head "https://ghcr.io/v2/mongrel-intelligence/cascade-worker/manifests/latest": unauthorized

The self-heal path could never have worked for private images — it only appeared to work while deploy-time pulls kept the image present on the host.

Fix

New optional router env (documented in CLAUDE.md, src/router/registry-auth.ts):

  • WORKER_IMAGE_REGISTRY_USERNAME / WORKER_IMAGE_REGISTRY_PASSWORD — passed as dockerode authconfig on every pullImageOnce pull (for ghcr.io a PAT with read:packages is the password).
  • WORKER_IMAGE_REGISTRY_SERVER — optional override for the serveraddress; otherwise derived from the image ref per Docker's reference grammar (first segment with a dot/port/localhost; Docker Hub https://index.docker.io/v1/ for bare refs).

Behavior is unchanged when unset (anonymous pulls). Partial credentials (only one of the pair) log a warning and stay anonymous, so public-image pulls keep working while the private-image 401 stays attributable.

This matters more as per-project worker images (specs 022/023) put more registry pulls on the router path.

Tests

  • New tests/unit/router/registry-auth.test.ts — host derivation (ghcr / port / localhost / hub namespaced / bare ref), credential gating (none → anonymous, partial → warn + anonymous, full → authconfig), server override, routerConfig defaulting.
  • pullImageOnce pins: authconfig pass-through for private registries, anonymous {} default.
  • Full unit suite green (11,082 passed).

🤖 Generated with Claude Code

All router-side registry pulls (spawn self-heal, worker-image validation,
Dockerfile-build base refresh) funnel through pullImageOnce, whose dockerode
pull sends no credentials — the daemon has no ambient login and never reads
~/.docker/config.json. Anonymous pulls 401 on private registries, so a host
image prune that removed the worker image turned every spawn into
`Head .../manifests/latest: unauthorized` (verified live 2026-08-20).

New optional env: WORKER_IMAGE_REGISTRY_USERNAME / _PASSWORD (+ _SERVER
override; host otherwise derived from the image ref, Docker Hub as the
no-host fallback). Unset preserves anonymous pulls; partial credentials warn
and stay anonymous.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.36842% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/router/registry-auth.ts 96.55% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

CI's `npm audit --omit=dev --audit-level=high` step began failing on the
newly disclosed CVE-2026-59870 (GHSA-5p4m-2wfm-xmqj): quadratic CPU
consumption in js-yaml's !!omap resolution, affecting >=4.0.0 <4.3.1.
js-yaml is a direct production dependency; the 4.3.1 patch release fixes
it and all transitive consumers (llmist, @llmist/cli) dedupe to it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@nhopeatall nhopeatall left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

APPROVE — Correct, well-scoped fix for anonymous private-registry 401s, with a single funnel point (pullImageOnce), deterministic auth resolution, fail-safe partial-credential handling, and thorough tests. Verified locally: 121/121 affected router tests pass, typecheck clean, CI 7/7 green.

What I verified

  • Single funnel holds. All three router-side pulls route through pullImageOnce: spawn self-heal (container-manager.ts:224), worker-image validation (worker-image-validation.ts:155), and the Dockerfile-build base refresh (worker-image-build.ts:279 via resolveBaseImageRef). No other docker.pull / createImage call sites exist, so the auth applies everywhere it needs to.
  • registryHostFromImageRef matches Docker's reference grammar. Host-detection (dot / port-colon / localhost, gated on presence of /) correctly distinguishes ghcr.io/..., host:port/..., localhost:5000/... from Docker Hub refs (library/redis:7, redis:7-alpine). Tag-colon-vs-port ambiguity is handled.
  • Gating is fail-safe and non-leaky. none → anonymous, partial → warn + anonymous (public pulls keep working, private 401 stays attributable), full → authconfig. The warning logs only imageRef; credentials are never logged.
  • Backward compatible. docker.pull(image, {}) is behavior-identical to the prior docker.pull(image); the updated pins in worker-snapshots/container-manager/snapshot-integration tests confirm the anonymous default.
  • js-yaml bump is a deliberate security fix (separate commit clearing a high-severity audit failure), not unrelated churn.
  • Documentation is updated in the canonical place (CLAUDE.md); .env.example is intentionally non-exhaustive (omits PM_COALESCE_WINDOW_MS, SLOT_WAIT_TIMEOUT_MS, etc.), so leaving the new vars out of it is consistent with convention.

Non-blocking note

  • Credentials are attached to every worker-image pull, with serveraddress derived per-ref. In today's architecture every router pull targets the single configured registry, and worker-image refs are superadmin-only (projects.ts:142), so this stays within a trusted boundary. Worth keeping in mind as per-project reference images (specs 022/023) proliferate: if a project's image ever lives on a different public registry than the configured credentials, the pull would send mismatched creds and 401 rather than falling back to anonymous. A future server/host match-gate could scope the auth to the intended registry. Not required for this PR.

🕵️ claude-code · claude-opus-4-8 · run details

@nhopeatall

nhopeatall commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator

⚠️ review agent failed

Error: Claude Code returned an error result: You've hit your weekly limit · resets 11pm (UTC)

Manual intervention may be required.

@zbigniewsobiecki
zbigniewsobiecki merged commit aee88e2 into dev Aug 25, 2026
9 checks passed
@zbigniewsobiecki
zbigniewsobiecki deleted the fix/worker-image-pull-registry-auth branch August 25, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants