fix(server): keep BuildKit token seeds off an unwritable docker config dir

[vmrm]

Summary

The BuildKit auth provider persists registry token seeds under the docker config dir and creates that directory on the first token request. A process without a writable home therefore fails every pull, anonymous ones included, with mkdir /.docker: read-only file system. A builtin backend runs exactly like that. Read the config file from the default location as before, then move only the seeds to a directory under os.TempDir() when the default dir cannot be created.

Key changes

• server/pkg/docker/buildkit.go: buildkitSessionAttachables reads config.json from the default location, then useWritableDockerConfigDirForTokenSeeds tries os.MkdirAll(config.Dir()) and, on failure, creates a directory private to this build with os.MkdirTemp (mode 0700) under the memory-backed /dev/shm where it exists, else under os.TempDir(), and points config.SetDir at it for exactly as long as authprovider.NewDockerAuthProvider needs to capture config.Dir(); the original directory is restored afterwards and a mutex serialises the window. The function now also returns a cleanup that removes the seed directory, which buildWithBuildkitClient defers, so nothing outlives the build and mounts sharing a process (several trdl engines, in one namespace or across namespaces) share nothing through it.
• server/pkg/docker/buildkit_config_dir_test.go: the tests drive the returned provider through GetTokenAuthority, the call buildkitd makes on a bearer challenge, and watch the candidate base directories: a writable config dir is kept and used and no seed dir appears; an unwritable one (a path under a regular file) leaves config.Dir() untouched after the call, one private 0700 dir with .token_seed appears and is gone after cleanup; eight concurrent builds get eight distinct dirs, all removed afterwards (run under -race).

Why

Observed on a Stronghold 1.19 stand where trdl is a builtin secrets engine: readOnlyRootFilesystem: true, no HOME, only /tmp mounted writable. With #426 applied the release reaches the build, the kubernetes builder pod starts, and the solve fails on #1 [internal] load remote build context with mkdir /.docker: read-only file system — tokenSeeds.getSeed in session/auth/authprovider/tokenseed.go does os.MkdirAll(config.Dir()), and config.Dir() resolves to /.docker there. Both BuildKit client paths (buildkitd_address and the kubernetes driver) attach this provider, so both fail the same way; the docker CLI path does not use it.

An external plugin never hit this: it runs with a home directory, and the e2e flow runs it on a workstation or a CI runner.

Review focus / risks

• config.SetDir is process-global state in docker/cli; the redirect is held only across provider construction, under dockerConfigDirMu, and restored, so later LoadDefaultConfigFile calls (the next build, or any other consumer in a builtin process) keep reading the default location. The first cut of this PR did not restore it and the independent review caught that; see the comments.
• Token seeds are per-host random values BuildKit regenerates when missing. A fresh seed per build means that on a shared buildkitd (buildkitd_address) the daemon cannot link a new build to the registry tokens a previous build fetched, so tokens are fetched per build; with the kubernetes driver there is a daemon per build and nothing to link. In /dev/shm the seed never reaches a disk. The file is neither a token nor a credential (see the code comment).
• When even os.MkdirTemp fails the redirect is skipped with a log line and BuildKit's original mkdir error surfaces unchanged.

🤖 Generated with Claude Code

[vmrm]

Verification

• task --yes server:lint — green (golangci-lint v2.11.0 + prettier), macOS arm64.
• task --yes server:test:unit — green, macOS arm64; the three new tests listed below pass.

Mutation evidence

Each row is applied on top of the fixed tree and the unit suite is rerun; the tree is restored from git and checked clean (git diff --quiet HEAD) after every round.

Mutation	Expected evidence	Result
drop config.SetDir(fallback) in dockerConfigDirForTokenSeeds	TestDockerConfigDirForTokenSeeds_FallsBackToTempDirWhenUnwritable and TestBuildkitSessionAttachables_MovesTokenSeedsOffUnwritableConfigDir fail on config.Dir(); …_KeepsWritableDir stays green	Killed, exactly those two
drop the dockerConfigDirForTokenSeeds(ctx) call in buildkitSessionAttachables	only TestBuildkitSessionAttachables_MovesTokenSeedsOffUnwritableConfigDir fails	Killed, exactly that one

The unwritable directory in the tests is a path under a regular file, so the check fails with ENOTDIR for root as well as for an ordinary user.

Not run

• The real trigger (BuildKit calling tokenSeeds.getSeed during a solve from a process with a read-only root and no home) is not reproduced in this repository's tests; the observed failure on a Stronghold stand with fix(server): read task storage from the tasks manager, not the request #426 applied is the field evidence, and the green run of the same stand on a build carrying this commit is pending and will be reported here.
• task e2e:test:e2e:flow-vault was not re-run locally; on a workstation the default dir is writable and the new code path is a no-op there. CI runs the e2e jobs.

[vmrm]

Green run on the stand

Stronghold release-1.19 with #426 and this commit compiled in (vmrm/trdl@b50a464): readOnlyRootFilesystem: true, no HOME, /tmp an emptyDir.

• Before this commit (same stand, fix(server): read task storage from the tasks manager, not the request #426 only): task 6b89b4a5… reached the build, the kubernetes builder pod started, and the solve failed with error reading next tar artifact header: can't build artifacts: build failed: failed to solve: mkdir /.docker: read-only file system.
• With this commit: task 04054dd3… SUCCEEDED in 3m00s; the task log carries Docker config dir "/.docker" is not wri… (the redirect message, the stored log is cut mid-line by something else), the builder pod pulled the base image through the auth provider, and the TUF repository now lists the eight 0.0.1 targets; the linux-amd64 binary downloads with a matching sha512.

[vmrm]

Independent review (Codex, static pass over the two-commit diff) and what changed

Accepted and fixed in e93a9e4:

• config.SetDir was never restored. The second build in the same process read config.json from the seed directory and lost credentials kept at the default location; two engines building at once could also observe each other's redirect through the unsynchronised global. Now the redirect holds only across NewDockerAuthProvider (which captures config.Dir() at construction), under a mutex, and is restored on return.
• The fallback was a guessable, unvalidated path in the shared temp dir. Now os.MkdirTemp("", "trdl-docker-config-"), mode 0700, created once per process; when even that fails the redirect is skipped with a log line and BuildKit's original error surfaces.
• The first tests asserted on the global variable, not on behaviour, so "construct the provider before redirecting" would have passed. The tests now call GetTokenAuthority on the returned attachable, the call buildkitd makes on a bearer challenge, and check where .token_seed lands.

Noted, out of scope for this PR (pre-existing, not touched by the diff): the tasks manager worker runs on context.Background() with no Clean hook, so queued tasks can outlive a plugin reload/unmount; the release task does not join the build producer on a consumer-side error; AGENTS.md references .agents/skills/rigorous-review/SKILL.md while the file is .agents/skills/review/SKILL.md.

Mutation evidence, reworked implementation

task --yes server:test:unit (ginkgo, --race) on macOS arm64 after each mutation, tree restored from git and checked clean after every round.

Mutation	Expected evidence	Result
restore function does nothing (return func() {})	…SeedsGoToPrivateTempDirWhenConfigDirUnwritable and …ConcurrentBuildsDoNotShareTheRedirect fail on config.Dir()	Killed, exactly those two
no redirect (config.SetDir(tokenSeedDir) removed)	same two fail, GetTokenAuthority returns the mkdir error	Killed, exactly those two
fixed guessable path + MkdirAll 0755 instead of MkdirTemp	…SeedsGoToPrivateTempDir… fails on the name prefix / mode	Killed, exactly that one
mutex removed	…ConcurrentBuilds… fails and the race detector reports DATA RACE on config.Dir/SetDir	Killed, 3 DATA RACE reports

golangci-lint: 0 issues. The repository's prettier step could not run locally (it needs the remote docker host, which was unreachable); the diff is Go only.

The green stand run reported above was taken on the first cut (b50a464); a re-run on a Stronghold build carrying e93a9e4 is pending and will be reported here.

[vmrm]

Green run on the reworked implementation

Same stand, Stronghold release-1.19 with vmrm/trdl@df6f0f7 (this branch at e93a9e4 plus #426 at 296f7cd) compiled in; the three replicas rolled onto the new bundle with zero restarts.

• trdl/git-signatures/release git_tag=v0.0.1 → task ce55cd37… SUCCEEDED in 3m00s, replica restarts 0 throughout.
• Builder pod trdl-builder-2b482239… in trdl-build created, ran the build, removed; namespace empty afterwards.
• The task log carries the redirect line (Docker config dir "/.docker" is not wri…, the stored log is still cut at the same offset by something unrelated to this PR).
• TUF targets.json moved to version 2 (expires 2026-12-07T12:07:41Z), same eight 0.0.1 targets; the linux-amd64 binary's sha512 is byte-identical to the previous run's, i.e. the build is reproducible across both cuts of this fix.

[vmrm]

Follow-up from the Stronghold review: seed directory per build (4a3a4fc)

Several trdl engines share one Stronghold process, and a per-process seed directory was shared between them; the reviewer asked how mounts are isolated. Answer: at the registry-auth level they never were (config.json is read process-wide, and in external-plugin mode every mount's process shares the same ~/.docker), but the seed directory can be made private by construction at no real cost. It is now created per build and removed when the build is over, under /dev/shm where that exists (a6576be), so nothing outlives the build and nothing reaches a disk.

Mutation evidence (task --yes server:test:unit, --race, tree restored and checked clean after each round):

Mutation	Expected evidence	Result
cleanup does not remove the dir	…SeedsGoToPrivateDirWhenConfigDirUnwritable (NoDirExists after cleanup) and …EveryBuildGetsItsOwnSeedDir (before == after) fail	Killed, exactly those two
config dir not restored	the same two fail on config.Dir()	Killed, exactly those two
one shared dir for all builds instead of MkdirTemp per build	…EveryBuildGetsItsOwnSeedDir fails (1 dir, not 8)	Killed, exactly that one

Not covered by a unit test: the defer removeTokenSeeds() at the call site in buildWithBuildkitClient (it needs a daemon to exercise); the stand runs below are the evidence for that wiring.

Stand, Stronghold release-1.19 with vmrm/trdl@8614031 (this branch at a6576be, per-process /dev/shm dir): release v0.0.1 task b912084d… SUCCEEDED in 3m00s, three replicas, zero restarts, builder pod created and removed. The per-build cut has not been run on the stand yet; the CI unit job on ubuntu exercises the /dev/shm branch, the macOS runs the os.TempDir() fallback.

[vmrm]

Stand run on the merged code (per-build seed directory)

trublast/trdl#1 (this branch's head, diff against 4a3a4fc on server/ empty) is merged as 61850079d and pinned in Stronghold main and release-1.19. The stand was switched to the release-1.19 build:

• three replicas rolled with zero restarts;
• trdl/git-signatures/release git_tag=v0.0.1 → task d970fe71… SUCCEEDED in 3m04s, builder pod trdl-builder-52538dad… created and removed;
• this exercises the defer removeTokenSeeds() wiring in buildWithBuildkitClient that the unit tests cannot reach, and the /dev/shm branch in the real pod.

That closes the "pending" note above; every cut of this PR has now been run end to end on the stand.

[vmrm]

Code Review Report (independent pass, repository review skill, subagent)

Base: origin/main @ a10e3b9 (fix(server): harden the kubernetes builder pod (#425)), fetched before the review. Head at review time: 4a3a4fc, 5 commits.
Diff: [3 files, +205/-5 lines]

Verdict

• Technical: Механизм корректен: порядок LoadDefaultConfigFile (buildkit.go:88) → проба MkdirAll (:129) → SetDir (:148) → NewDockerAuthProvider (:94–96) → восстановление (:149) верен; credential-хелперы после конструирования config.Dir() не читают (в docker/cli@v29.5.3 configfile/ и credentials/ — ноль обращений к Dir()); других вызовов config.Dir() в server/ нет; Solve дожидается session-горутины (client/solve.go:243–248, 399), поэтому RemoveAll не гонится с getSeed. Одна дыра в доказательствах: defer removeTokenSeeds() (:174) не покрыт — мутант выжил, при этом герметичный тест на него пишется: bkclient.New набирает соединение лениво (client/client.go:156, без WithBlock), закрытый локальный порт отказывает solve за миллисекунды.
• Product: Поведение при записываемом ~/.docker не меняется; на общем buildkitd для аутентифицированных хостов токены теперь запрашиваются на каждую сборку (util/resolver/authorizer.go:77–87: линковка fetcher'а идёт через VerifyTokenAuthority, новый seed её ломает), для анонимных хостов seed в MAC не входит (authprovider.go:295–297). Описано в PR честно.
• Risk: Низкий. Остаточные риски — мусор в /tmp при SIGKILL процесса во внешнем режиме и хрупкость тестов, наблюдающих общий каталог машины.

DoD Criteria

Criteria	Inferred?	Met?	Evidence
Процесс с несоздаваемым config.Dir() не падает; seeds уходят в приватный каталог сборки	yes	✅ (unit)	buildkit.go:127–149; мутации b, c, e, g убиты; ветка /dev/shm локально недостижима (macOS)
config.json по-прежнему читается из штатного каталога	yes	✅	buildkit.go:88 до :89; configfile/file.go:321–390 и credentials/* к config.Dir() не обращаются
config.Dir() восстанавливается; окно сериализовано	yes	✅	buildkit.go:85–86, :149; мутации b и d убиты (d — 7 каталогов вместо 8 + 5 DATA RACE)
Каталог seeds — на сборку, 0700, удаляется через defer на месте вызова	yes	⚠️	функция очистки и per-build доказаны (a, c); defer в buildWithBuildkitClient (:174) не покрыт — мутация f выжила
Обе базы недоступны → лог, редирект пропущен, ошибка BuildKit	yes	⚠️	buildkit.go:139–144 — только чтение
При записываемом штатном каталоге поведение прежнее	yes	✅	KeepsWritableConfigDir; мутация g убита

Issues

• Critical — нет.
• Major — server/pkg/docker/buildkit.go:173–174: единственная проводка, делающая истинным «ничего не переживает сборку», не имеет теста — мутация «убрать defer removeTokenSeeds()» проходит набор зелёным. Герметичный тест возможен: bkclient.New(ctx, "tcp://<закрытый порт>") + buildWithBuildkitClient(...) с ненаписываемым config.Dir() → connection refused, и «новых trdl-docker-config-* не осталось» убивает мутанта (проверено пробным тестом в копии ревьюера).
• Minor — buildkit.go:102: ошибка os.RemoveAll молча отбрасывается.
• Minor — buildkit_config_dir_test.go:114–124: тесты наблюдают общий для машины каталог (/dev/shm, os.TempDir()); падение любого require до cleanup() оставляет каталоги; чужой остаток с совпавшим именем сделал один замер мутации ревьюера недействительным.
• Minor — buildkit_config_dir_test.go:92–106: GetAuthConfig → GetCredentialsStore может запустить docker-credential-* из PATH и прочитать DOCKER_AUTH_CONFIG.
• Minor — buildkit.go:106–125: комментарий привязан к var dockerConfigDirMu, а описывает функцию; часть пересказывает порядок вызовов.

Risks

№	Risk	Type	Likelihood	Severity	Location	Circumstances	Consequences	Recommendation
1	Каталог seeds переживает процесс	Operational	Unlikely	Low	buildkit.go:102, :174	SIGKILL/OOM плагина во время сборки во внешнем режиме	0700-каталог с 16-байтным seed в /tmp; в builtin /dev/shm/emptyDir умирают с подом	Принять
2	Окно редиректа видно другому потребителю docker/cli в том же процессе	Technical	Unlikely	Low	buildkit.go:148–149	другой плагин в builtin-процессе читает config.Dir() в те же микросекунды	чужой config.Dir() вернёт seed-каталог	Принять; в server/ других вызовов нет
3	Флак и мусор от тестов, наблюдающих общий каталог	Technical	Possible	Low	buildkit_config_dir_test.go:114–124	два прогона на одной машине; падение require	ложный красный; каталоги в $TMPDIR	t.Cleanup(cleanup), t.Setenv("TMPDIR", …)
4	Повторный запрос токенов на каждую сборку на общем buildkitd	UX/Product	Likely	Low	buildkit.go:136; authorizer.go:77–87	buildkitd_address + аутентифицированный реестр	лишний token-запрос на сборку	задокументировано в PR

Mutation table

macOS arm64; /dev/shm отсутствует, работала ветка os.TempDir(). После каждого раунда git checkout -- и git diff --quiet HEAD → чисто.

Mutation	Test that must fail	Result
(a) os.RemoveAll(seedDir) → no-op	SeedsGoToPrivateDir…, EveryBuild…	Killed, ровно эти две
(b) восстановление config.Dir() убрано	те же две	Killed, ровно эти две
(c) один общий каталог вместо MkdirTemp	EveryBuild… («1, not 8»)	Killed
(d) мьютекс убран	EveryBuild… + DATA RACE	Killed: 7 каталогов вместо 8, 5 DATA RACE
(e) NewDockerAuthProvider до редиректа	SeedsGoToPrivateDir…, EveryBuild…	Killed
(f) defer removeTokenSeeds() → _ = removeTokenSeeds	ничего в наборе	Survived
(g) проба err == nil инвертирована	KeepsWritableConfigDir	Killed

Ворота: golangci-lint v2.11.0 0 issues.; prettier не выполнился (удалённый docker-хост недоступен); task --yes server:test:unit — Test Suite Passed.

Not verified

• task server:build (только Linux/CGO); ветка /dev/shm (macOS); ветка «обе базы недоступны»; стендовые прогоны из комментариев PR; prettier; task e2e:test:e2e; другие потребители config.Dir() в builtin-процессе Stronghold вне этого репозитория.

---

Author's response

Принято в 021b17c:

• Major: TestBuildWithBuildkitClient_RemovesTokenSeedsAfterTheBuild — слушатель на 127.0.0.1:0, закрыт, bkclient.New("tcp://<addr>"), buildWithBuildkitClient с ненаписываемым config.Dir() → ошибка, новых каталогов нет. Мутация (f) теперь убита ровно этим тестом; (a) дополнительно ловится им же.
• os.RemoveAll — ошибка логируется через logboek.
• Тесты изолированы: t.Setenv("TMPDIR", t.TempDir()), PATH="", DOCKER_AUTH_CONFIG=""; каждый cleanup зарегистрирован через t.Cleanup, падение require каталогов не оставляет.
• Комментарий перенесён на функцию, пересказ порядка вызовов убран; у dockerConfigDirMu — одна строка о том, почему мьютекс нужен.

Риски 1, 2, 4 приняты как описано в отчёте.