[tweeks-reify]

what

Targeted applies (-p, -d, -w) of projects in different directories or workspaces on one pull request run in parallel again, as they did before v0.46.0. Projects that share a directory and workspace still apply one at a time, as on master. Plan and untargeted apply keep the exclusive pull-level lock from #6605.

• WorkingDirLocker.TryLockPullShared: a pull-level lock any number of targeted applies can hold together. It still conflicts with the exclusive lock (TryLockPull) and with in-flight plans, and TryLockPull now conflicts with shared holders.
• WorkingDirLocker.TryLockApplyDir: once targets resolve, every apply claims the workspace and directory of each project until it has recorded results. Projects that share a directory share it on disk, so the claim leaves out the project name, and one apply covering several of them claims the directory once. A second apply into a claimed directory is rejected with the existing "workspace ... is currently locked" message, which links the running job, or says several jobs are running when more than one project in that directory is applying.
• ApplyCommandRunner records results, publishes the combined status and automerges one apply at a time per pull request (in-process lock). A waiter logs when it starts waiting, and every 2 minutes after that it warns which apply holds the lock and for how long. The pull lock stays held through that step, because plans record results without the mutex. The silenced no-project path re-reads recorded status under the same mutex before publishing, and leaves the status alone if that read fails, since it changed nothing.
• atlantis cancel is scoped to the commands running when it is issued. Each command records a mark from CancellationTracker when it arrives and checks cancels against it. Before, every command cleared the pull-wide flag when it finished, so a finishing targeted apply could un-cancel a parallel one. This also stops a cancel issued while nothing was running from cancelling the next command. Closing or merging a pull request now records a cancel, so commands still running on it skip work they have not started instead of running in working dirs that were just deleted. Before, the PR-close executor was built without the tracker. Clear is gone, because dropping a pull request's entry would un-cancel its running commands. The tracker keeps one entry per cancelled or closed pull request until restart. DefaultCommandRunner.CancellationTracker is now validate:"required", because a command without a tracker gets mark 0, and with nothing clearing cancels, one cancel would then stop every later command on that pull request.
• Every working dir lock records an owner. After atlantis cancel (UnlockByPull), a still-running command's release cannot drop a newer holder's lock.
• Pull-level lock and claim keys start with $, which workspace names cannot contain, so a workspace such as .shared-pull cannot collide with them.
• The post-workflow hook runner warns when it skips hooks because the default workspace is locked, naming the command that holds the lock. The caller discards that error.
• Docs: locking.md describes concurrency inside a pull request, what atlantis cancel and closing a pull request do, and the steps that share a workspace's root lock, plan preparation included. AGENTS.md records the lock order.

why

• fix: validate current head and base plan state #6605 made every apply take an exclusive lock keyed only by repo and pull number, before project resolution. atlantis apply -w daily-1 and atlantis apply -w daily-2 now reject each other even though they share no working directory, plan file or state.
• Each workspace has its own clone, so the working directory a project applies in is its workspace plus directory. Every other lock on the apply path includes the project name, so claiming by project would let project1-staging and project1-production from the custom workflows example run Terraform in one directory at once. Releases before v0.46.0 allowed that; the claim keeps master's protection.
• That lock does three jobs. This change keeps each one:

Job the pull lock does today	How it is kept
Keep apply away from plan files while a plan runs	Plan and autoplan stay exclusive and conflict with shared holders. Shared holders still check for in-flight plan locks
Stop two commands running Terraform in one directory	Workspace-and-directory apply claim, held from after command build until results are recorded
Keep result recording, combined status and automerge consistent	Per-pull result mutex, taken while the pull lock is held. Redis UpdatePullWithResults is read-modify-write without a transaction, so parallel applies could otherwise lose a result. Statuses published from recorded results go out one at a time

• The project runner already rechecks recorded plan status from the DB right before each project applies, and it rejects already-applied projects. Holding the claim until results are recorded closes the window between one apply finishing and its DB write.
• The result lock waits without a bound. Giving up would leave an applied project's result unrecorded, which is worse than waiting while the holder is slow. If the holder hangs in a VCS call (the GitHub user-credentials client sets no timeout), later applies on that pull request hang with it, and atlantis cancel cannot free them. On master, cancel released the exclusive lock and the next apply finished. The lock is a channel semaphore, so waiters warn every 2 minutes with the holder's command, user and targets. Letting atlantis cancel abandon the lock is a follow-up that can build on the same semaphore.
• The combined status is still last-write-wins. A rejected apply, or one that fails before recording results, publishes FAILED outside the result mutex, so a parallel apply's status can land before or after it. master has the same race between sequential applies. Taking the mutex for those writes would not change which write lands last.
• atlantis cancel releases every working dir lock, claims included, while running commands carry on, so after a cancel a second apply of a project still being applied can start. That matches master, where cancel releases the exclusive lock, and locking.md says so.
• These are unchanged from v0.45.0 and stay out of scope. locking.md documents the first three:
  • Preparing a -d or -w apply locks the workspace root (., no project name), so two of them in one workspace can collide during command build. Planning or applying an unnamed root project holds that lock for its whole run.
  • Pre-workflow and post-workflow hooks take the default workspace's root lock. A command that cannot take it skips its hooks, or fails with --fail-on-pre-workflow-hook-error.
  • API applies take neither pull locks nor claims.
  • Import, state, approve_policies and the locks UI discard button write pull status outside the result mutex.
  • The apply comment is posted before results are recorded, so a failed result write still follows a success comment.
• atlantis cancel releases claims, so an apply that was already running can still claim its directories and apply its first execution order group after a cancel. That matches master, where a running apply carries on after cancel releases the exclusive lock.
• Follow-ups, also on master: a failed result write leaves the apply status pending; the command-build error path does not mark the command errored; a duplicate targeted apply arriving after the first one finished records the validator's rejection as apply_errored over applied.
• Follow-ups for this design: let atlantis cancel abandon a hung result lock; take the cancel mark in the events controller rather than inside the command goroutine, so a cancel processed before an earlier apply's goroutine starts still reaches it; give the mark its own type; pin the $ reservation with a test against valid.ValidateWorkspaceName.

tests

• TestApplyCommandRunner_TargetedAppliesForDifferentWorkspacesRunInParallel reproduces the issue: two targeted applies each block inside their project apply until the other has started. It fails on master ("apply for workspace daily-1 did not start while workspace daily-2 was applying") and passes here. It also asserts both projects are recorded as applied, that the last combined status published is 2/2 success, and that result writes never overlap. That last check uses a DB wrapper that widens the read-modify-write window Redis has.
• TestRunProjectCmdsWithCancellationTracker_FinishingCommandKeepsOtherCommandCancelled: two commands run on one pull request, atlantis cancel arrives, one finishes, and the other's next execution order group is still cancelled. TestRunCommentCommand_HonorsOnlyCancelsAfterCommandArrives covers the mark taken when a comment command arrives.
  TestRunProjectCmdsWithCancellationTracker_ParallelGroupIgnoresCancelBeforeCommandArrived and TestRunAutoplanCommand_HonorsOnlyCancelsAfterCommandArrives cover the mark on the parallel path and on autoplan. TestCleanUpPullCancelsRunningCommands: closing a pull request cancels commands still running on it, keeps an earlier cancel in force, and leaves later commands alone.
• TestApplyCommandRunner_TargetedApplyBlocksWhileProjectInSameDirectoryApplies: apply -p project1-production is rejected while project1-staging, in the same directory, is applying. TestApplyCommandRunner_ApplyClaimsSharedDirectoryOnce: an apply covering both succeeds and releases the claim.
• TestApplyCommandRunner_HoldsPullLockAndClaimsWhileRecordingResults: while an apply records results, a plan cannot take the pull lock and another apply cannot claim its directory, targeted or not.
• TestPullResultLockerWarnsWhileWaiting (synctest): a waiter warns after 2 minutes with the holder's description and how long it has held the lock.
• TestApplyCommandRunner_NoProjectTargetedApplyWaitsForParallelResults: a silenced no-project apply waits while a parallel apply records results, then publishes a status that includes them. It waits on the lock's "waiting" log line rather than a timeout window.
• TestPullResultLockerKeepsEntryWhileCallersWait: a third caller still waits while the second holds the lock, so the per-pull entry is not dropped early. It runs a fixed sequence instead of a goroutine stress test, so it fails on every run when the reference count is wrong.
• TestApplyCommandRunner_ReleasesLocksOnErrorExits: the pull lock and every claim are released when the plan status read fails, the command build fails, a project is already claimed, or recording results fails, for targeted and untargeted applies. A further apply then finishes, so the result lock was released too. The parallel test also checks that both claims are released after success.
• Generic apply is rejected while a targeted apply is in flight. A targeted apply is rejected while its directory is claimed. A silenced no-project apply publishes the current recorded status, and leaves it alone when it cannot read it.
• Locker unit tests cover shared/exclusive conflicts, in-flight plans, directory claims and the jobs they link, no conflict between a claim and TryLock, UnlockByPull releasing shared locks and claims, stale releases after UnlockByPull for every lock kind, and workspace names that spell the old reserved keys.
• Mutation check, each against its test: claiming by project name, not deduplicating a shared directory, not releasing claims, passing a zero mark to the parallel path or autoplan, dropping the result lock entry early, removing the no-project result mutex, releasing the pull lock before recording results, not clearing on PR close, linking jobs from other directories, not cancelling on PR close, releasing claims or leaking the result lock before results are recorded, dropping the waiter's warning or the holder's description, restoring the clear-on-finish cancel, removing the owner check, and the . key marker. Every one fails. A server built without DefaultCommandRunner.CancellationTracker fails NewServer validation.
• go test -race -count=10 on 59 lock, cancel and apply concurrency tests: 590 runs, no races, no failures.
• make test, make check-fmt, make build-service pass, except cmd TestExecute_ConfigFile. It fails identically on clean master in my local environment, where a GitHub token in the shell overrides the config-file value.
• mocks/mock_working_dir_locker.go and mocks/mock_cancellation_tracker.go were generated with pegomock. Under Go 1.27, pegomock fails locally for every interface because golang.org/x/tools v0.49.0 cannot read export data version 5. I generated with a newer x/tools through a scratch -modfile. go.mod is unchanged.

references

• closes Parallel Workspace Applies Fail since 0.46.0 #6843
• Related: Pull-level apply lock rejects targeted apply -p for projects the instance does not own, breaking --silence-no-projects in multi-instance setups #6786 / fix: silence unowned targeted applies before pull locking #6811 (unowned targeted applies under --silence-no-projects) and fix: prevent concurrent PR commands across Redis-backed replicas #6825 (Redis-backed pull lease across replicas). Whichever lands second needs to account for the shared mode. The in-process locks here do not coordinate across replicas, which matches the behavior on master today.

architecture decision record

• I checked the ADR criteria.

ADR: Not required. This is a regression fix restoring pre-v0.46 targeted-apply concurrency while keeping the #6605 protections. It does touch locking and cancel, so I can write one if maintainers prefer.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

A finishing parallel apply can clear another apply’s cancellation state, allowing queued projects to execute after atlantis cancel.

1 open issue

• Prevent concurrent applies from clearing shared cancellation state · New

1 file not reviewed

• server/events/mocks/mock_working_dir_locker.go: Generated file

What changed in this PR

Restores parallel targeted applies for different projects on the same pull request, addressing the regression reported in #6843 while retaining exclusive locking for plans and untargeted applies.

Changes:

• Adds shared pull locks and owner-tracked project apply claims.
• Serializes result recording, combined status publication, and automerge per pull request.
• Adds concurrency regression tests, updates mocks, and documents locking behavior.

File	Description
server/​events/​working_dir_locker.go	Adds shared locks, project claims, and owner-safe release.
server/​events/​working_dir_locker_test.go	Tests lock conflicts, cancellation cleanup, and stale releases.
server/​events/​project_command_runner_test.go	Updates the test locker for the extended interface.
server/​events/​mocks/​mock_working_dir_locker.go	Adds generated mocks for the new lock methods.
server/​events/​apply_command_runner.go	Enables targeted concurrency and serializes result handling.
server/​events/​apply_command_runner_test.go	Tests concurrent applies, project claims, and status consistency.
server/​events/​apply_command_runner_internal_test.go	Tests per-pull result lock serialization and cleanup.
runatlantis.io/​docs/​locking.md	Documents concurrency within a pull request.

_{Review effort: Balanced}