Detect active deployments before provisioning

[spboyer]

Summary

Fixes #7248

Before starting a deployment, azd now checks for active deployments on the target scope. If another deployment is in progress, it warns the user and waits for it to complete — avoiding the DeploymentActive ARM error that wastes ~5 minutes of the user's time.

Telemetry Context

• 199 DeploymentActive failures in March (~270/month projected)
• Average wait before failure: 5.3 minutes (P90: 12.2 min)
• 78% from provision, 19% from up

Changes

Pre-deployment active check (bicep_provider.go)

Added waitForActiveDeployments() between preflight validation and deployment submission:

• Lists deployments filtered for active provisioning states
• If found: warns with deployment names, polls at 30s intervals
• Timeout: 30 minutes (matches typical long deployments)
• Only ignores ErrDeploymentsNotFound (scope doesn't exist yet); other errors propagate

Active state classification (deployments.go)

IsActiveDeploymentState() classifies 11 provisioning states as active, including transitional states (Canceling, Deleting, DeletingResources, UpdatingDenyAssignments) that can still block new deployments.

Scope interface (scope.go)

Added ListActiveDeployments() to both ResourceGroupScope and SubscriptionScope.

Error suggestion (error_suggestions.yaml)

Added DeploymentActive rule with user-friendly message and ARM troubleshooting link.

Test Coverage (8 tests, 24 subtests)

Test	Coverage
TestIsActiveDeploymentState	17 subtests covering all provisioning states
TestWaitForActiveDeployments_NoActive	Happy path
TestWaitForActiveDeployments_InitialListError_NotFound	RG doesn't exist yet
TestWaitForActiveDeployments_InitialListError_Other	Auth/throttle errors propagate
TestWaitForActiveDeployments_ActiveThenClear	Polling until clear
TestWaitForActiveDeployments_CancelledContext	Context cancellation
TestWaitForActiveDeployments_PollError	Error during polling
TestWaitForActiveDeployments_Timeout	30min timeout

Related

• Improve Container Apps provisioning error handling and classification #7249 — Container Apps error handling (separate PR Add Container Apps and ARM error suggestions #7250)
• docs: local preflight — storage account policy check investigation #7179 — Preflight Azure Policy check (vhvb1989)

[Copilot]

Pull request overview

Adds a pre-deployment check that detects in-progress ARM deployments at the target scope and waits for them to complete, avoiding DeploymentActive failures during provisioning.

Changes:

• Introduces waitForActiveDeployments() in the Bicep provisioning flow and polls until deployments clear or a timeout is reached.
• Adds IsActiveDeploymentState() plus new tests to classify which provisioning states are considered “active”.
• Extends infra.Scope with ListActiveDeployments() and adds a DeploymentActive error suggestion rule.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
cli/azd/resources/error_suggestions.yaml	Adds a user-facing suggestion for DeploymentActive ARM errors.
cli/azd/pkg/infra/scope.go	Extends scope interface + implements ListActiveDeployments() for RG and subscription scopes.
cli/azd/pkg/infra/provisioning/bicep/bicep_provider.go	Adds wait loop before deployment submission, with polling/timeout defaults.
cli/azd/pkg/infra/provisioning/bicep/bicep_provider_test.go	Updates mocked scope to satisfy the new Scope interface.
cli/azd/pkg/infra/provisioning/bicep/active_deployment_check_test.go	Adds tests covering wait-loop behavior, errors, cancellation, and timeout.
cli/azd/pkg/azapi/deployments.go	Adds IsActiveDeploymentState() helper.
cli/azd/pkg/azapi/deployment_state_test.go	Adds unit tests for active/inactive state classification.

[spboyer]

Telemetry Context: DeploymentActive + Retry Behavior

This PR addresses DeploymentActive (199 errors/month). Additional context from the deep dive:

Retry behavior makes this especially valuable

Of machines that hit InvalidTemplateDeployment errors (which includes DeploymentActive in the chain):

• 66% retry without changing anything — for DeploymentActive, this means they re-submit and hit the same active deployment again
• Average 3.6 retries per machine before they either succeed or give up
• The detect-and-wait pattern in this PR would break that retry loop immediately

Time savings

• DeploymentActive users currently wait an average deployment duration before failing, then retry
• With this PR: one wait period (with progress feedback) instead of N failed attempts × deployment time each

This is a clean win — the fix is architecturally simple (poll + wait) and eliminates a category of failure that can never be solved by retrying.

[kristenwomack]

@copilot - will you check to ensure we have metrics coverage so we can see how often this error and fix occur after merging this change?

[Copilot]

@kristenwomack I've opened a new pull request, #7288, to work on those changes. Once the pull request is ready, I'll request review from you.

[spboyer]

Converting to draft — the active deployment check integration point in Deploy() causes nil-pointer panics in existing test mocks. The test infrastructure creates Deployment objects with nil inner scopes, and scopeForTemplate requires provider state that isn't available in all test paths.

Need to either:

1. Restructure as a standalone pre-deployment middleware (avoids touching the Deploy code path)
2. Add the check at the provision action level (internal/cmd/provision.go) before it calls into the bicep provider

The standalone tests for waitForActiveDeployments, IsActiveDeploymentState, and the error suggestion rule all pass. Only the integration with the existing Deploy flow is problematic.

[spboyer]

/azp run

[azure-pipelines]

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

[spboyer]

/azp run azure-dev - cli

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[wbreza]

Code Review — PR #7251

Detect active deployments before provisioning by @spboyer

What's Done Well

• Clean three-layer architecture — state classifier in azapi, filter helper in infra, wait loop in bicep. Each layer independently testable and follows DRY
• Comprehensive test coverage — 8 test functions, 24 subtests; activeDeploymentScope mock with per-call control is elegant
• All 9 prior review findings addressed — ErrDeploymentsNotFound in poll, spinner status, scopeForTemplate logging, error wrapping consistency, scope-agnostic messaging
• Error suggestion as defense-in-depth — DeploymentActive YAML rule catches the inherent race window
• Modern Go — for i := range 200, t.Context(), clean switch exhaustiveness
• Excellent PR description — telemetry context (199 failures, 5.3 min avg waste) justifies the feature clearly

Findings

Priority	Count
Critical	0
High	0
Medium	1
Low	2
Total	3

🟡 Medium

1. Root cause: Ctrl+C doesn't cancel ARM deployments — follow-up opportunity

bicep_provider.go — Deploy() / deployModule()

@spboyer This PR elegantly handles the symptom (detecting and waiting for active deployments), but the most likely root cause of those 199 DeploymentActive failures is users pressing Ctrl+C during provisioning. When azd receives SIGINT, PollUntilDone() stops polling and the process exits — but the ARM deployment continues running on Azure. The DeploymentService interface has no Cancel method, and there's no signal-handler cleanup that sends a cancellation request to ARM.

Follow-up idea: Consider a future PR that:

1. Adds a CancelDeployment() method to DeploymentService
2. Registers a signal handler (or context.AfterFunc) in the Deploy flow that calls cancel on SIGINT
3. Shows "Cancelling deployment..." feedback to the user

This would prevent the issue entirely rather than waiting for orphaned deployments to finish. Combined with this PR's detection, it'd be a complete solution.

🟢 Low

2. Transient poll errors are hard failures — waitForActiveDeployments() poll loop

A single transient ARM API error during polling (e.g., throttling 429, network blip) aborts the entire provisioning attempt. Consider retrying 2-3 times with backoff before surfacing the error.

3. Timeout message could include provisioning states — waitForActiveDeployments() deadline case

The timeout error shows deployment names but not their provisioning states. Showing stuck-deploy (Canceling) vs stuck-deploy (Running) would help users decide whether to wait or take manual action.

Summary

Overall Assessment: Approve — the PR is well-implemented and directly addresses user pain. The medium finding about ARM cancellation is a follow-up enhancement, not a blocker for this PR.

Review performed with GitHub Copilot CLI

[vhvb1989]

Concern: Blocking on ALL active deployments creates a bottleneck for shared subscriptions

The current implementation of waitForActiveDeployments calls ListActiveDeployments, which lists all deployments at the target scope and blocks if any of them are in an active provisioning state — regardless of deployment name.

Why this is a problem

Using the same subscription for a large team is a very common approach with azd. In our own CI, we have parallel tests that each deploy a different template to the same subscription concurrently, and this works fine because ARM allows concurrent deployments with different names at the same scope.

With this PR, those parallel deployments would serialize — the second azd up would see the first one as "active" and wait up to 30 minutes, even though ARM would happily accept both. This introduces an artificial bottleneck that does not exist today:

• Shared subscription teams: Developer A deploys env dev-alice, Developer B deploys env dev-bob to the same subscription scope → B is blocked waiting for A, even though the deployments are completely independent.
• CI pipelines: Parallel test jobs deploying different templates to the same subscription would queue up one-by-one instead of running concurrently.
• Multi-environment workflows: Deploying staging and production from the same subscription scope at the same time would be blocked.

Root cause in the code

azd already generates unique deployment names per environment via GenerateDeploymentName ({envName}-{unixTimestamp}). ARM's DeploymentActive error is triggered when a deployment with the same name is submitted while another with that name is already running — not when any deployment in the scope is active.

The 199 DeploymentActive errors from telemetry are almost certainly same-user retries hitting the same deployment name collision, not cross-team conflicts. The fix should target that specific scenario.

Suggested approach

Instead of blocking on all active deployments, we should investigate how ARM actually determines a conflict. The signal is likely the deployment name. The check should:

1. Generate the deployment name that azd is about to use (or match the environment name prefix pattern)
2. Only block if an active deployment with a matching name (or name prefix based on the environment) exists
3. Allow unrelated deployments to proceed concurrently, matching ARM's actual behavior

This way, the retry scenario (same env, same deployment name pattern) is handled correctly, while teams sharing a subscription or running parallel CI jobs are not penalized.

Please investigate what makes ARM fail with DeploymentActive — whether it's the deployment name, a template hash, or something else — and scope the detection to match that logic.

[jongio]

PR Review - #7251

Detect active deployments before provisioning by @spboyer

Summary

What: Adds a pre-deployment check that detects in-progress ARM deployments at the target scope, warns the user, and polls until they clear - preventing the DeploymentActive error that wastes ~5 min per occurrence.

Why: 199 DeploymentActive failures in March, users averaging 3.6 blind retries. This check breaks the retry loop immediately.

Assessment: Solid implementation that follows existing codebase patterns well - state classifier in azapi, filter helper in infra, wait loop in bicep. The design avoids breaking the Scope interface (standalone function instead of method). Previous review rounds addressed spinner status, error wrapping consistency, and poll-loop ErrDeploymentsNotFound handling. Two remaining items below.

Findings

Category	Critical	High	Medium	Low
Performance	0	0	1	0
Logic	0	0	1	0
Tests	0	0	2	1
Total	0	0	4	1

Key Findings

1. [MEDIUM] Performance: time.After leaks a 30-min timer goroutine; same file already uses time.NewTimer (see inline)
2. [MEDIUM] Logic: SubscriptionScope.ListDeployments never produces ErrDeploymentsNotFound - the recovery paths in waitForActiveDeployments are dead code for subscription-scoped templates. Not a bug today, but worth a code comment documenting the asymmetry.
3. [MEDIUM] Tests: CancelledContext test is subtly racy (see inline)
4. [MEDIUM] Tests: No integration test for the Deploy() call site - scopeForTemplate failure silent-skip path and ordering relative to preflight are untested.
5. [LOW] Tests: TestIsActiveDeploymentState doesn't test an unknown/future state string against the default branch. A one-liner would document the design choice.

Test Coverage Estimate

• Well covered: IsActiveDeploymentState (17 subtests), waitForActiveDeployments (8 tests covering happy path, errors, polling, timeout, cancellation)
• Indirectly covered: ListActiveDeployments / filterActiveDeployments exercised through wait tests
• Missing: Deploy() integration with active-deployment check; no dedicated ListActiveDeployments unit test in scope_test.go

What's Done Well

• Clean layering: state classifier (azapi) -> filter (infra) -> orchestrator (bicep). Each layer testable independently.
• Standalone ListActiveDeployments function avoids breaking the exported Scope interface - good API hygiene.
• ErrDeploymentsNotFound handling in poll loop correctly handles resource group deletion during wait.
• Test helper activeDeploymentScope is well designed with per-call response maps and atomic call counter.

---

2 inline comments below.

[jongio]

Reviewed the incremental changes in 142aa47 (since my last review at 9d66ad5). Both fixes look correct:

1. Timer leak - time.After replaced with time.NewTimer + deferred Stop(), matching the existing pattern in the same file's Deploy() progress goroutine. The <-deadline channel updated to <-deadlineTimer.C.

2. Racy mock - CancelledContext test now seeds poll indices 0-4 so the ticker can't return nil when it wins the select race against ctx.Done(). Comment added explaining the intent.

No novel findings after reviewing 6 changed files (435+ lines) and deduplicating against existing feedback.

[jongio]

Previous issues from 142aa47 all addressed. No code changes in the latest push (00ef2a8, CI retrigger only).

Re @vhvb1989's design concern about blocking on all active deployments - ARM's constraint is per-scope, not per-deployment-name. The docs explicitly say "Wait for concurrent deployment to this resource group to complete" for DeploymentActiveAndUneditable (https://learn.microsoft.com/azure/azure-resource-manager/troubleshooting/common-deployment-errors). Only one deployment can be active per resource group or subscription-level scope at a time, regardless of name. Concurrent deployments to different resource groups aren't affected since scopeForTemplate correctly narrows the check to the specific scope being targeted.

The CI scenario mentioned (parallel tests deploying different templates to the same subscription) likely works because each test targets a different resource group - those have independent deployment locks.

[vhvb1989]

The CI scenario mentioned (parallel tests deploying different templates to the same subscription) likely works because each test targets a different resource group - those have independent deployment locks.

We can deploy the same template at the same time from Windows, Mac and Ubuntu to the same subscription. The template uses subscriptio-level deployments. Yes, we use a different env-name to use a unique deployment-name object and create different resource groups for each, but they we do the subscription-scope deployments at the same time. That's the concern I am referring to, @jongio --- with the change, a new subscription-level deployment will not continue if there is another deployment going on in the same subscription, right? -

[jongio]

@vhvb1989 Good catch. Your concern is valid. The current implementation of waitForActiveDeployments blocks on any active deployment at the scope, regardless of deployment name. This would serialize parallel CI runs that use different env-names (and therefore different ARM deployment names), even though ARM allows concurrent deployments with different names at the same scope.

Docs backing this up:

1. Deployment naming (Azure CLI docs): "If you run another deployment and give it the same name, the earlier entry is replaced with the current deployment. If you want to maintain unique entries in the deployment history, give each deployment a unique name." This confirms the concurrency boundary is per-deployment-name, not per-scope.

2. Common deployment errors: The DeploymentActiveAndUneditable error description says "Wait for concurrent deployment to this resource group to complete" but this applies to same-name collisions. The table also lists AnotherOperationInProgress separately for resource-level conflicts.

3. Subscription deployment scoping: "For each deployment name, the location is immutable." Deployments are tracked by name. Different names are independent entries.

The issue: ListActiveDeployments in scope.go calls scope.ListDeployments(ctx) and filters for any active provisioning state across ALL deployments. In the CI scenario described (Windows/Mac/Ubuntu deploying simultaneously with different env-names), deployment B would see deployment A as "active" and wait for it, even though they use different deployment names and would never hit DeploymentActive.

Suggested fix: Pass the current deployment name into the wait function and filter to only match that name:

// In bicep_provider.go, pass the deployment name through
if activeScope, err := p.scopeForTemplate(planned.Template); err == nil {
    if err := p.waitForActiveDeployments(ctx, activeScope, deployment.Name()); err != nil {
        return nil, err
    }
}

// Updated waitForActiveDeployments signature
func (p *BicepProvider) waitForActiveDeployments(
    ctx context.Context,
    scope infra.Scope,
    currentDeploymentName string,
) error {
    active, err := infra.ListActiveDeploymentsByName(ctx, scope, currentDeploymentName)
    // ... rest unchanged
}

// In scope.go, name-scoped filter
func ListActiveDeploymentsByName(
    ctx context.Context,
    scope Scope,
    deploymentName string,
) ([]*azapi.ResourceDeployment, error) {
    all, err := scope.ListDeployments(ctx)
    if err != nil {
        return nil, err
    }

    var active []*azapi.ResourceDeployment
    for _, d := range all {
        if d.Name == deploymentName && azapi.IsActiveDeploymentState(d.ProvisioningState) {
            active = append(active, d)
        }
    }
    return active, nil
}

This preserves the pre-check for same-name conflicts (the actual DeploymentActive error scenario from #7248) while allowing parallel deployments with different names to proceed without blocking each other.

For stack deployments (azd-stack-{envName}) this is especially important since the name is deterministic and same-env reruns are exactly the case that triggers DeploymentActive. For standard deployments ({envName}-{timestamp}), name collisions are rare but possible within the same second.

[spboyer]

@jongio

Implemented in b504502. Changes:

• Added ListActiveDeploymentsByName() in scope.go that filters by both deployment name and active provisioning state
• Updated waitForActiveDeployments to accept a deploymentName parameter
• Caller now passes deployment.Name() so only same-name conflicts are detected
• Updated all 8 test cases with appropriate deployment names

This allows parallel CI runs with different env-names to proceed without blocking each other, while still catching same-name DeploymentActive errors.

[vhvb1989]

I think you need to generate new recordings for some tests to pass CI