Skip to content

deploy: UpdateRelease has no per-attempt deadline, so a hung API response blocks fly deploy before any Machine is updated #5203

Description

@csechuan

Summary

fly deploy can block indefinitely inside updateReleaseInBackend(ctx, "running"), before any Machine is updated. The function wraps uiexClient.UpdateRelease in a 3-attempt retry, but no attempt has a deadline, and the underlying HTTP client has no Timeout. A response that never arrives therefore never becomes an error, so the retry never fires.

Observed

flyctl v0.4.95, fly deploy --config <cfg> --image registry.fly.io/<app>:<sha> (image already pushed, single Machine, rolling strategy):

15:41:17  Watch your deployment at https://fly.io/apps/<app>/monitoring
          ... no output for 358 s ...
15:47:15  Error: failed to set release status to 'running': terminated signal received

The SIGTERM came from our own outer timeout. No Machine was touched: afterwards fly machine list showed the Machine still started on the previous image. Re-running the identical command ~8 minutes later completed in 80 s. On a normal run, Updating existing machines ... follows Watch your deployment within ~2 s (median over 37 deploys).

Code path (also present on v0.4.104 at the time of writing)

  • internal/command/deploy/machines_deploymachinesapp.go: DeployMachinesApp calls md.updateReleaseInBackend(ctx, "running", nil) before deployMachinesApp.
  • internal/command/deploy/machines.go: updateReleaseInBackend runs retry.Do(..., retry.Context(ctx), retry.Attempts(releaseStatusRetryAttempts) /* 3 */, ...), and each attempt calls md.uiexClient.UpdateRelease(ctx, ...) with the parent ctx.
  • internal/uiex/releases.go: UpdateRelease uses http.NewRequestWithContext(ctx, PATCH, ...) and c.httpClient.Do(req).
  • internal/uiex/client.go builds that client with fly.NewHTTPClient. In fly-go http.go, that returns &http.Client{Transport: ...} with no Timeout.

Expected

Give each attempt its own deadline, e.g. attemptCtx, cancel := context.WithTimeout(ctx, 30*time.Second) inside the retry closure. A hung response then fails that attempt, and the existing 3-attempt backoff covers hangs as well as errors. The call is idempotent (it sets a status), which is already why it is retried.

A client-level Timeout on the uiex HTTP client would also bound the other uiex calls (CreateRelease, GetRelease, ...), which have the same shape.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions