Summary
fly deploy can block indefinitely inside updateReleaseInBackend(ctx, "running"), before any Machine is updated. The function wraps uiexClient.UpdateRelease in a 3-attempt retry, but no attempt has a deadline, and the underlying HTTP client has no Timeout. A response that never arrives therefore never becomes an error, so the retry never fires.
Observed
flyctl v0.4.95, fly deploy --config <cfg> --image registry.fly.io/<app>:<sha> (image already pushed, single Machine, rolling strategy):
15:41:17 Watch your deployment at https://fly.io/apps/<app>/monitoring
... no output for 358 s ...
15:47:15 Error: failed to set release status to 'running': terminated signal received
The SIGTERM came from our own outer timeout. No Machine was touched: afterwards fly machine list showed the Machine still started on the previous image. Re-running the identical command ~8 minutes later completed in 80 s. On a normal run, Updating existing machines ... follows Watch your deployment within ~2 s (median over 37 deploys).
Code path (also present on v0.4.104 at the time of writing)
internal/command/deploy/machines_deploymachinesapp.go: DeployMachinesApp calls md.updateReleaseInBackend(ctx, "running", nil) before deployMachinesApp.
internal/command/deploy/machines.go: updateReleaseInBackend runs retry.Do(..., retry.Context(ctx), retry.Attempts(releaseStatusRetryAttempts) /* 3 */, ...), and each attempt calls md.uiexClient.UpdateRelease(ctx, ...) with the parent ctx.
internal/uiex/releases.go: UpdateRelease uses http.NewRequestWithContext(ctx, PATCH, ...) and c.httpClient.Do(req).
internal/uiex/client.go builds that client with fly.NewHTTPClient. In fly-go http.go, that returns &http.Client{Transport: ...} with no Timeout.
Expected
Give each attempt its own deadline, e.g. attemptCtx, cancel := context.WithTimeout(ctx, 30*time.Second) inside the retry closure. A hung response then fails that attempt, and the existing 3-attempt backoff covers hangs as well as errors. The call is idempotent (it sets a status), which is already why it is retried.
A client-level Timeout on the uiex HTTP client would also bound the other uiex calls (CreateRelease, GetRelease, ...), which have the same shape.
Summary
fly deploycan block indefinitely insideupdateReleaseInBackend(ctx, "running"), before any Machine is updated. The function wrapsuiexClient.UpdateReleasein a 3-attempt retry, but no attempt has a deadline, and the underlying HTTP client has noTimeout. A response that never arrives therefore never becomes an error, so the retry never fires.Observed
flyctl v0.4.95,
fly deploy --config <cfg> --image registry.fly.io/<app>:<sha>(image already pushed, single Machine, rolling strategy):The SIGTERM came from our own outer timeout. No Machine was touched: afterwards
fly machine listshowed the Machine stillstartedon the previous image. Re-running the identical command ~8 minutes later completed in 80 s. On a normal run,Updating existing machines ...followsWatch your deploymentwithin ~2 s (median over 37 deploys).Code path (also present on v0.4.104 at the time of writing)
internal/command/deploy/machines_deploymachinesapp.go:DeployMachinesAppcallsmd.updateReleaseInBackend(ctx, "running", nil)beforedeployMachinesApp.internal/command/deploy/machines.go:updateReleaseInBackendrunsretry.Do(..., retry.Context(ctx), retry.Attempts(releaseStatusRetryAttempts) /* 3 */, ...), and each attempt callsmd.uiexClient.UpdateRelease(ctx, ...)with the parentctx.internal/uiex/releases.go:UpdateReleaseuseshttp.NewRequestWithContext(ctx, PATCH, ...)andc.httpClient.Do(req).internal/uiex/client.gobuilds that client withfly.NewHTTPClient. Infly-gohttp.go, that returns&http.Client{Transport: ...}with noTimeout.Expected
Give each attempt its own deadline, e.g.
attemptCtx, cancel := context.WithTimeout(ctx, 30*time.Second)inside the retry closure. A hung response then fails that attempt, and the existing 3-attempt backoff covers hangs as well as errors. The call is idempotent (it sets a status), which is already why it is retried.A client-level
Timeouton the uiex HTTP client would also bound the other uiex calls (CreateRelease,GetRelease, ...), which have the same shape.