[nstepien]

playwright install --no-shell --with-deps kept the vp run ci step from finishing within its 6 minutes in 46 of the 237 CI attempts from 2026-10-21 to 10-27. All of them ran on runners in Azure's westus (12 of 21 attempts) and westus3 (34 of 46) regions. None of the 170 attempts in other regions hung.

Why apt hung

GitHub's Ubuntu images list azure.archive.ubuntu.com first in /etc/apt/apt-mirrors.txt. It resolves to a mirror in the runner's region, like cloud-mirror-lb.westus3.cloudapp.azure.com. The westus and westus3 mirrors serve the package lists quickly, but relay the packages through a Squid proxy that almost always misses its cache (X-Cache: MISS from 192.168.0.4, Via: 1.1 192.168.0.4 (squid/5.2)), at 40–60 kB/s, so the 119 MB of packages would take about 40 minutes. In a job that I let run for 15 minutes, apt downloaded 47 of the ~190 packages.

apt only moves on to the next mirror after an error, and Acquire::http::Timeout only fires when no data arrives for that long. Data arrived every 5–90 ms, so apt never timed out, or tried archive.ubuntu.com, the next mirror in the list. This is actions/runner-images#14823, open since 2026-09-30.

.github/apt.conf's retries and timeouts never applied either, as the image's /etc/apt/apt.conf.d/zz-retries sorts after the 99-ci that we copied it to. Only its quiet "2" applied, which hid where apt was stuck.

The other mirrors slow down at times too. In one sample, archive.ubuntu.com downloaded at under 0.5 MB/s for 4 of 14 jobs running at the same time, and in another, the eastus mirror was slow for all 6 jobs that ran there. A different mirror order alone would only move the problem.

Changes

• On CI, the install-browsers task runs .github/install-browsers.sh. It runs Playwright's apt-get install in download-only mode, in attempts of 30 seconds, as the downloads normally take about 10. After an attempt that downloads less than 10 MB, the next one switches between archive.ubuntu.com (.github/apt-mirrors.txt) and the runner image's mirrors, and the fifth such attempt fails. Each attempt keeps the packages that the previous ones downloaded. playwright install --no-shell --with-deps then installs them, so the time limit can't interrupt dpkg.
• The downloads start from archive.ubuntu.com, as in GitHub's proposed hotfix, Hotfix: demote azure.archive.ubuntu.com mirror priority, add apt acquire timeouts actions/runner-images#14596.
• .github/apt.conf turns on pipelining over HTTPS, which the image turns off. Downloading the packages from archive.ubuntu.com took 5–16 s with it (median 8.5 s) and 23–169 s without it (median 60.5 s), in 10 jobs each that ran at the same time. It's now installed as /etc/apt/apt.conf, which apt reads after /etc/apt/apt.conf.d/. Its ineffective retries and timeouts are gone, and so is quiet "2", so the log shows which mirror served each package, and how fast.
• When timeout stops Playwright, sudo only relays the signal to the sh running apt-get, so the orphaned apt-get kept downloading and held apt's locks, which failed the next attempt. The script kills it.

Results

I ran vp run install-browsers in 3 batches of 20 jobs with the final script, and all 58 jobs that ran it succeeded (in the first batch, my setup for 2 of them failed before they did):

• 46 jobs as in CI, 14 of them in westus or westus3: 42–128 s for the whole task (median 54 s), against about 45 s before, when the Azure mirror was healthy. 12 of them needed more than one attempt, because of slow mirrors.
• 6 jobs whose first mirror list was the degraded westus mirror, pinned in /etc/hosts: 68–156 s.
• 6 jobs that dropped archive.ubuntu.com's packets above 1500/s: 48–135 s.

CI passed on this PR's last commit, which ran in westus. Before the fix, CI hung on this PR in westus3.

Upstream

actions/runner-images#14823 tracks the slow Azure mirrors.

🤖 Generated with Claude Code