Skip to content

GH-49057: [Python][Packaging] Remove test base docker image and use single windows base image for both build and test containers #48996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
raulcd wants to merge 10 commits into
base: main
Choose a base branch
from
Open

GH-49057: [Python][Packaging] Remove test base docker image and use single windows base image for both build and test containers #48996

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ac76f96
WIP: [Python] Simplify Windows wheel build ecosystem
raulcd Jan 26, 2026
054b8dc
Add python_variant_suffix for testing free-threaded
raulcd Jan 27, 2026
a07afbf
Try upgrading wheel windows image revision
raulcd Jan 27, 2026
e1ab7da
Fix base image
raulcd Jan 28, 2026
6665098
Remove unnecessary change
raulcd Jan 28, 2026
2aa3a1f
Base for testing images should be the base image not the build image
raulcd Jan 28, 2026
1ca2938
Dockerfiles for test should specify PYTHON_CMD
raulcd Jan 28, 2026
f4d33c2
Use pymanager to install Python for GCS testbench
raulcd Jan 29, 2026
eac026f
Try removing python installation and see if there's a python on the b…
raulcd Jan 30, 2026
98410dd
Use base image Python instead for gcs
raulcd Jan 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ VCPKG="4334d8b4c8916018600212ab4dd4bbdc343065d1" # 2025.09.17 Release
# ci/docker/python-*-windows-*.dockerfile or the vcpkg config.
# This is a workaround for our CI problem that "archery docker build" doesn't
# use pulled built images in dev/tasks/python-wheels/github.windows.yml.
PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2026-01-22
PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2026-01-22
PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2026-01-27
PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2026-01-27

# Use conanio/${CONAN_BASE}:{CONAN_VERSION} for "docker compose run --rm conan".
# See https://github.com/conan-io/conan-docker-tools#readme and
Expand Down
18 changes: 6 additions & 12 deletions ci/docker/python-free-threaded-wheel-windows-test-vs2022.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,12 @@ FROM ${base}

ARG python=3.13

# hadolint ignore=SC1072
RUN (if "%python%"=="3.13" setx PYTHON_VERSION "3.13.1") & \
(if "%python%"=="3.14" setx PYTHON_VERSION "3.14.0")

SHELL ["powershell", "-NoProfile", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN $version = $env:PYTHON_VERSION; \
$filename = 'python-' + $version + '-amd64.exe'; \
$url = 'https://www.python.org/ftp/python/' + $version + '/' + $filename; \
Invoke-WebRequest -Uri $url -OutFile $filename; \
Start-Process -FilePath $filename -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait

ENV PYTHON_CMD="py -${python}t"
ARG python_variant_suffix=""
ENV PYTHON_VERSION=${python}${python_variant_suffix}

RUN pymanager install %PYTHON_VERSION%

ENV PYTHON_CMD="py -${python}${python_variant_suffix}"

SHELL ["cmd", "/S", "/C"]
RUN %PYTHON_CMD% -m pip install -U pip setuptools & \
Expand Down
65 changes: 0 additions & 65 deletions ci/docker/python-wheel-windows-test-vs2022-base.dockerfile
View file
Open in desktop

This file was deleted.

15 changes: 7 additions & 8 deletions ci/docker/python-wheel-windows-test-vs2022.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,15 @@ FROM ${base}

# hadolint shell=cmd.exe

# Define the full version number otherwise choco falls back to patch number 0 (3.10 => 3.10.0)
ARG python=3.10
RUN (if "%python%"=="3.10" setx PYTHON_VERSION "3.10.11" && setx PYTHON_CMD "py -3.10") & \
(if "%python%"=="3.11" setx PYTHON_VERSION "3.11.9" && setx PYTHON_CMD "py -3.11") & \
(if "%python%"=="3.12" setx PYTHON_VERSION "3.12.10" && setx PYTHON_CMD "py -3.12") & \
(if "%python%"=="3.13" setx PYTHON_VERSION "3.13.9" && setx PYTHON_CMD "py -3.13") & \
(if "%python%"=="3.14" setx PYTHON_VERSION "3.14.0" && setx PYTHON_CMD "py -3.14")

# hadolint ignore=DL3059
RUN choco install -r -y --pre --no-progress --force python --version=%PYTHON_VERSION%
ARG python_variant_suffix=""
ENV PYTHON_VERSION=${python}${python_variant_suffix}

RUN pymanager install %PYTHON_VERSION%

ENV PYTHON_CMD="py -${python}${python_variant_suffix}"

# hadolint ignore=DL3059
RUN %PYTHON_CMD% -m pip install -U pip setuptools

Expand Down
10 changes: 10 additions & 0 deletions ci/docker/python-wheel-windows-vs2022-base.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,16 @@ RUN choco install --no-progress -r -y git gzip ninja wget
# Add UNIX tools to PATH
RUN setx path "%path%;C:\Program Files\Git\usr\bin"

# Install git, wget, minio
RUN choco install --no-progress -r -y git wget
RUN curl https://dl.min.io/server/minio/release/windows-amd64/archive/minio.RELEASE.2025-01-20T14-49-07Z `
--output "C:\Windows\Minio.exe"

# Install the GCS testbench using a well-known Python version.
COPY ci/scripts/install_gcs_testbench.bat C:/arrow/ci/scripts/
RUN call "C:\arrow\ci\scripts\install_gcs_testbench.bat" && `
storage-testbench -h

# Install vcpkg
#
# Compiling vcpkg itself from a git tag doesn't work anymore since vcpkg has
Expand Down
1 change: 0 additions & 1 deletion ci/docker/python-wheel-windows-vs2022.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
ARG base
FROM ${base}

# Define the full version number otherwise choco falls back to patch number 0 (3.10 => 3.10.0)
ARG python=3.10

ARG python_variant_suffix=""
Expand Down
9 changes: 3 additions & 6 deletions ci/scripts/install_gcs_testbench.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,12 @@
set GCS_TESTBENCH_VERSION="v0.55.0"

set PIPX_FLAGS=--verbose
if NOT "%PIPX_PYTHON%"=="" (
set PIPX_FLAGS=--python %PIPX_PYTHON% %PIPX_FLAGS%
)

python -m pip install -U pipx || exit /B 1
python -m pip install -U pipx|| exit /B 1

@REM Install GCS testbench %GCS_TESTBENCH_VERSION%
pipx install %PIPX_FLAGS% ^
python -m pipx install --fetch-missing-python --python 3.11 %PIPX_FLAGS% ^
"https://github.com/googleapis/storage-testbench/archive/%GCS_TESTBENCH_VERSION%.tar.gz" ^
|| exit /B 1

pipx list --verbose
python -m pipx list --verbose
13 changes: 3 additions & 10 deletions compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,6 @@ x-hierarchy:
- python-wheel-windows-vs2022-base:
- python-wheel-windows-vs2022
- python-free-threaded-wheel-windows-vs2022
- python-wheel-windows-test-base:
- python-wheel-windows-test
- python-free-threaded-wheel-windows-test

Expand Down Expand Up @@ -1416,18 +1415,11 @@ services:
volumes: *python-wheel-windows-vs2022-volumes
command: arrow\\ci\\scripts\\python_wheel_windows_build.bat

python-wheel-windows-test-base:
image: ${REPO}:python-wheel-windows-test-vs2022-base-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
build:
context: .
dockerfile: ci/docker/python-wheel-windows-test-vs2022-base.dockerfile
volumes: *python-wheel-windows-vs2022-volumes

python-wheel-windows-test:
image: ${REPO}:python-${PYTHON}-wheel-windows-test-vs2022-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
build:
args:
base: ${REPO}:python-wheel-windows-test-vs2022-base-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
base: ${REPO}:python-wheel-windows-vs2022-base-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
python: ${PYTHON}
context: .
dockerfile: ci/docker/python-wheel-windows-test-vs2022.dockerfile
Expand All @@ -1438,8 +1430,9 @@ services:
image: ${REPO}:python-${PYTHON}-free-threaded-wheel-windows-test-vs2022-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
build:
args:
base: ${REPO}:python-wheel-windows-test-vs2022-base-${PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION}
base: ${REPO}:python-wheel-windows-vs2022-base-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
python: ${PYTHON}
python_variant_suffix: t
context: .
dockerfile: ci/docker/python-free-threaded-wheel-windows-test-vs2022.dockerfile
volumes: *python-wheel-windows-vs2022-volumes
Expand Down
Loading