Skip to content

ci: fix Python version mismatch and add Python 3.14 to matrix - #18

Merged
adityamehra merged 8 commits into
mainfrom
fix/python-version-matrix-pinning
Jun 12, 2026
Merged

ci: fix Python version mismatch and add Python 3.14 to matrix#18
adityamehra merged 8 commits into
mainfrom
fix/python-version-matrix-pinning

Conversation

@adityamehra

@adityamehra adityamehra commented Jun 10, 2026

Copy link
Copy Markdown
Member

Problem

CI jobs were silently running under the wrong Python interpreter — a job labeled Python 3.10 would execute with Python 3.12 (Ubuntu default) or Python 3.14 (macOS default). A validation step was added to assert the active interpreter matched the matrix target; every non-default version failed.

Additionally, Python 3.14 install jobs on macOS were hitting the 30-minute timeout due to source compilation of a C++ dependency.

Root-cause chain

1 — virtualenv >= 20.33 ignores poetry env use (Poetry 2.1.3)

actions/setup-python@v6 with cache: "poetry" internally calls poetry env use . In Poetry 2.1.3 + virtualenv ≥ 20.33 this call is silently ignored: the venv is always created with the runner's ambient Python (3.12 on Ubuntu, 3.14 on macOS). The resulting wrong-Python venv is then written to the cache under a key named for the matrix Python (e.g. …-3.10-…).

Evidence: only the jobs whose matrix Python matched the runner default passed — Python 3.12 on Linux and Python 3.14 on macOS.

2 — Poisoned cache survives even after the Poetry fix

Once Poetry was upgraded to 2.4.1 (which correctly handles virtualenv), the stale 3.12-in-a-3.10-labelled venv was still being restored from cache. Poetry matches cached venvs by name, not by the actual Python binary inside, so it reused the corrupted entry without complaint.

3 — grpcio 1.74.0 has no cp314 wheels

opentelemetry-exporter-otlp → opentelemetry-exporter-otlp-proto-grpc pulls in grpcio with only a loose lower bound (>=1.63.2). On a cold cache, pip found no pre-built cp314 wheel for grpcio 1.74.0 and compiled the C++ library from source — ~20 min on macOS arm64, exceeding the 30-min job timeout.

Changes

poetry==2.1.3 → poetry==2.4.1 — core fix

Poetry 2.1.4 added virtualenv<20.33 as a temporary workaround. Poetry 2.4.1 re-enabled virtualenv>=20.33 after the underlying virtualenv bug was fixed upstream. Upgrading to 2.4.1 resolves the root cause cleanly.

cache-dependency-path: "pyproject.toml" → "poetry.lock"

Changing the cache-dependency-path produces a new cache key, evicting all poisoned venvs written under the old Poetry version. poetry.lock is also a better cache signal: it invalidates whenever any transitive dependency is updated, not only when a direct constraint changes.

Removed id: setup-python and Configure Poetry Python step

These were scaffolding added to work around the poetry env use failure. With Poetry 2.4.1 the venv is created correctly by poetry install using the PATH Python placed there by setup-python — no extra wiring needed.

Python 3.14 added to the test matrix

crewai and litellm are already gated to python_version < "3.14" in pyproject.toml, so they are skipped cleanly. All other extras install and pass.

Added grpcio (>=1.80.0,<2.0.0) as an explicit optional dependency

grpcio was a transitive dep with a loose lower bound. Any resolver (Poetry, uv, pip) could legitimately pick a version below 1.80.0, which has no cp314 wheels. grpcio 1.80.0 is the first release with pre-built wheels for Python 3.14 on all platforms.

The constraint is added in both [project.optional-dependencies] (PEP 621 — read by uv/pip) and [tool.poetry.dependencies] (read by Poetry), under the otel and all extras. poetry.lock is regenerated via poetry lock and resolves to grpcio 1.81.1.

adityamehra and others added 7 commits June 10, 2026 14:58
actions/setup-python@v6 with Poetry caching was using the runner's
ambient `python` (e.g. 3.12 on Ubuntu, 3.14 on macOS) to create the
venv instead of the matrix-specified version.  This caused jobs labelled
"Python 3.10/3.11/3.13" to silently execute under the wrong interpreter,
even on a cold cache.

Fix:
- Add `id: setup-python` to capture the versioned python-path output.
- Change `cache-dependency-path` from pyproject.toml → poetry.lock so
  the cache key is tied to exact locked dependencies.
- Add "Configure Poetry Python" step that wipes any stale venv with
  `poetry env remove --all` and then calls
  `poetry env use <steps.setup-python.outputs.python-path>` to pin the
  venv to the exact binary installed by setup-python.
- Add "Verify Poetry Python version" assertion that fails fast if the
  active venv ever drifts from the matrix target again.
- Extend the matrix to include Python 3.14 (crewai/litellm are already
  gated to python_version < '3.14' in pyproject.toml so they are
  skipped cleanly on that version).

Verified locally: all 2015 tests pass on 3.13; 1925 pass / 95 skip on
3.14 (expected — crewai/litellm extras excluded).

Co-authored-by: Cursor <cursoragent@cursor.com>
Root cause of the failure: `cache: "poetry"` in setup-python restores
a potentially-wrong venv, and `poetry env remove --all` then deletes it.
At that point `poetry env use <path>` only _records_ the Python choice
without creating the venv.  When `poetry run python` fired in the next
step there was no venv yet, so Poetry fell through to the system Python
(3.14.5 on macOS), causing every job to report the wrong version.

Changes:
- Remove `cache: "poetry"` and `cache-dependency-path` from setup-python.
  Without a restored cache there is nothing to wipe, so the venv is
  always freshly created by `poetry install` using the PATH Python that
  setup-python puts first (the matrix-specified version).
- Drop `poetry env remove --all` (no stale venv to clear).
- Simplify "Configure Poetry Python" to a single
  `poetry env use <python-path>` call.
- Move "Install invoke" and "Install Dependencies" _before_ the verify
  step so the venv is fully populated before `poetry run python` executes.

Co-authored-by: Cursor <cursoragent@cursor.com>
Poetry 2.1.3 installs the latest virtualenv (>=20.33) which has a known
bug: it ignores `poetry env use <path>` and always creates the venv with
the runner's ambient Python instead of the matrix-specified one.

This was confirmed by observing that only the jobs whose matrix Python
matched the runner default (3.12 on Ubuntu, 3.14 on macOS) passed the
version assertion, while all others failed.

Timeline of the fix upstream:
  - 2.1.4: added `virtualenv<20.33` as a temporary workaround (#10491)
  - 2.4.1: re-enabled `virtualenv>=20.33` after the underlying issue was
           fixed properly in virtualenv itself (#10506)

Upgrading to 2.4.1 (latest stable) resolves the root cause cleanly
without needing to pin virtualenv separately.

Co-authored-by: Cursor <cursoragent@cursor.com>
… 2.4.1)

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The first CI run on this branch used Poetry 2.1.3 + virtualenv>=20.33
(the wrong-Python bug), which cached venvs whose names said py3.10/py3.11
but whose executables were the runner default (3.12 on Ubuntu, 3.14 on
macOS).  Poetry matches cached venvs by name, not by the actual Python
inside, so even Poetry 2.4.1 restores and reuses the poisoned venv.

Changing cache-dependency-path from pyproject.toml to poetry.lock
changes the cache key, forcing a cold build on next run.  Poetry 2.4.1
will then create a correct venv and persist it under the new key.
All subsequent runs get the right venv from cache.

poetry.lock is also a better cache-dependency-path in general: it
captures exact resolved versions, so the cache is invalidated whenever
any transitive dependency is updated, not just direct-dep spec changes.

Co-authored-by: Cursor <cursoragent@cursor.com>
keith-decker
keith-decker previously approved these changes Jun 11, 2026
grpcio is a transitive dependency pulled in by
opentelemetry-exporter-otlp → opentelemetry-exporter-otlp-proto-grpc,
which only requires grpcio>=1.63.2/1.66.2. Any resolver (Poetry, uv,
pip) could legitimately pick an older version that has no cp314 wheels,
forcing source compilation (~20 min) on Python 3.14 CI runners and
causing the 30-minute job timeout.

Changes:
- Add grpcio (>=1.80.0,<2.0.0) as an explicit optional dependency in
  both [project.optional-dependencies] (PEP 621 / uv) and
  [tool.poetry.dependencies] (Poetry), included in the otel and all
  extras.
- Regenerate poetry.lock via `poetry lock`: resolves to grpcio 1.81.1
  (cp314 wheels available for macOS arm64, Linux x86_64/aarch64,
  Windows) while keeping grpcio 1.74.0 pinned for the crewai extra on
  Python <=3.13 where the looser constraint still applies.

Verified locally on macOS Python 3.14: install completes in ~3s
(pre-built wheel), 1925 tests pass, 95 skipped.

Co-authored-by: Cursor <cursoragent@cursor.com>
@adityamehra
adityamehra force-pushed the fix/python-version-matrix-pinning branch from bbc6a99 to 1d12c49 Compare June 12, 2026 18:12

@fercor-cisco fercor-cisco left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The PR description is stale. It describes a poetry env remove --all && poetry env use step and id: setup-python that no longer exist — they were removed in commit 965e64e once Poetry was bumped to 2.4.1. The real fix is the version bump + assertion, not explicit venv recreation. This is the one thing I'd ask be corrected before merge.

@adityamehra
adityamehra merged commit 7469273 into main Jun 12, 2026
29 of 31 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 12, 2026
@fercor-cisco
fercor-cisco deleted the fix/python-version-matrix-pinning branch June 12, 2026 23:34
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants