ci: fix Python version mismatch and add Python 3.14 to matrix - #18
Merged
Conversation
actions/setup-python@v6 with Poetry caching was using the runner's ambient `python` (e.g. 3.12 on Ubuntu, 3.14 on macOS) to create the venv instead of the matrix-specified version. This caused jobs labelled "Python 3.10/3.11/3.13" to silently execute under the wrong interpreter, even on a cold cache. Fix: - Add `id: setup-python` to capture the versioned python-path output. - Change `cache-dependency-path` from pyproject.toml → poetry.lock so the cache key is tied to exact locked dependencies. - Add "Configure Poetry Python" step that wipes any stale venv with `poetry env remove --all` and then calls `poetry env use <steps.setup-python.outputs.python-path>` to pin the venv to the exact binary installed by setup-python. - Add "Verify Poetry Python version" assertion that fails fast if the active venv ever drifts from the matrix target again. - Extend the matrix to include Python 3.14 (crewai/litellm are already gated to python_version < '3.14' in pyproject.toml so they are skipped cleanly on that version). Verified locally: all 2015 tests pass on 3.13; 1925 pass / 95 skip on 3.14 (expected — crewai/litellm extras excluded). Co-authored-by: Cursor <cursoragent@cursor.com>
Root cause of the failure: `cache: "poetry"` in setup-python restores a potentially-wrong venv, and `poetry env remove --all` then deletes it. At that point `poetry env use <path>` only _records_ the Python choice without creating the venv. When `poetry run python` fired in the next step there was no venv yet, so Poetry fell through to the system Python (3.14.5 on macOS), causing every job to report the wrong version. Changes: - Remove `cache: "poetry"` and `cache-dependency-path` from setup-python. Without a restored cache there is nothing to wipe, so the venv is always freshly created by `poetry install` using the PATH Python that setup-python puts first (the matrix-specified version). - Drop `poetry env remove --all` (no stale venv to clear). - Simplify "Configure Poetry Python" to a single `poetry env use <python-path>` call. - Move "Install invoke" and "Install Dependencies" _before_ the verify step so the venv is fully populated before `poetry run python` executes. Co-authored-by: Cursor <cursoragent@cursor.com>
Poetry 2.1.3 installs the latest virtualenv (>=20.33) which has a known
bug: it ignores `poetry env use <path>` and always creates the venv with
the runner's ambient Python instead of the matrix-specified one.
This was confirmed by observing that only the jobs whose matrix Python
matched the runner default (3.12 on Ubuntu, 3.14 on macOS) passed the
version assertion, while all others failed.
Timeline of the fix upstream:
- 2.1.4: added `virtualenv<20.33` as a temporary workaround (#10491)
- 2.4.1: re-enabled `virtualenv>=20.33` after the underlying issue was
fixed properly in virtualenv itself (#10506)
Upgrading to 2.4.1 (latest stable) resolves the root cause cleanly
without needing to pin virtualenv separately.
Co-authored-by: Cursor <cursoragent@cursor.com>
… 2.4.1) Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The first CI run on this branch used Poetry 2.1.3 + virtualenv>=20.33 (the wrong-Python bug), which cached venvs whose names said py3.10/py3.11 but whose executables were the runner default (3.12 on Ubuntu, 3.14 on macOS). Poetry matches cached venvs by name, not by the actual Python inside, so even Poetry 2.4.1 restores and reuses the poisoned venv. Changing cache-dependency-path from pyproject.toml to poetry.lock changes the cache key, forcing a cold build on next run. Poetry 2.4.1 will then create a correct venv and persist it under the new key. All subsequent runs get the right venv from cache. poetry.lock is also a better cache-dependency-path in general: it captures exact resolved versions, so the cache is invalidated whenever any transitive dependency is updated, not just direct-dep spec changes. Co-authored-by: Cursor <cursoragent@cursor.com>
keith-decker
previously approved these changes
Jun 11, 2026
grpcio is a transitive dependency pulled in by opentelemetry-exporter-otlp → opentelemetry-exporter-otlp-proto-grpc, which only requires grpcio>=1.63.2/1.66.2. Any resolver (Poetry, uv, pip) could legitimately pick an older version that has no cp314 wheels, forcing source compilation (~20 min) on Python 3.14 CI runners and causing the 30-minute job timeout. Changes: - Add grpcio (>=1.80.0,<2.0.0) as an explicit optional dependency in both [project.optional-dependencies] (PEP 621 / uv) and [tool.poetry.dependencies] (Poetry), included in the otel and all extras. - Regenerate poetry.lock via `poetry lock`: resolves to grpcio 1.81.1 (cp314 wheels available for macOS arm64, Linux x86_64/aarch64, Windows) while keeping grpcio 1.74.0 pinned for the crewai extra on Python <=3.13 where the looser constraint still applies. Verified locally on macOS Python 3.14: install completes in ~3s (pre-built wheel), 1925 tests pass, 95 skipped. Co-authored-by: Cursor <cursoragent@cursor.com>
adityamehra
force-pushed
the
fix/python-version-matrix-pinning
branch
from
June 12, 2026 18:12
bbc6a99 to
1d12c49
Compare
fercor-cisco
approved these changes
Jun 12, 2026
fercor-cisco
left a comment
Collaborator
There was a problem hiding this comment.
Note: The PR description is stale. It describes a poetry env remove --all && poetry env use step and id: setup-python that no longer exist — they were removed in commit 965e64e once Poetry was bumped to 2.4.1. The real fix is the version bump + assertion, not explicit venv recreation. This is the one thing I'd ask be corrected before merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI jobs were silently running under the wrong Python interpreter — a job labeled Python 3.10 would execute with Python 3.12 (Ubuntu default) or Python 3.14 (macOS default). A validation step was added to assert the active interpreter matched the matrix target; every non-default version failed.
Additionally, Python 3.14 install jobs on macOS were hitting the 30-minute timeout due to source compilation of a C++ dependency.
Root-cause chain
1 — virtualenv >= 20.33 ignores poetry env use (Poetry 2.1.3)
actions/setup-python@v6 with cache: "poetry" internally calls poetry env use . In Poetry 2.1.3 + virtualenv ≥ 20.33 this call is silently ignored: the venv is always created with the runner's ambient Python (3.12 on Ubuntu, 3.14 on macOS). The resulting wrong-Python venv is then written to the cache under a key named for the matrix Python (e.g. …-3.10-…).
Evidence: only the jobs whose matrix Python matched the runner default passed — Python 3.12 on Linux and Python 3.14 on macOS.
2 — Poisoned cache survives even after the Poetry fix
Once Poetry was upgraded to 2.4.1 (which correctly handles virtualenv), the stale 3.12-in-a-3.10-labelled venv was still being restored from cache. Poetry matches cached venvs by name, not by the actual Python binary inside, so it reused the corrupted entry without complaint.
3 — grpcio 1.74.0 has no cp314 wheels
opentelemetry-exporter-otlp → opentelemetry-exporter-otlp-proto-grpc pulls in grpcio with only a loose lower bound (>=1.63.2). On a cold cache, pip found no pre-built cp314 wheel for grpcio 1.74.0 and compiled the C++ library from source — ~20 min on macOS arm64, exceeding the 30-min job timeout.
Changes
poetry==2.1.3 → poetry==2.4.1 — core fix
Poetry 2.1.4 added virtualenv<20.33 as a temporary workaround. Poetry 2.4.1 re-enabled virtualenv>=20.33 after the underlying virtualenv bug was fixed upstream. Upgrading to 2.4.1 resolves the root cause cleanly.
cache-dependency-path: "pyproject.toml" → "poetry.lock"
Changing the cache-dependency-path produces a new cache key, evicting all poisoned venvs written under the old Poetry version. poetry.lock is also a better cache signal: it invalidates whenever any transitive dependency is updated, not only when a direct constraint changes.
Removed id: setup-python and Configure Poetry Python step
These were scaffolding added to work around the poetry env use failure. With Poetry 2.4.1 the venv is created correctly by poetry install using the PATH Python placed there by setup-python — no extra wiring needed.
Python 3.14 added to the test matrix
crewai and litellm are already gated to python_version < "3.14" in pyproject.toml, so they are skipped cleanly. All other extras install and pass.
Added grpcio (>=1.80.0,<2.0.0) as an explicit optional dependency
grpcio was a transitive dep with a loose lower bound. Any resolver (Poetry, uv, pip) could legitimately pick a version below 1.80.0, which has no cp314 wheels. grpcio 1.80.0 is the first release with pre-built wheels for Python 3.14 on all platforms.
The constraint is added in both [project.optional-dependencies] (PEP 621 — read by uv/pip) and [tool.poetry.dependencies] (read by Poetry), under the otel and all extras. poetry.lock is regenerated via poetry lock and resolves to grpcio 1.81.1.