Skip to content

ci: [DO NOT MERGE] Windows test-duration instrumentation probe - #20

Closed
fercor-cisco wants to merge 12 commits into
mainfrom
ci/windows-timing-probe
Closed

ci: [DO NOT MERGE] Windows test-duration instrumentation probe#20
fercor-cisco wants to merge 12 commits into
mainfrom
ci/windows-timing-probe

Conversation

@fercor-cisco

@fercor-cisco fercor-cisco commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Investigation branch — do not merge. Diagnostic instrumentation to localize the ~1s-per-test silent overhead on Python 3.11+ Windows (3.10 Windows runs the same 2015-test suite in ~161s; 3.11/3.12/3.13/3.14 take ~1390s — a flat ~1s added to the majority of tests).

Based on fix/python-version-matrix-pinning (not main) so the diff is just the instrumentation.

What this branch changes

  • Matrix trimmed to windows-latest × [3.10, 3.11]; timeout-minutes 30 → 45.
  • Test steps replaced with layered, timestamped probes (cheap → expensive, so fast data lands even if the full run times out):
    1. Env + poetry show dump — catch a pure-Python wheel fallback or dependency diff between 3.10/3.11.
    2. Microbenchmarks (.ci/probe_microbench.py) — getaddrinfo("localtest") (the bogus host in GALILEO_CONSOLE_URL), asyncio loop churn, and GalileoPythonConfig.get() (the per-test autouse fixture), each timed in isolation.
    3. Subset isolation variants on tests/test_configuration.py, each toggling exactly one factor vs baseline: no-xdist, no-cov, and no-socket-plugin (decisive for the getaddrinfo theory).
    4. Serial subset under cProfile with DEBUG logs + ResourceWarnings — names the slow function and surfaces connection attempts.
    5. Full instrumented suite with a conftest phase-timing plugin that attributes wall time to setup/call/teardown and prints live [PHASE] lines.

Leading hypothesis

The autouse set_validated_config fixture (tests/conftest.py) runs GalileoPythonConfig.get(console_url="http://localtest:8088", ...) on every test. If name resolution of the bogus localtest host is slow on Windows (and runs before --disable-socket/--allow-hosts can block it), that's a ~1s per-test cost. These probes confirm or kill that.

🤖 Generated with Claude Code

adityamehra and others added 12 commits June 10, 2026 14:58
@adityamehra @cursoragent
ci: fix Python version mismatch and add Python 3.14 to matrix
868c7ab 
actions/setup-python@v6 with Poetry caching was using the runner's
ambient `python` (e.g. 3.12 on Ubuntu, 3.14 on macOS) to create the
venv instead of the matrix-specified version.  This caused jobs labelled
"Python 3.10/3.11/3.13" to silently execute under the wrong interpreter,
even on a cold cache.

Fix:
- Add `id: setup-python` to capture the versioned python-path output.
- Change `cache-dependency-path` from pyproject.toml → poetry.lock so
  the cache key is tied to exact locked dependencies.
- Add "Configure Poetry Python" step that wipes any stale venv with
  `poetry env remove --all` and then calls
  `poetry env use <steps.setup-python.outputs.python-path>` to pin the
  venv to the exact binary installed by setup-python.
- Add "Verify Poetry Python version" assertion that fails fast if the
  active venv ever drifts from the matrix target again.
- Extend the matrix to include Python 3.14 (crewai/litellm are already
  gated to python_version < '3.14' in pyproject.toml so they are
  skipped cleanly on that version).

Verified locally: all 2015 tests pass on 3.13; 1925 pass / 95 skip on
3.14 (expected — crewai/litellm extras excluded).

Co-authored-by: Cursor <cursoragent@cursor.com>
@adityamehra @cursoragent
ci: fix venv-before-install ordering and drop Poetry venv cache
665ba50 
Root cause of the failure: `cache: "poetry"` in setup-python restores
a potentially-wrong venv, and `poetry env remove --all` then deletes it.
At that point `poetry env use <path>` only _records_ the Python choice
without creating the venv.  When `poetry run python` fired in the next
step there was no venv yet, so Poetry fell through to the system Python
(3.14.5 on macOS), causing every job to report the wrong version.

Changes:
- Remove `cache: "poetry"` and `cache-dependency-path` from setup-python.
  Without a restored cache there is nothing to wipe, so the venv is
  always freshly created by `poetry install` using the PATH Python that
  setup-python puts first (the matrix-specified version).
- Drop `poetry env remove --all` (no stale venv to clear).
- Simplify "Configure Poetry Python" to a single
  `poetry env use <python-path>` call.
- Move "Install invoke" and "Install Dependencies" _before_ the verify
  step so the venv is fully populated before `poetry run python` executes.

Co-authored-by: Cursor <cursoragent@cursor.com>
@adityamehra @cursoragent
ci: upgrade Poetry 2.1.3 → 2.4.1 to fix virtualenv Python selection bug
e8989d2 
Poetry 2.1.3 installs the latest virtualenv (>=20.33) which has a known
bug: it ignores `poetry env use <path>` and always creates the venv with
the runner's ambient Python instead of the matrix-specified one.

This was confirmed by observing that only the jobs whose matrix Python
matched the runner default (3.12 on Ubuntu, 3.14 on macOS) passed the
version assertion, while all others failed.

Timeline of the fix upstream:
  - 2.1.4: added `virtualenv<20.33` as a temporary workaround (#10491)
  - 2.4.1: re-enabled `virtualenv>=20.33` after the underlying issue was
           fixed properly in virtualenv itself (#10506)

Upgrading to 2.4.1 (latest stable) resolves the root cause cleanly
without needing to pin virtualenv separately.

Co-authored-by: Cursor <cursoragent@cursor.com>
@adityamehra @cursoragent
ci: remove poetry env use and id: setup-python (no longer needed with…
965e64e 
… 2.4.1)

Co-authored-by: Cursor <cursoragent@cursor.com>
@adityamehra @cursoragent
ci: restore cache: poetry and cache-dependency-path
1cdcc21 
Co-authored-by: Cursor <cursoragent@cursor.com>
@adityamehra @cursoragent
ci: switch cache-dependency-path to poetry.lock to bust poisoned cache
47cccc1 
The first CI run on this branch used Poetry 2.1.3 + virtualenv>=20.33
(the wrong-Python bug), which cached venvs whose names said py3.10/py3.11
but whose executables were the runner default (3.12 on Ubuntu, 3.14 on
macOS).  Poetry matches cached venvs by name, not by the actual Python
inside, so even Poetry 2.4.1 restores and reuses the poisoned venv.

Changing cache-dependency-path from pyproject.toml to poetry.lock
changes the cache key, forcing a cold build on next run.  Poetry 2.4.1
will then create a correct venv and persist it under the new key.
All subsequent runs get the right venv from cache.

poetry.lock is also a better cache-dependency-path in general: it
captures exact resolved versions, so the cache is invalidated whenever
any transitive dependency is updated, not just direct-dep spec changes.

Co-authored-by: Cursor <cursoragent@cursor.com>
@adityamehra
trigger ci
dd1659f
@adityamehra @cursoragent
deps: pin grpcio>=1.80.0 to guarantee Python 3.14 pre-built wheels
1d12c49 
grpcio is a transitive dependency pulled in by
opentelemetry-exporter-otlp → opentelemetry-exporter-otlp-proto-grpc,
which only requires grpcio>=1.63.2/1.66.2. Any resolver (Poetry, uv,
pip) could legitimately pick an older version that has no cp314 wheels,
forcing source compilation (~20 min) on Python 3.14 CI runners and
causing the 30-minute job timeout.

Changes:
- Add grpcio (>=1.80.0,<2.0.0) as an explicit optional dependency in
  both [project.optional-dependencies] (PEP 621 / uv) and
  [tool.poetry.dependencies] (Poetry), included in the otel and all
  extras.
- Regenerate poetry.lock via `poetry lock`: resolves to grpcio 1.81.1
  (cp314 wheels available for macOS arm64, Linux x86_64/aarch64,
  Windows) while keeping grpcio 1.74.0 pinned for the crewai extra on
  Python <=3.13 where the looser constraint still applies.

Verified locally on macOS Python 3.14: install completes in ~3s
(pre-built wheel), 1925 tests pass, 95 skipped.

Co-authored-by: Cursor <cursoragent@cursor.com>
@fercor-cisco @claude
ci: [DO NOT MERGE] Windows test-duration instrumentation probe
c260afe 
Investigation branch (not for merge). Trims the CI matrix to windows-latest x
[3.10, 3.11] and replaces the test steps with layered, timestamped diagnostics
to localize the ~1s/test silent overhead seen on Python 3.11+ Windows but not
3.10:

- env + `poetry show` dump (catch a pure-Python wheel fallback / dep diff)
- microbenchmarks: getaddrinfo("localtest"), asyncio loop churn, config init
- subset isolation variants toggling one factor each: xdist, coverage, and the
  pytest-socket plugin (decisive for the getaddrinfo theory)
- serial subset under cProfile with DEBUG logs + ResourceWarnings
- full instrumented suite with a conftest phase-timing plugin that attributes
  wall time to setup/call/teardown and prints live timestamped [PHASE] lines

timeout-minutes raised 30 -> 45 so the full run isn't truncated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@fercor-cisco @claude
ci: [probe] CPython verification — Proactor vs Selector loop A/B
4338d17 
Replaces the (thread-blind) cProfile step and the 24-min full suite with a
causal experiment, since cProfile only profiles the calling thread while the
async validation work runs in galileo_core's background EventLoopThread:

- probe_microbench.py now measures cross-thread dispatch latency
  (run_coroutine_threadsafe round-trip) — the exact shape of async_run — plus
  timer-granularity probes, and honors PROBE_EVENT_LOOP=selector.
- conftest forces WindowsSelectorEventLoopPolicy when PROBE_EVENT_LOOP=selector.
- Workflow runs microbench + the trivial subset twice (Proactor vs Selector);
  compare the per-test "setup avg" in each TIMING SUMMARY.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@fercor-cisco @claude
ci: [probe] add yappi thread-aware profile of mocked config.get
f6fa9f6 
cProfile only profiles the calling thread; the validation requests run on a
background EventLoopThread. yappi (wall clock, all threads, builtins) attributes
the ~12x-more timer-quantized waits on 3.11 Windows to a concrete function and
call count. Reproduces the respx-mocked path (the real ~685ms test conditions).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@fercor-cisco @claude
ci: [probe] verify fix — bypass slow async config validation + fake.t…
c0d1216 
…est rename

On the timing probe branch, apply the proposed fix and the URI rename together,
keeping all instrumentation, to confirm via CI whether the fix removes the
Windows Py3.11+ slowness.

- conftest set_validated_config: wrap the per-test GalileoPythonConfig.get build
  in _fast_config_validation(), which stubs ApiClient.make_request/request with
  canned, await-free results. Skips the 3 async validation round-trips
  (healthcheck/login/current_user) whose Windows IOCP poll is ~11x slower on
  3.11+. Scoped to the build only; per-test reset and test-body validation are
  unchanged. (2015 passed / 5 skipped locally — no regressions.)
- Rename test host localtest -> fake.test (RFC 6761; dotted). .ci probe scripts
  keep localtest intentionally.
- Workflow: replace selector A/B + yappi with subset + full-suite timing runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Base automatically changed from fix/python-version-matrix-pinning to main June 12, 2026 23:34
@fercor-cisco
fercor-cisco deleted the ci/windows-timing-probe branch June 12, 2026 23:38
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 12, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fercor-cisco @adityamehra