fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
Conversation
The deepeval adapter imports deepeval at module scope (src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval unit tests could not be collected, exiting pytest with code 2 and cancelling the whole matrix. Both are pinned with an upper bound so major versions cannot drift in. The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter imports deepeval.test_case.conversational_test_case.Turn, which does not exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with strands-agents-evals (rich >=14). Corrected to >=3.5.0.
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
tejaskash
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ci.ymlunit tests have been failing onmainsince 9b7d38e (#568), on all four Python versions:src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6importsdeepevalat module scope:But
deepevalwas not in thedevdependency group, andci.ymlinstalls viauv sync --dev. So the tests could not even be collected — pytest exits 2, andfail-fastcancels the whole matrix.The
autoevalssibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.Change
Add both to the
devgroup, pinned with an upper bound so a new major version cannot drift in:Also corrects an unsatisfiable floor in the
deepevalextra since ConversationalTurn was introduced in v3Testing
uv lockregenerated; diff adds only the two packages, no other version moveduv sync --dev(CI's exact command) thenuv run pytest tests/on Python 3.10 → the 2 collection errors are gone; 2930 passeddeepeval==3.5.0,autoevals==0.3.0) and current latest (deepeval==4.1.4) → 108 passed eachtest_label_required_without_error_code,test_evaluation_with_empty_trajectory, 2 ×TestAsyncMode) fail identically on cleanmain— pre-existing, out of scope hereFollow-up worth considering
deepevalis declared optional inpyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.Separate from #608, which fixes the integration-test workflow.