Skip to content

ci: install deepeval and autoevals for evaluation integ tests - #608

Merged
jariy17 merged 1 commit into
mainfrom
fix/eval-integ-third-party-deps
Jul 31, 2026
Merged

ci: install deepeval and autoevals for evaluation integ tests#608
jariy17 merged 1 commit into
mainfrom
fix/eval-integ-third-party-deps

Conversation

@jariy17

@jariy17 jariy17 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Test (evaluation) in the Secure Integration test workflow is failing on main (run 30494098538):

ERROR tests_integ/evaluation/test_third_party_adapters.py::TestDeepEvalAdapterIntegration::test_bias_metric_passes - ModuleNotFoundError: No module named 'deepeval'
ERROR tests_integ/evaluation/test_third_party_adapters.py::TestAutoEvalsAdapterIntegration::test_factuality_scorer - ModuleNotFoundError: No module named 'autoevals'
======= 36 passed, 1 skipped, 41 warnings, 6 errors in 78.52s (0:01:18) ========

test_third_party_adapters.py (added in #568) imports deepeval and autoevals, but the evaluation matrix group only installed strands-agents-evals. The bare import calls in the autouse fixtures raise ModuleNotFoundError, which pytest reports as errors rather than skips, so the job exits 1.

Both packages are already declared as optional extras in pyproject.toml ([deepeval], [autoevals]) — they were simply never installed in that job.

Change

One line — add both packages to the evaluation group's extra-deps.

Testing

  • python3 -c "yaml.safe_load(...)" — workflow YAML parses, matrix entry correct
  • Installed the full evaluation dep set on Python 3.10 (CI's version); deepeval==4.1.4 and autoevals both resolve and import cleanly, no conflicts with strands-agents-evals==1.0.3
  • Confirmed the two ModuleNotFoundErrors are gone, and confirmed the residual OpenAI-credential failures documented above

tests_integ/evaluation/test_third_party_adapters.py imports deepeval and
autoevals, but the evaluation matrix group only installed
strands-agents-evals. The bare imports in the autouse fixtures raised
ModuleNotFoundError, which pytest reports as errors (not skips), failing
the job with exit code 1.

Both packages are already declared as optional extras in pyproject.toml.
@jariy17
jariy17 requested a review from a team July 30, 2026 22:33
@github-actions github-actions Bot added the size/xs PR size: XS label Jul 30, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jul 30, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@jariy17
jariy17 merged commit 5d23292 into main Jul 31, 2026
29 of 34 checks passed
@jariy17
jariy17 deleted the fix/eval-integ-third-party-deps branch July 31, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xs PR size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants