FEAT: Add partner integration tests for azure-ai-evaluation red team …#1533
Merged
hannahwestra25 merged 8 commits intomicrosoft:mainfrom Apr 8, 2026
Merged
Conversation
…module Add tests/partner_integration/azure_ai_evaluation/ with contract tests validating PyRIT API stability for the azure-ai-evaluation red team module, which depends on 45+ PyRIT imports across 14 files. Test coverage includes: - PromptChatTarget interface contract (extended by 4 SDK classes) - CentralMemory/SQLiteMemory lifecycle (used in RedTeam.__init__) - Data models: Message, MessagePiece, Score, seed models, AttackResult - PromptConverter base + 19 specific converters importability - Scorer/TrueFalseScorer interface (extended by RAIServiceScorer) - Foundry scenario APIs: FoundryScenario, FoundryStrategy, DatasetConfiguration - Exception types and retry decorators - Import smoke tests for azure-ai-evaluation (skipped if not installed) Also adds partner-integration-test target to Makefile. All 84 tests pass with no Azure credentials required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix test_ascii_smuggler_converter_importable to test AsciiSmugglerConverter (was incorrectly testing AsciiArtConverter, duplicating parametrized coverage) - Move module-level asyncio.run(initialize_pyrit_async) to session-scoped fixture - Remove duplicate TestAttackModels (already covered in test_foundry_contract.py) - Extract MinimalTarget to module-level helper (was defined 3x inline) - Add docstring clarifying intentional private API imports in smoke tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tests/partner_integration/azure_ai_evaluation/test_prompt_target_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_foundry_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_import_smoke.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_model_contract.py
Outdated
Show resolved
Hide resolved
…orts, add seed model structural tests - Update PromptChatTarget to PromptTarget per PR microsoft#1532 deprecation - Move ScenarioStrategy import to top-level in test_foundry_contract.py - Add rationale for explicit inheritance checks in test_import_smoke.py - Expand seed model tests with structural validation (value, data_type, harm_categories, role, metadata, SeedGroup composition) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tests/partner_integration/azure_ai_evaluation/test_foundry_contract.py
Outdated
Show resolved
Hide resolved
hannahwestra25
approved these changes
Mar 26, 2026
Contributor
hannahwestra25
left a comment
hannahwestra25
left a comment
There was a problem hiding this comment.
a few small comments; will set up the pipeline today!
- Rename test_foundry_contract.py → test_redteam_scenario_contract.py (reviewer F1) - Add ScenarioStrategy base class test (reviewer F2 import ordering) - Add inheritance check justification comments (reviewer F3) - Expand seed model tests with prompt_group_id, sequence, context metadata patterns from PR #46151 tool context propagation (reviewer F4) - Add PromptChatTarget transitional compatibility test (reviewer F5) - Add MathPromptConverter to converter list + CharSwap naming note - Add OpenAIChatTarget contract tests - Create test_auth_contract.py for get_azure_openai_auth - Create test_orchestrator_contract.py with try/except import pattern - Expand test_memory_contract.py with 5 memory query method tests (get_scenario_results, add_scores_to_memory, get_message_pieces, get_prompt_request_pieces, get_conversation) - Add MessagePiece.prompt_metadata field tests for context extraction All 119 tests pass. ACA has zero direct PyRIT imports — SDK contract tests provide transitive coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tests/partner_integration/azure_ai_evaluation/test_orchestrator_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_orchestrator_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_converter_contract.py
Outdated
Show resolved
Hide resolved
- Remove test_orchestrator_contract.py (orchestrators removed from PyRIT) - Add _restore_central_memory fixture to prevent state leakage - Remove redundant 'assert X is not None' after top-level imports; convert existence-only tests to local imports so they actually validate importability instead of being dead code 104 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tests/partner_integration/azure_ai_evaluation/test_prompt_target_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_memory_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_memory_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_model_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_model_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_foundry_scenario_contract.py
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_redteam_scenario_contract.py
Outdated
Show resolved
Hide resolved
tests/partner_integration/azure_ai_evaluation/test_foundry_scenario_contract.py
Show resolved
Hide resolved
- Fix _MinimalTarget.send_prompt_async signature to match PyRIT - Rename test_memory_contract → test_sqlite_memory_contract - Use sqlite_instance fixture for memory query tests (eliminate boilerplate) - Add data_type/prompt_group_id/value assertions to context pattern test - Remove is_objective pattern tests (deprecated PyRIT metadata pattern) - Rename test_redteam_scenario_contract → test_foundry_scenario_contract 103 tests pass. Pre-commit (ruff-format, ruff-check, trailing-whitespace, end-of-file-fixer) all pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…module
Add tests/partner_integration/azure_ai_evaluation/ with contract tests validating PyRIT API stability for the azure-ai-evaluation red team module, which depends on 45+ PyRIT imports across 14 files.
Test coverage includes:
Also adds partner-integration-test target to Makefile.
All 84 tests pass with no Azure credentials required.
Description
Tests and Documentation