Skip to content

Add MLflow scorers integration page#1829

Open
debu-sinha wants to merge 4 commits into
google:mainfrom
debu-sinha:feature/mlflow-scorers-integration
Open

Add MLflow scorers integration page#1829
debu-sinha wants to merge 4 commits into
google:mainfrom
debu-sinha:feature/mlflow-scorers-integration

Conversation

@debu-sinha

Copy link
Copy Markdown

Summary

Adds a new integration page at docs/integrations/mlflow-scorers.md documenting MLflow's five Google ADK scorers for agent evaluation. The integration wraps ADK's TrajectoryEvaluator, RougeEvaluator, FinalResponseMatchV2Evaluator, SafetyEvaluatorV1, and HallucinationsV1Evaluator behind MLflow's scorer interface, so ADK users can evaluate agents inside mlflow.genai.evaluate() runs.

Complements the existing MLflow Tracing and MLflow AI Gateway integration pages by covering evaluation, the third leg of the MLflow stack for ADK.

Changes

  • docs/integrations/mlflow-scorers.md — new page covering:
    • Use cases, prerequisites, install dependencies
    • Available scorers table (5 scorers grouped by deterministic vs LLM-judge)
    • Quick start examples (single scorer and mlflow.genai.evaluate composition)
    • How tool calls are resolved (expectations[\"actual_tool_calls\"] and TOOL span fallback)
    • LLM-judge configuration including the Safety Vertex AI requirement
    • Cross-links to the existing MLflow Tracing and Gateway integration pages

Why this is useful for ADK users

ADK already exposes a rich evaluator suite in google.adk.evaluation, but users running evaluations in MLflow-tracked agent applications previously had to build their own glue code to invoke ADK evaluators on MLflow data. The integration removes that glue. ADK evaluators can now be composed in the same mlflow.genai.evaluate() call as MLflow's built-in or other third-party scorers (TruLens, DeepEval, Guardrails AI), with the same Feedback return type and the same MLflow evaluation UI.

Upstream MLflow PRs

This documentation describes work that has already shipped in MLflow:

Verification

  • File follows the integration template described in CONTRIBUTING.md (frontmatter, Use cases, Prerequisites, Install dependencies, Quick start, Resources).
  • All code claims verified against the merged source at mlflow/genai/scorers/google_adk/__init__.py on mlflow/mlflow master.
  • Frontmatter catalog_icon reuses the existing /integrations/assets/mlflow.png asset already used by the other two MLflow pages.
  • Internal links use the absolute path style (/integrations/mlflow-tracing/) matching the convention in the existing pages.
  • No mkdocs.yml changes needed: integration pages are excluded from nav via not_in_nav: /integrations/*.md and discovered through the catalog frontmatter.

CLA

I have signed the Google CLA through my employer Databricks (covers all OSS contributions). Happy to confirm via any process the maintainers prefer.

Adds a new integration page at docs/integrations/mlflow-scorers.md
covering MLflow's five Google ADK scorers (ToolTrajectory, ResponseMatch,
ResponseEvaluation, Safety, Hallucination) for agent evaluation. The
integration wraps ADK's TrajectoryEvaluator, RougeEvaluator,
FinalResponseMatchV2Evaluator, SafetyEvaluatorV1, and HallucinationsV1Evaluator
behind MLflow's scorer interface, so ADK users can evaluate agents inside
mlflow.genai.evaluate() runs without leaving the ADK ecosystem.

Complements the existing MLflow Tracing and MLflow AI Gateway integration
pages by covering evaluation, the third leg of the MLflow stack for ADK.

Signed-off-by: debu-sinha <debusinha2009@gmail.com>
@netlify

netlify Bot commented Jun 10, 2026

Copy link
Copy Markdown

Deploy Preview for adk-docs-preview ready!

Name Link
🔨 Latest commit ecd84ec
🔍 Latest deploy log https://app.netlify.com/projects/adk-docs-preview/deploys/6a30d762a2f1a4000869e7d0
😎 Deploy Preview https://deploy-preview-1829--adk-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@google-cla

google-cla Bot commented Jun 10, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@debu-sinha

Copy link
Copy Markdown
Author

I have signed the Google CLA. Please re-run the CLA check.

Signed-off-by: debu-sinha <debusinha2009@gmail.com>
@debu-sinha

Copy link
Copy Markdown
Author

@PattaraS Built this page to live alongside your tracing and gateway integrations. Followed your template (frontmatter, language-support-tag, Use cases section, cross-links between all three pages) so the three pages read as a set. Flagging in case you want to align on tone or structure before review. Happy to incorporate any feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants