You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support batch insights in BatchEvaluationRunner #678
BatchEvaluationRunner does not expose the batch insights workflow.
Requested behavior
Support requesting batch insights through BatchEvaluationRunner, with job polling and access to the resulting insights. The original report identifies these StartBatchEvaluation insights:
Builtin.Insight.FailureAnalysis
Builtin.Insight.UserIntent
Builtin.Insight.ExecutionSummary
The workflow should enforce the API constraint that insights and evaluators cannot be requested in the same job.
Why this matters
The reporter uses insights to cluster failures and identify root causes that can be addressed through prompt changes. Without SDK support, the evaluation improvement workflow requires custom orchestration outside BatchEvaluationRunner.
Open design question
Confirm whether the runner's dataset-based workflow is compatible with insights and which inputs are supported. Use that finding to determine how to expose insights through the runner, or whether a separate insights runner is needed. This compatibility question remains unresolved.
Original report environment: bedrock-agentcore 1.23.0, Python 3.13, us-west-2.
Activity