feat(eval): add eval online-eval CLI commands - #1841
Closed
jariy17 wants to merge 2 commits into
Closed
Conversation
github-actions
Bot
added
agentcore-harness-reviewing
jariy17
force-pushed
the
feat/eval-evaluator-cli
branch
from
July 28, 2026 20:31
0a2afdb to
122b13f
Compare
Adds `agentcore eval online-eval create/get/list/update/pause/resume/delete`, following the conventions established in the eval evaluator review (#1822). - `--agent <id>` accepts a plain AgentCore Runtime ID or a Harness ID and derives the CloudWatch data source from it. A harness is itself backed by a runtime, so resolution tries GetAgentRuntime and falls back to GetHarness, reading the underlying runtime out of the harness environment. The log group is keyed by runtime *id*, the service name by runtime *name* — verified against live resources, the two are not interchangeable. - `--log-group-name`/`--service-name` are the escape hatch for agents emitting traces under a custom OTel service name rather than AgentCore's default path. - No execution role required at create time: Core auto-provisions and reuses a default one, mirroring the harness execution-role pattern. The policy grants Logs Insights query access over `aws/spans` plus the runtime's log-group prefix, since the service validates query access at the runtime level and rejects a role scoped to a single endpoint. - `update` merges over the current config because UpdateOnlineEvaluationConfig replaces the whole `rule`, so unset fields are preserved. - `list` exposes only the API's server-side pagination; no client-side filters.
jariy17
force-pushed
the
feat/eval-online-eval-cli
branch
from
July 30, 2026 13:58
679b473 to
13ee2dd
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## refactor #1841 +/- ##
============================================
- Coverage 95.61% 95.46% -0.15%
============================================
Files 181 190 +9
Lines 8842 9439 +597
============================================
+ Hits 8454 9011 +557
- Misses 388 428 +40 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Core sub-clients are scoped to a CLI namespace, not an AWS resource type: HarnessClient spans harness, versions, endpoints, invoke, and exec. Online evaluation configs live under `agentcore eval` alongside evaluators, so they belong on EvalClient / CoreEvalClient rather than a sibling core.onlineEval. - Merged OnlineEvalClient's methods into EvalClient; deleted core/onlineEval.tsx. - Moved CreateOnlineEvalInput / UpdateOnlineEvalInput into eval/types.tsx and the method signatures into CoreEvalClient; deleted online-eval/types.tsx. - Dropped `onlineEval` from Core and CoreClient; handlers call core.eval.*. - Folded TestOnlineEvalClient into TestEvalClient behind a shared record() helper. Pure restructuring: no request shape changed, so every recorded fixture and golden file still matches (513 tests green, unchanged).
Contributor
Author
|
Superseded by a rebase onto the latest |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agentcore eval online-eval create/get/list/update/pause/resume/delete.--agent <id>accepts a plain AgentCore Runtime ID or a Harness ID and derives the CloudWatch log group + service name from it (a harness is itself backed by a runtime; resolution falls back from GetAgentRuntime to GetHarness). Verified against real resources in a live account.--log-group-name/--service-nameare the escape hatch for agents that emit traces under a custom OTel service name rather than AgentCore's default runtime path.updatemerges over the existing config (the API replaces the wholerule/dataSourceConfig), so unset fields are preserved.Test plan
bun x tsc --noEmitcleanbun test— 391/391 passget/listagainst real production online-eval configs in account 725476964917