feat(eval): add eval evaluator CLI commands (CRUD + LLaJ/code-based) by jariy17 · Pull Request #1822 · aws/agentcore-cli

jariy17 · 2026-07-23T22:15:44Z

What

Adds the imperative agentcore eval evaluator command surface (CLI only). Covers LLM-as-a-Judge and code-based evaluator create/update plus type-agnostic get/list/delete, following the DevX Evaluations/Optimization refactor doc and the existing identity handler conventions.

Command structure

agentcore eval                              # evaluate and optimize AgentCore agents
└── evaluator                               # manage AgentCore evaluators
    ├── llm-as-a-judge                       # LLM-as-a-Judge evaluators
    │   ├── create
    │   └── update
    ├── code-based                           # code-based (Lambda-backed) evaluators
    │   ├── create
    │   └── update
    ├── get                                  # get an evaluator by id (type-agnostic)
    ├── list                                 # list evaluators (client-side --type filter)
    └── delete                               # delete an evaluator by id (requires --yes)

Flags

`eval evaluator llm-as-a-judge create`

Flag	Required	Notes
`--name`	✓	evaluator name
`--level`	✓	`SESSION \| TRACE \| TOOL_CALL`
`--model`	✓	Bedrock model id used to judge
`--instructions`	✓	source-aware: inline, `file://<path>`, or `-` (stdin)
`--rating-scale`	✓*	preset: `1-5-quality \| 1-3-simple \| pass-fail \| good-neutral-bad`
`--rating-scale-json`	✓*	raw `RatingScale` JSON (source-aware). Mutually exclusive with `--rating-scale`; exactly one required
`--kms-key-arn`		customer-managed KMS key ARN
`--tags`		JSON object (source-aware)
`--client-token`		idempotency token

`eval evaluator llm-as-a-judge update`

Flag	Required	Notes
`--id`	✓	evaluator id
`--instructions`		source-aware
`--model`
`--rating-scale` / `--rating-scale-json`		mutually exclusive
`--kms-key-arn`
`--client-token`

Fields left unset are preserved: UpdateEvaluator replaces the whole evaluatorConfig union, so the client does get-then-merge over the current definition.

`eval evaluator code-based create`

Flag	Required	Notes
`--name`	✓	evaluator name
`--level`	✓	`SESSION \| TRACE \| TOOL_CALL`
`--lambda-arn`	✓	Lambda that scores a session
`--timeout`		seconds (1–300). No CLI default; the service applies its own (60s)
`--kms-key-arn`
`--tags`		JSON object (source-aware)
`--client-token`

`eval evaluator code-based update`

Flag	Required	Notes
`--id`	✓	evaluator id
`--lambda-arn`
`--timeout`
`--kms-key-arn`
`--client-token`

`eval evaluator get`

Flag	Required	Notes
`--id`	✓	evaluator id

`eval evaluator list`

Flag	Required	Notes
`--next-token`		pagination token (server-side)
`--max-results`		max items (server-side)
`--type`		filters the returned page client-side: `Builtin \| code-based \| llm-as-a-judge`. The ListEvaluators API paginates only, so the filter is page-local

`eval evaluator delete`

Flag	Required	Notes
`--id`	✓	evaluator id
`--yes`	✓	confirm deletion (required in this headless/JSON-only path)

Notes / decisions

Source-aware field values (--instructions, --rating-scale-json, --tags): inline, file://<path>, or - for stdin, following the AWS CLI file:// convention. One flag reads stdin per command.
Rating scale offers presets (the common defaults) and a raw-JSON escape hatch (what the API supports directly).
--timeout has no CLI default; the service default applies when omitted.
Dependency inversion: CoreEvalClient is declared next to the handlers (src/handlers/eval/types.tsx) and implemented in src/core/eval.tsx.

Testing

bun test — full suite green (356 tests).
bun run typecheck, bun run lint:check, bun run format:check — clean.
bun run build + bundle smoke (node dist/index.js eval evaluator --help) — works.
New tests: handler behavior + validation/source/list-filter via TestCoreClient, EvalClient get-then-merge unit tests, and source-resolver tests.

TUI flows for these commands will come in a later PR.

codecov-commenter · 2026-07-23T22:16:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.21%. Comparing base (ccd4024) to head (c0338a9).

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #1822      +/-   ##
============================================
+ Coverage     94.92%   95.21%   +0.29%     
============================================
  Files           155      170      +15     
  Lines          7565     8026     +461     
============================================
+ Hits           7181     7642     +461     
  Misses          384      384

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

AlexanderRichey

Really good work here. Left a few comments. Main blocking things are using types vs interfaces appropriately and handling duplication of source reading.

AlexanderRichey · 2026-07-24T16:07:04Z

+        throw new TypeError("required option '--lambda-arn <lambda-arn>' not specified");
+      }
+
+      const source = new SourceResolver(io);


Both @aidandaly24 and @nborges-aws wrote something that does the same thing in their PRs. Let's sync to align on a plan to avoid duplication here. Someone will have to merge first and the others will have to rebase their PRs against the changes, but it should be pretty easy.

I'll wait till @nborges-aws's PR to get merged

AlexanderRichey · 2026-07-24T16:08:53Z

+    },
+  });
+
+export const createCodeBasedUpdateHandler = (core: Core) =>


I think we should pull this into its own file/dir. The pattern is documented here: https://github.com/aws/agentcore-cli/tree/refactor#adding-a-new-handler

I'll update the PR to follow this convention

AlexanderRichey · 2026-07-24T16:10:14Z

+      // This branch is headless/JSON only (no TUI prompt yet), so confirmation is
+      // required up front. `-y` shorthand is not wired: the router flag layer only
+      // supports long option names today.
+      flag("yes", "confirm deletion (required in non-interactive mode)", z.boolean().optional()),


We don't have this convention for the other CRUDL commands. Let's leave this out to be consistent.

Removed it now

AlexanderRichey · 2026-07-24T16:15:51Z

+    },
+  });
+
+export const createLlmAsAJudgeUpdateHandler = (core: Core, io: AppIO) =>


Same here. I think this should follow the same convention: https://github.com/aws/agentcore-cli/tree/refactor#adding-a-new-handler

I'll follow this convention

AlexanderRichey · 2026-07-24T16:16:35Z

+// the AgentCore UpdateEvaluator API replaces the whole evaluatorConfig union, and
+// the llmAsAJudge arm requires instructions + ratingScale + modelConfig together,
+// so a partial update is only possible by merging over the current definition.
+export interface LlmAsAJudgeUpdate {


These should be types and not interfaces. Interfaces are abstract and need to be implemented by something else, whereas types are more concrete and can hold data themselves.

Changed LlmAsAJudgeUpdate and CodeBasedUpdate to type aliases (they just carry data). Left CoreEvalClient as an interface since EvalClient implements it.

aidandaly24

A few things:

aidandaly24 · 2026-07-28T15:47:59Z

+    const existing =
+      current.evaluatorConfig && "llmAsAJudge" in current.evaluatorConfig
+        ? current.evaluatorConfig.llmAsAJudge
+        : undefined;


If its undefined should we throw? Since there should definitely be an existing, right?

We could reject immediately when the current config is not llmAsAJudge. Supplying all LLM fields currently converts a code-based evaluator. The code-based path has the same issue with --lambda-arnso I would change both.

Yes, this makes sense, I'll throw a InputValidationError

aidandaly24 · 2026-07-28T16:06:07Z

+      llmAsAJudge: {
+        instructions,
+        ratingScale,
+        modelConfig: { bedrockEvaluatorModelConfig: { modelId } },


This rebuilds the model config with only modelId, could this not silently remove existing inferenceConfig and additionalModelRequestFields. Could we preserve the existing bedrockEvaluatorModelConfig, override only modelId, and maybe add a regression test if you agree?

Nice catch!, i'll take override the existing bedrockEvaluatorModelConfig rather than creating a new object.

aidandaly24 · 2026-07-28T16:07:50Z

+│       │   └── update             # update (merged over the existing config)
+│       ├── get                    # get an evaluator by id (type-agnostic)
+│       ├── list                   # list evaluators (client-side --type filter)
+│       └── delete                 # delete an evaluator by id (requires --yes)


Wasn't --yes removed from the delete handler?

Yes, I forgot about this, I remove this

aidandaly24 · 2026-07-28T16:28:24Z

+// run drives the real router (parsing → middleware → handler) against a
+// TestCoreClient so we can assert on the exact request a handler built, plus the
+// captured stdout. Optional `stdin` seeds the in-memory input stream for `-`.
+function run(core: TestCoreClient, args: string[], stdin?: string) {


Is there a reason we're using TestCoreClient for these handler tests? For ordinary headless CRUD flows, Harness, Runtime, and Identity use the real CoreClient with fixture-backed SDK clients and golden output. TestCoreClient is reserved for interactive or streaming behavior. Following that pattern here would exercise Eval's SDK command construction and avoid adding a separate test seam.

And for golden output generate them using the account 685197708687 for parity w what we have for Identity and Runtime.

Yes, I'll push a change to follow this fixture pattern

Hweinstock · 2026-07-28T16:33:08Z

@@ -0,0 +1,102 @@
+import { test, expect } from "bun:test";


are we able to test this functionality through src/handlers/eval/evaluator/evaluator.test.tsx in a more e2e style? Looking at the existing integrations, it looks like we don't test core implementations directly, but instead test them through the handlers.

If the rest of handlers are doing this, I'm fine with it.

Hweinstock · 2026-07-28T18:28:26Z

+      flag("client-token", "idempotency token", z.string().optional()),
+    ],
+    handle: async (ctx, flags) => {
+      if (!flags["name"]) throw new TypeError("required option '--name <name>' not specified");


should we swap these to the modeled input validation errors? This change might have been done before that was merged yesterday, but it should safe us some effort in the future.

Yes, I'll throw an InputValidationError

Hweinstock · 2026-07-28T18:29:27Z

+        : undefined);
+
+    if (!instructions || !ratingScale || !modelId) {
+      throw new TypeError(


should we throw a modeled exception here?

yes, Ill used the modeled exceptions introduced by ur pr.

Hweinstock · 2026-07-28T18:33:53Z

+      flag("max-results", "maximum number of items to return", z.number().optional()),
+      flag(
+        "type",
+        `filter the returned page by type (${TYPE_FILTERS.join(" | ")})`,


could this client side filtering result in some behavior where we get empty responses (but there is data on another page)? I'm wondering if it would be simpler to only support api-level filters.

Yes, it would. I wanted agentcore cli to be more than just aws cli wrapper so adding client side filters make sense to me. I'll just remove it for now until it's a customer ask.

Hweinstock · 2026-07-28T18:38:46Z

@@ -0,0 +1,39 @@
+import z from "zod";


i believe the strands team avoids generic 'utils' or 'common' files/modules because they end up becoming a dumping ground of functionality that makes it difficult to find things (which I've definitely experience on past projects). What do you think of following the same pattern here and naming this more specifically for the functionality it provides (like sharedFlags)?

I agree with this sentiment.

I like this pattern

Add the imperative `agentcore eval evaluator` command surface: llm-as-a-judge create/update, code-based create/update, and type-agnostic get/list/delete. CLI-only for now; the TUI follows later. - New CoreEvalClient (consumer-owned interface) + EvalClient impl over the Bedrock AgentCore control plane. Update paths do get-then-merge since UpdateEvaluator replaces the whole evaluatorConfig union. - Rating scale accepts a preset (--rating-scale) or raw JSON (--rating-scale-json). - Source-aware field values: inline, file://<path>, or - for stdin. - --timeout has no CLI default; the service applies its own. - list --type filters the returned page client-side (the API paginates only): Builtin | code-based | llm-as-a-judge. - delete requires --yes (headless-safe confirmation). Tests: handler behavior via TestCoreClient, EvalClient get-then-merge unit tests, and source resolver tests.

A single --rating-scale flag now takes either a preset id or a source-aware custom RatingScale (JSON inline, file://<path>, or -). A value matching a known preset id expands to that preset; anything else is parsed as a RatingScale JSON value.

…rop --yes - Split llm-as-a-judge and code-based create/update into their own directories per the documented handler convention; shared flags/helpers live in a sibling utils.tsx. - LlmAsAJudgeUpdate / CodeBasedUpdate are now type aliases (data-carrying), leaving CoreEvalClient as the implemented interface. - Drop --yes from evaluator delete to stay consistent with the other CRUDL commands.

- Reject a type mismatch in both update paths before merging: UpdateEvaluator replaces the whole evaluatorConfig union, so merging into the wrong arm silently converted an evaluator to the other type. - Preserve the existing bedrockEvaluatorModelConfig (inferenceConfig, additionalModelRequestFields) and override only modelId; same for lambdaConfig. - Throw the modeled InputValidationError instead of TypeError in the handlers and core, per the errors module. - Drop the client-side `--type` filter on list: ListEvaluators has no server-side type filter, so filtering a page could return empty results while later pages held matches. - Move the handler tests to the real CoreClient with fixture-backed SDK clients and golden output, recorded against the shared test account, matching the Harness/Runtime/Identity pattern. Removes src/core/eval.test.ts, which tested the core implementation directly. - Rename llm-as-a-judge/utils.tsx to sharedFlags.tsx and hoist the duplicated LEVELS into evaluator/levels.tsx. - Fix the stale README entries for `--yes` and the `--type` filter.

The update path used to rebuild bedrockEvaluatorModelConfig from modelId alone, dropping inferenceConfig and additionalModelRequestFields. The existing tests only proved instructions and ratingScale survived, because no CLI flag can set inferenceConfig: `--model` carries a model id. Seed an evaluator carrying an inferenceConfig through the SDK during recording, then assert an instructions-only update leaves the tuning intact. Fixtures are keyed by request input, so the recorded UpdateEvaluator fixture also pins the exact request: reintroducing the bug changes the hash and fails the test.

aidandaly24

A few more comments

aidandaly24 · 2026-07-28T20:55:33Z

+      flag("level", `evaluation level (${LEVELS.join(" | ")})`, z.enum(LEVELS).optional()),
+      flag("lambda-arn", "ARN of the Lambda function that scores a session", z.string().optional()),
+      // No default; the service applies its own timeout (60s) when omitted.
+      flag("timeout", "Lambda timeout in seconds (1-300)", z.number().optional()),


It seems --timeout is not enforcing the documented 1-300 range. I tested 0, 301, and 1.5, and all three reached the Core client. This should use z.number().int().min(1).max(300) for both create and update, with boundary tests.

Yeah, that makes sense. My instinct was to let the service throw the ValidationException, but harness/exec establishes the client-side pattern (z.number().min(1).max(3600)), so I'll follow it — z.number().int().min(1).max(300) on both create and update. However, I won't add boundary tests — that would be testing the shared flag layer's zod validation, which is already covered there

aidandaly24 · 2026-07-28T20:56:23Z

+    matchGolden(FIXTURES, "code-based-delete.golden.json", stdout);
+  });
+
+  test("propagates ResourceNotFoundException from get", async () => {


This is not actually testing ResourceNotFoundException. The recorded fixture contains a ValidationException because missing-evaluator-000 is not a valid evaluator ID. We should use a valid nonexistent ID and assert the error name, as Runtime and Identity do.

aidandaly24 · 2026-07-28T20:58:17Z

+  --name order-support-quality \
+  --level SESSION \
+  --model us.anthropic.claude-sonnet-4-5 \
+  --instructions "Judge whether the order-support agent answered correctly." \


nit: doesn't this require an allowed placeholder like {context} like

--instructions "Judge from {context} whether the agent answered correctly." \

- `--timeout` documented a 1-300 range but validated nothing: 0, 301, and 1.5 all reached the Core client. Declare z.number().int().min(1).max(300) on create and update, matching harness/exec. - The not-found test used an id that fails the service's evaluator-id pattern, so it recorded a ValidationException while claiming to cover ResourceNotFoundException, and a bare rejects.toThrow() let the mismatch pass. Use a well-formed absent id and assert the error name, as Identity does. - Fix the README llm-as-a-judge example: SESSION instructions require an allowed placeholder, and the model id was truncated. Fixtures re-recorded, so the evaluator ids in them change.

Hweinstock · 2026-07-28T23:06:17Z

one thing I noticed testing e2e is that the help pages no longer tell us whats required since we aren't using commander's required option:

Usage: agentcore eval evaluator code-based create [options]

create a code-based (Lambda-backed) evaluator

Options:
  --name <name>                  the name of the evaluator
  --level <level>                evaluation level (SESSION | TRACE | TOOL_CALL)
  --lambda-arn <lambda-arn>      ARN of the Lambda function that scores a session
  --timeout <timeout>            Lambda timeout in seconds (1-300)
  --kms-key-arn <kms-key-arn>    customer managed KMS key ARN for evaluator data
  --tags <tags>                  tags to apply (JSON object of key/value strings; inline, file://<path>, or - for stdin)
  --client-token <client-token>  idempotency token
  -h, --help                     display help for command

I think this is an issue with a lot of the work we've done though, not specific to here.

the apis are definitely a little awkward, but the commands themselves work great!

Hweinstock · 2026-07-28T22:56:12Z

+    // a code-based evaluator into an LLM-as-a-Judge one.
+    if (!current.evaluatorConfig || !("llmAsAJudge" in current.evaluatorConfig)) {
+      throw new InputValidationError(`Evaluator "${id}" is not an LLM-as-a-Judge evaluator`, {
+        meta: { evaluatorId: id },


oooh, i like the use of meta!

Hweinstock · 2026-07-28T22:59:31Z

@@ -0,0 +1,61 @@
+import { describe, expect, test } from "bun:test";


is it possible to get coverage for this through the handler tests? The advantage being that we can change the implementation of how we resolve the rating scale as long as it doesn't impact e2e behavior.

github-actions Bot added agentcore-harness-reviewing AgentCore Harness review in progress and removed agentcore-harness-reviewing AgentCore Harness review in progress labels Jul 23, 2026

AlexanderRichey requested changes Jul 24, 2026

View reviewed changes

jariy17 force-pushed the feat/eval-evaluator-cli branch from 6db704d to 0a2afdb Compare July 27, 2026 15:05

aidandaly24 requested changes Jul 28, 2026

View reviewed changes

Hweinstock reviewed Jul 28, 2026

View reviewed changes

jariy17 added 8 commits July 28, 2026 19:26

fix(eval): lowercase 'builtin' in list --type filter

b27c4e2

refactor(eval): use shared IO utilities

2e73cd0

docs(eval): drop stale --type and --yes from the README examples

7696310

jariy17 force-pushed the feat/eval-evaluator-cli branch from 0a2afdb to 122b13f Compare July 28, 2026 20:31

aidandaly24 reviewed Jul 28, 2026

View reviewed changes

Hweinstock approved these changes Jul 28, 2026

View reviewed changes

		@@ -0,0 +1,61 @@
		import { describe, expect, test } from "bun:test";

Uh oh!

Conversation

jariy17 commented Jul 23, 2026

What

Command structure

Flags

eval evaluator llm-as-a-judge create

eval evaluator llm-as-a-judge update

eval evaluator code-based create

eval evaluator code-based update

eval evaluator get

eval evaluator list

eval evaluator delete

Notes / decisions

Testing

Uh oh!

codecov-commenter commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

AlexanderRichey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aidandaly24 left a comment

Choose a reason for hiding this comment

Uh oh!

aidandaly24 Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

`eval evaluator llm-as-a-judge create`

`eval evaluator llm-as-a-judge update`

`eval evaluator code-based create`

`eval evaluator code-based update`

`eval evaluator get`

`eval evaluator list`

`eval evaluator delete`

codecov-commenter commented Jul 23, 2026 •

edited

Loading

jariy17 Jul 24, 2026 •

edited

Loading

jariy17 Jul 24, 2026 •

edited

Loading

aidandaly24 Jul 28, 2026 •

edited

Loading

Hweinstock commented Jul 28, 2026 •

edited

Loading