feat(project): add `project add evaluator llm-as-a-judge` by notgitika · Pull Request #2124 · aws/agentcore-cli

notgitika · 2026-08-27T08:01:55Z

What

Adds agentcore project add evaluator llm-as-a-judge. this attaches a custom LLM-as-a-Judge evaluator to a project. The judge is another LLM prompted with scoring instructions and a rating scale, written into spec.evaluators in agentcore.json. The existing CDK constructs render it as an AWS::BedrockAgentCore::Evaluator, so project deploy provisions it.

Command

agentcore project add evaluator llm-as-a-judge \
  --name <name> \
  --level <SESSION|TRACE|TOOL_CALL> \
  --model <bedrock-id|inference-profile-arn> \
  --instructions <text | file://path | ->  \
  --rating-scale <1-5-quality|1-3-simple|pass-fail|good-neutral-bad> | --rating-scale-file file://scale.json \
  [--description <text>] [--kms-key-arn <arn>] [--tags '{"k":"v"}']

Details

New evaluator subrouter under project add with an llm-as-a-judge subcommand (mirrors the credentials subrouter pattern). Wires a new evaluator resource type through AddResourceInput and FsProjectManager.addResource / toProjectSpecKey.
Rating scales: four presets (1-5-quality, 1-3-simple, pass-fail, good-neutral-bad) that expand into the schema's numerical/categorical shapes with judge-facing definitions, plus --rating-scale-file for a fully custom scale.
Instructions: accept inline text, file://<path>, or - (stdin) via the shared SourceResolver.
Placeholder validation: instructions must embed at least one level-appropriate placeholder (e.g. {context} for SESSION) and use no placeholder outside that level's set. This mirrors AgentCoreEvaluationControlPlaneService's InstructionValidator, so invalid instructions are rejected locally at add/build instead of surfacing as an opaque CloudFormation rollback at deploy time.

Testing

Unit tests for the handler (presets, custom scale file, file:// instructions, description/KMS/tags, duplicate name, invalid spec, and all validation error paths) and for the schema-level placeholder validation.
Verified end-to-end: project add evaluator llm-as-a-judge → project deploy provisions a real AWS::BedrockAgentCore::Evaluator (CREATE_COMPLETE) in a dev account. Test resources were cleaned up afterward.

Scope

Spec-write + deploy of the evaluator resource. Consistent with sibling add commands.

agentcore-devx-automation · 2026-08-27T08:03:52Z

Claude Security Review: no high-confidence findings. (run)

codecov-commenter · 2026-08-27T08:04:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.24%. Comparing base (2595b9d) to head (963be2d).
⚠️ Report is 2 commits behind head on refactor.

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2124      +/-   ##
============================================
+ Coverage     97.22%   97.24%   +0.01%     
============================================
  Files           463      466       +3     
  Lines         28160    28341     +181     
============================================
+ Hits          27378    27559     +181     
  Misses          782      782

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Nice PR — well-structured and consistent with sibling add subcommands. Highlights:

Handler is a thin flag-parsing layer; EvaluatorSchema.safeParse is the single source of truth for validation.
Tests use real temp directories and drive the CLI end-to-end via createRootHandler (no fs mocks) — mocking is limited to TestCoreClient / TestGlobalConfigAccessor at true I/O boundaries. test.each covers all validation branches.
Placeholder validator in src/projectSchemas/evaluator.ts fails fast locally with a clear message instead of surfacing as a CloudFormation rollback at deploy time. structuredClone on presets in resolvePreset guards against downstream mutation of the shared table.
Wiring through AddResourceInput, FsProjectManager.addResource, and toProjectSpecKey is correct and matches the existing pattern.

A couple of very minor observations, not blockers:

LEVEL_ALLOWED_PLACEHOLDERS duplicates a list that lives server-side in InstructionValidator; if the service ever expands the allowed set (e.g. lifts the account-feature gate on skill placeholders for non-TOOL_CALL levels), this table will silently reject valid instructions. A short comment pointing at the service source is already there; consider a follow-up to keep them in sync (or accept unknown placeholders with a warning rather than a hard error).
findInstructionPlaceholders trims whitespace inside { ... }, so { context } passes local validation but is sent to Bedrock verbatim. If the service does exact matching, the local check would be a false positive. Cheap fix if it matters: don't trim, or normalize the stored instructions.

Neither of these needs to be addressed before merging.

agentcore-devx-automation · 2026-08-27T17:24:00Z

Claude Security Review: no high-confidence findings. (run)

jariy17 · 2026-08-27T19:22:37Z

+  },
+} as const satisfies Record<string, RatingScale>;
+
+export type RatingScalePreset = keyof typeof RATING_SCALE_PRESETS;


RATING_SCALE_PRESET file should be left under llm-as-a-judge

jariy17 · 2026-08-27T19:24:33Z

+        "scoring instructions for the judge (inline text, 'file://<path>', or '-' for stdin); must embed at least one level placeholder, e.g. '{context}' for SESSION",
+        z.string().optional(),
+      ),
+      flag(


combine --rating-scale and --rating-scales-files flags together

combined and removed flag

jariy17 · 2026-08-27T19:25:40Z

            llmAsAJudge: {
              model: "anthropic.claude-v2",
-              instructions: "Judge the answer",
+              instructions: "Judge the answer given {context}",


no need here

jariy17 · 2026-08-27T19:26:36Z

+  try {
+    return JSON.parse(raw);
+  } catch (error) {
+    throw new InputValidationError("'--rating-scale-file' must contain valid JSON", {


Doesn't SourceResolver handle this for us.

jariy17 · 2026-08-27T19:28:39Z

+          },
+        },
+        kmsKeyArn: flags["kms-key-arn"],
+        tags: parseJsonFlag<Record<string, string>>("tags", flags["tags"]),


there is special helper for this i think...

uses parseJsonFlagWithSchema(..., TagsSchema) now

jariy17 · 2026-08-27T19:31:28Z

  ratingScale: RatingScaleSchema,
 });
 export type LlmAsAJudgeConfig = z.infer<typeof LlmAsAJudgeConfigSchema>;
+// The CreateEvaluator API templates the judge prompt: instructions must embed at


nit: no code comments.

jariy17 · 2026-08-27T19:33:23Z

 export type LlmAsAJudgeConfig = z.infer<typeof LlmAsAJudgeConfigSchema>;
+// The CreateEvaluator API templates the judge prompt: instructions must embed at
+// least one level-specific placeholder (e.g. "{context}") and use no placeholder
+// outside that level's allowed set. This mirrors AgentCoreEvaluationControlPlaneService's


no validation on placeholder emu,s because if evals introduces new ones, customers will get confused why the cli doesn't accepted.

jariy17

apply fixes we discussed offline and in comments and ill approve

Adds a CLI command to attach a custom LLM-as-a-Judge evaluator to a project. The judge is another LLM prompted with scoring instructions and a rating scale, written into spec.evaluators (deployed as an AWS::BedrockAgentCore::Evaluator by the existing CDK constructs). - New `evaluator` subrouter under `project add` with an `llm-as-a-judge` subcommand. - Flags: --name, --level (SESSION|TRACE|TOOL_CALL), --model (Bedrock id/ARN), --instructions (inline/file:///stdin), --rating-scale, --description, --kms-key-arn, --tags. - --rating-scale accepts either a named preset (1-5-quality, 1-3-simple, pass-fail, good-neutral-bad) or an inline JSON rating scale; presets live beside the subcommand and expand into the schema's numerical/categorical shapes. - --tags is parsed via parseJsonFlagWithSchema against TagsSchema. - Instruction placeholder validation is left to the CreateEvaluator service so the CLI never rejects placeholders the service later adds. - Wire the new `evaluator` resource type through AddResourceInput and FsProjectManager.addResource / toProjectSpecKey. Verified end-to-end: preset and inline-JSON rating scales both deploy a real evaluator (CREATE_COMPLETE) and write deployed-state.json.

agentcore-devx-automation · 2026-08-27T20:06:09Z

Claude Security Review: no high-confidence findings. (run)

nborges-aws · 2026-08-27T20:50:46Z

+          "required option '--rating-scale <rating-scale>' not specified",
+        );
+
+      if (!isValidBedrockModelId(flags["model"]))


Looking at this: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_FoundationModelSummary.html
it looks like foundation models don't have account Id in the ARN, but this validation method expects it.

If this is right, the fix would be simple, just add : before foundation-model in the regex

real bug let me fix it

nborges-aws · 2026-08-27T20:50:49Z

+  if (isRatingScalePreset(value)) {
+    return structuredClone(RATING_SCALE_PRESETS[value]) as RatingScale;
+  }
+  if (!value.trim().startsWith("{")) {


I see we dropped the --rating-scales-files. Do we still intend to support --rating-scale file://...? If so, this resolution method needs to change. File will always be rejected by the starts with { check.

discussed offline we are not supporting it anymore

Bedrock foundation-model ARNs omit the account id (arn:aws:bedrock:<region>::foundation-model/<id>) while inference-profile ARNs include it. Make the account segment optional in the evaluator model-id ARN validator so a valid foundation-model ARN is not rejected.

agentcore-devx-automation · 2026-08-27T21:01:26Z

Claude Security Review: no high-confidence findings. (run)

nborges-aws

LGTM thanks for fix

…e's account format (#2134) The prior validator made the account segment optional for both resource types, which also accepted impossible combinations (account-scoped foundation-model, accountless inference-profile). Pin each type to its documented shape: foundation-model ARNs omit the account, while (application-)inference-profile ARNs carry it. Also accept application-inference-profile ARNs, which the prior pattern rejected. Follow-up to #2124 (nborges review). Co-authored-by: gitikavj <gitikavj@amazon.com>

github-actions Bot added the size/l PR size: L label Aug 27, 2026

agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Aug 27, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 27, 2026

agentcore-devx-automation Bot reviewed Aug 27, 2026

View reviewed changes

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 27, 2026

notgitika force-pushed the feat/project-add-evaluator branch from e14f409 to defe35b Compare August 27, 2026 17:20

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 27, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

notgitika marked this pull request as ready for review August 27, 2026 19:21

jariy17 reviewed Aug 27, 2026

View reviewed changes

notgitika force-pushed the feat/project-add-evaluator branch from defe35b to 9bd43e3 Compare August 27, 2026 20:04

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 27, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

jariy17 previously approved these changes Aug 27, 2026

View reviewed changes

nborges-aws reviewed Aug 27, 2026

View reviewed changes

notgitika dismissed jariy17’s stale review via 963be2d August 27, 2026 20:59

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 27, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 27, 2026

nborges-aws approved these changes Aug 27, 2026

View reviewed changes

notgitika merged commit 7bf13af into aws:refactor Aug 27, 2026
19 of 21 checks passed

notgitika deleted the feat/project-add-evaluator branch August 27, 2026 21:05

notgitika mentioned this pull request Aug 27, 2026

fix(project): tie evaluator model ARN validation to each resource type's account format #2134

Merged

Uh oh!

Conversation

notgitika commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Command

Details

Testing

Scope

Uh oh!

agentcore-devx-automation Bot commented Aug 27, 2026

Uh oh!

codecov-commenter commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

agentcore-devx-automation Bot left a comment

Choose a reason for hiding this comment

AgentCore Harness Review

Uh oh!

agentcore-devx-automation Bot commented Aug 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 left a comment

Choose a reason for hiding this comment

Uh oh!

agentcore-devx-automation Bot commented Aug 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agentcore-devx-automation Bot commented Aug 27, 2026

Uh oh!

nborges-aws left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

notgitika commented Aug 27, 2026 •

edited

Loading

codecov-commenter commented Aug 27, 2026 •

edited

Loading

jariy17 Aug 27, 2026 •

edited

Loading