Problem
agentcore traces list finds traces and agentcore traces get exports raw
CloudWatch records, but there is no human-friendly way to compare two agent
invocations.
Users testing runtime versions or named endpoints must manually download JSON,
inspect CloudWatch’s incomplete single-span view, and calculate latency/token
differences themselves. This makes performance regression testing impractical.
Proposed command
agentcore traces compare <baseline-trace-id> <candidate-trace-id> \
--runtime <runtime-name> \
[--since <time>] \
[--until <time>] \
[--json]
The command should fetch both traces directly from the existing CloudWatch
trace/span sources. It must not require users to export JSON files first.
Example output:
Trace comparison: baseline abc123 → candidate def456
Metric Baseline Candidate Delta
End-to-end latency 10.00s 6.33s -3.66s (-36.6%)
LLM latency 7.36s 3.67s -3.69s (-50.1%)
Tool latency 2.26s 2.58s +0.32s (+14.3%)
LLM calls 2 2 0 (0.0%)
Tool calls 1 1 0 (0.0%)
Input tokens 151,266 2,848 -148,418 (-98.1%)
Output tokens 595 300 -295 (-49.6%)
Total tokens 151,861 3,148 -148,713 (-97.9%)
Baseline model(s): us.anthropic.claude-haiku-4-5-20251001-v1:0
Candidate model(s): us.anthropic.claude-haiku-4-5-20251001-v1:0
Required behavior
-
Resolve the runtime/project target using the existing traces command
conventions.
-
Fetch structured span records from CloudWatch directly.
-
Use the POST /invocations server span for end-to-end latency when present.
If absent, fall back to earliest span start through latest span end and label
that fallback clearly.
-
Report LLM and tool time separately using existing GenAI span attributes.
-
Avoid double-counting nested provider spans. For example, a Strands internal
LLM span and its nested Bedrock client span represent the same model call.
-
Include LLM/tool call counts and input/output/total token counts when
available.
-
Show absolute and percentage deltas; handle a zero baseline safely.
-
Provide stable machine-readable output with --json.
-
Fail clearly if either trace cannot be found or has no usable timed spans.
-
Do not claim a “critical path” calculation: CloudWatch trace parent/child
relationships may be incomplete.
-
Show comparability warnings when observable characteristics differ, such as
LLM-call count, tool-call count, or token usage. The CLI cannot prove that
Scope
This is a trace-analysis and latency-comparison feature. It is independent of:
- runtime endpoint selection for agentcore invoke
- AgentCore Gateway A/B-test infrastructure
- SigNoz or other third-party OTEL observability backends
Acceptance Criteria
-
agentcore traces compare works for two trace IDs from the same runtime.
-
It produces the table above or equivalent concise terminal output.
-
--json returns documented structured data suitable for CI benchmarking.
-
Unit tests cover root-span selection, fallback timing, nested-span
de-duplication, token aggregation, deltas, warnings, and missing-trace
errors.
-
Existing traces list and traces get behavior remains unchanged.
Additional Context
No response
Problem
agentcore traces listfinds traces andagentcore traces getexports rawCloudWatch records, but there is no human-friendly way to compare two agent
invocations.
Users testing runtime versions or named endpoints must manually download JSON,
inspect CloudWatch’s incomplete single-span view, and calculate latency/token
differences themselves. This makes performance regression testing impractical.
Proposed command
The command should fetch both traces directly from the existing CloudWatch
trace/span sources. It must not require users to export JSON files first.
Example output:
Required behavior
Resolve the runtime/project target using the existing traces command
conventions.
Fetch structured span records from CloudWatch directly.
Use the POST /invocations server span for end-to-end latency when present.
If absent, fall back to earliest span start through latest span end and label
that fallback clearly.
Report LLM and tool time separately using existing GenAI span attributes.
Avoid double-counting nested provider spans. For example, a Strands internal
LLM span and its nested Bedrock client span represent the same model call.
Include LLM/tool call counts and input/output/total token counts when
available.
Show absolute and percentage deltas; handle a zero baseline safely.
Provide stable machine-readable output with --json.
Fail clearly if either trace cannot be found or has no usable timed spans.
Do not claim a “critical path” calculation: CloudWatch trace parent/child
relationships may be incomplete.
Show comparability warnings when observable characteristics differ, such as
LLM-call count, tool-call count, or token usage. The CLI cannot prove that
Scope
This is a trace-analysis and latency-comparison feature. It is independent of:
Acceptance Criteria
agentcore traces compare works for two trace IDs from the same runtime.
It produces the table above or equivalent concise terminal output.
--json returns documented structured data suitable for CI benchmarking.
Unit tests cover root-span selection, fallback timing, nested-span
de-duplication, token aggregation, deltas, warnings, and missing-trace
errors.
Existing traces list and traces get behavior remains unchanged.
Additional Context
No response