Skip to content

feat(traces): compare two AgentCore traces from the CLI #1830

Description

@OrenAshkenazy

Problem

agentcore traces list finds traces and agentcore traces get exports raw
CloudWatch records, but there is no human-friendly way to compare two agent
invocations.

Users testing runtime versions or named endpoints must manually download JSON,
inspect CloudWatch’s incomplete single-span view, and calculate latency/token
differences themselves. This makes performance regression testing impractical.

Proposed command

agentcore traces compare <baseline-trace-id> <candidate-trace-id> \
  --runtime <runtime-name> \
  [--since <time>] \
  [--until <time>] \
  [--json]

The command should fetch both traces directly from the existing CloudWatch
trace/span sources. It must not require users to export JSON files first.

Example output:

Trace comparison: baseline abc123 → candidate def456

Metric                      Baseline     Candidate                   Delta
End-to-end latency            10.00s         6.33s         -3.66s (-36.6%)
LLM latency                    7.36s         3.67s         -3.69s (-50.1%)
Tool latency                   2.26s         2.58s         +0.32s (+14.3%)
LLM calls                          2             2                0 (0.0%)
Tool calls                         1             1                0 (0.0%)
Input tokens                 151,266         2,848       -148,418 (-98.1%)
Output tokens                    595           300           -295 (-49.6%)
Total tokens                 151,861         3,148       -148,713 (-97.9%)

Baseline model(s): us.anthropic.claude-haiku-4-5-20251001-v1:0
Candidate model(s): us.anthropic.claude-haiku-4-5-20251001-v1:0

Required behavior

  • Resolve the runtime/project target using the existing traces command
    conventions.

  • Fetch structured span records from CloudWatch directly.

  • Use the POST /invocations server span for end-to-end latency when present.
    If absent, fall back to earliest span start through latest span end and label
    that fallback clearly.

  • Report LLM and tool time separately using existing GenAI span attributes.

  • Avoid double-counting nested provider spans. For example, a Strands internal
    LLM span and its nested Bedrock client span represent the same model call.

  • Include LLM/tool call counts and input/output/total token counts when
    available.

  • Show absolute and percentage deltas; handle a zero baseline safely.

  • Provide stable machine-readable output with --json.

  • Fail clearly if either trace cannot be found or has no usable timed spans.

  • Do not claim a “critical path” calculation: CloudWatch trace parent/child
    relationships may be incomplete.

  • Show comparability warnings when observable characteristics differ, such as
    LLM-call count, tool-call count, or token usage. The CLI cannot prove that

Scope

This is a trace-analysis and latency-comparison feature. It is independent of:

  • runtime endpoint selection for agentcore invoke
  • AgentCore Gateway A/B-test infrastructure
  • SigNoz or other third-party OTEL observability backends

Acceptance Criteria

  • agentcore traces compare works for two trace IDs from the same runtime.

  • It produces the table above or equivalent concise terminal output.

  • --json returns documented structured data suitable for CI benchmarking.

  • Unit tests cover root-span selection, fallback timing, nested-span
    de-duplication, token aggregation, deltas, warnings, and missing-trace
    errors.

  • Existing traces list and traces get behavior remains unchanged.

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions