Skip to content

Add AWS/Bedrock user simulator (roadmap: "User Simulator Options") + optional Bedrock agent STT/TTS & judges #182

Description

@nevo-biton

Roadmap alignment

Your roadmap lists User Simulator Options (Framework): "Introduce additional options for the user simulator, including open-source model support for greater flexibility and ease of adoption." This proposal adds a new AWS/Bedrock user-simulator option (an audio-native Amazon Nova Sonic caller), and also lets the Extended Leaderboard goal of "additional speech-to-speech systems" cover Nova Sonic. It then completes the picture so an eval can run entirely on AWS/Bedrock — extending the same AWS support to the agent-under-test (STT/TTS) and the validation judges — for users who don't have OpenAI/Deepgram/ElevenLabs/Gemini keys.

Proposed additions (implemented & live-validated; happy to split into separate PRs)

  1. bedrock_s2s user simulator (the roadmap item) — audio-native Nova Sonic caller over Bedrock bidirectional streaming; the AWS analog of OpenAIRealtimeUserSimulator (reuses BotToBotAudioBridge, _build_prompt(), the end_call tool).
  2. Amazon Transcribe (STT) + Amazon Polly (TTS) as assistant providers in create_stt_service/create_tts_service, wiring pipecat's AWSTranscribeSTTService/AWSPollyTTSService. No new deps (uses websockets + aiobotocore, already present via aioboto3). The agent LLM already runs on Bedrock via the LiteLLM model_list.
  3. Bedrock LLM validation judges — text judges via JUDGE_MODEL; the audio judge (user_speech_fidelity) auto-skips on AWS-only setups since no Bedrock model accepts audio input.

Honest caveats

  • The Nova Sonic caller needs the experimental aws-sdk-bedrock-runtime (pre-1.0, Python ≥3.12). It's isolated as an optional eva[bedrock-s2s] extra behind a lazy factory guard, so EVA's 3.11 floor and base install are unaffected.
  • Speech-fidelity validation has no Bedrock equivalent (no audio-input Bedrock model), so it's skipped on AWS-only runs — not silently broken. (Tangentially related to the Metrics roadmap note that current audio judges can't assess prosody.)

Questions

Given "User Simulator Options" is on the roadmap: would you prefer this as three separate PRs (caller / agent STT+TTS / judges) — with the caller PR landing the roadmap item first — or one combined PR? A combined PR is opened for review; I'm happy to split it however you prefer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions