Vad latency interrupt - #2661
Open
25marcusb wants to merge 28 commits into
Open
Conversation
…rape jobs Adds a Grafana dashboard for quality metrics, wires up historical/live Prometheus scrape targets, fixes the Grafana dashboards mount path, and enables the tracer in the conversation config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Saving edits in the Grafana UI forks a provisioned dashboard into Grafana's DB and permanently disconnects it from the source file, so future changes to the JSON silently stop applying until the DB copy is manually cleared. Disabling UI updates keeps the file the single source of truth. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gives an observer a way to see the robot's conversational quality trend over time, not just a live snapshot. Each panel uses increase() over 15-minute buckets on the existing live-run gauges, split into three queries (incoherent/negative, marginal, coherent/positive) so the stack order is guaranteed bottom-to-top for easy comparison. Also widens the default time window to 6h so multiple buckets are visible by default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…pshot Now that the scorer's counters are genuine all-time cumulative totals (see dataAnalysis#restart-safe-live-metrics), the pie charts need their own PromQL window to stay "recent" rather than showing everything ever scored. Switches from a raw instant value to increase(metric[1h]), matching the windowing already used by the 15-minute histogram panels. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a live Zenoh topic ("om/tracer/event") that Tracer.Gauge publishes
each turn to, using the exact same JSON bytes written to
traces/<date>.jsonl. This lets external consumers (e.g. dataAnalysis's
live_quality_scorer.py) subscribe directly to the runtime instead of
tailing the trace file over a shared filesystem.
Follows AvatarProvider's existing lazy-open pattern (avatar.go): the
Zenoh session/publisher is opened on first use and, if unavailable,
publishing is silently disabled -- a missing Zenoh router degrades to
file-only tracing rather than breaking the runtime. Gated on the
existing UseTracer flag; no new config surface. Not added to
cloudsession's CloudTopics, so this stays local-only for now.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gauge() was lazily opening the Zenoh session/publisher on first use while holding Tracer.mu, on the synchronous cortex-tick path (before executeActions). SetGeneration(), which runCortexLoop calls on every mode transition, takes the same mutex, and mode transitions are handled strictly sequentially -- so a slow/unreachable Zenoh router stalled not just trace publishing but every future mode transition (reported as the robot going silent after its first conversation turn). Move the one-time session open into Enable() (called once per mode at init time) and run it in a goroutine so it can never block initializeMode/onModeTransition even if it's the first tracer-enabled mode entered via a transition rather than at startup. Gauge() now just checks the already-resolved publisher under the lock and skips publishing if it isn't ready yet.
The previous fix moved the one-time Zenoh session open off the Gauge()/ tick path, but Gauge() still called publisher.Put() while holding Tracer.mu on every trace write. SetGeneration() takes the same mutex and runs on every mode transition, and transitions are processed strictly sequentially -- so any later Put() that stalls (degraded session, no subscriber draining, backpressure) reproduces the exact same freeze, just on a subsequent conversation instead of the first one (e.g. the robot greets one person fine, returns to searching, but never recovers after that). Snapshot the publisher pointer and release the lock before calling Put(), matching the same reasoning already applied to the session open.
Turns on the tracer (and, with it, the Zenoh trace publish + the mode-transition lock fixes) so the fix committed in 254b6191/d7898f00 is actually exercised when running this config locally. Not intended for shipping as-is -- revert before merging if this shouldn't be enabled by default for this config.
The only consumer of the "om/tracer/event" Zenoh topic was dataAnalysis's live_quality_scorer.py. That pipeline is moving in-process (next commit), so there's no remaining reason to serialize trace records onto a pub/sub bus at all -- an in-process channel is strictly more direct. Tracer.Subscribe() returns a buffered channel appended to a subscriber list; Gauge() sends to each subscriber non-blocking (select/default) after releasing t.mu, reusing the exact same lock-release seam the last two tracer bugfixes established -- a stuck subscriber must never be able to freeze SetGeneration()/mode transitions the way the Zenoh Put() call once did. traceRecord is exported as TraceRecord so it can be consumed from another package.
Replaces dataAnalysis's scripts/live_quality_scorer.py (which ran as a separate process, fed either by tailing traces/<date>.jsonl or, most recently, subscribing to Zenoh) with a component living inside OM1 itself, subscribed directly to Tracer.Subscribe()'s in-process channel -- no serialization, no broker, no second process. For each trace record it extracts the prompt/response (ported from extract_prompt_response_pairs.py), detects language locally (CJK script-override + English-stopword heuristic ported from lang_utils.py, falling back to whatlanggo instead of langdetect -- an approximation, not a byte-identical port, since no Go library shares langdetect's training corpus), and classifies input tone + response coherence via two structured-output chat-completions calls using the exact system prompts from classify_user_inputs.py / classify_prompt_response_coherence.py. Metrics are emitted by a custom prometheus.Collector (the Go equivalent of the Python LiveQualityCollector) registered on prometheus.DefaultRegisterer -- the same registry internal/metrics' existing :9090 server already serves at /metrics, so no new port. Metric/label names match the Python version exactly, so the existing Grafana "OM1 Quality Scores" dashboard needs no changes. Unlike the Python version, all-time and trailing-1h counts are tracked in memory rather than re-derived from a log file on every scrape -- that design existed so an independent script restart couldn't lose state, which doesn't apply once this lives inside OM1's own process. A JSONL classification log is still written for offline/historical parity. Config: new optional quality_scorer object on SystemConfig (enabled/model/ base_url/api_key), mirroring knowledge_base's nested-object pattern, with matching schema entries in both schema files. Depends on use_tracer being enabled on the same config; logs a startup warning otherwise. prometheus.yml's now-redundant om1_quality_live scrape job (port 9201) is removed -- those metrics are on the existing om1 job's port 9090 now. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sources the API key from OPENAI_API_KEY (empty default) so this safely no-ops with a warning, rather than crashing, if that env var isn't set.
Every existing log line was a warning for a failure case -- there was no confirmation that the service started, and no visibility into a successful scoring, so a fully-working run produced zero log output. Mirrors the Python original's per-turn print statement.
Falls back to the plugin's default (10fps) instead of the previously hardcoded 30fps.
Replaces the dataAnalysis/live_quality_scorer.py clone-and-run instructions with the quality_scorer config block, since scoring now runs inside OM1 itself rather than as a separate Python process.
…ector Replaces the custom prometheus.Collector (in-memory recent-events slice, manual pruning, _last_1h gauges) with plain per-run CounterVec/Counter/Gauge vars registered in internal/metrics, alongside the rest of OM1's metrics. The dashboard's short/long-window panels already compute "last hour"/"15m" via increase() over the plain counters -- the custom windowing was redundant with that. Also fixes one dashboard panel that referenced om1_quality_live_turns_scored_total (a name never actually exposed).
- config: use_tracer becomes an object ({enabled, quality_scorer}) instead
of a bare bool, since quality_scorer depends entirely on tracing being on.
- qualityscorer now defaults to Gemini via OpenMind's own gateway
(gemini-3.1-flash-lite, https://api.openmind.com/api/core/gemini) instead
of hitting OpenAI directly -- matching every other LLM plugin in this repo.
No request/response-shape changes needed, since OpenMind's gateway is
already OpenAI-compatible.
- quality_scorer.api_key now falls back to the top-level system api_key when
unset (same fallback every LLM plugin gets via buildSystemMeta/addMeta),
so it no longer needs its own separate key.
- Pre-register every named language (language.go's langNames) at zero on
startup, same fix already applied to classification/coherence: a label's
first real occurrence is otherwise invisible to increase()/rate() over
short windows.
- Renamed qualityscorer.StartServer to Start -- it never starts a server of
its own, just a background goroutine and some Prometheus registrations.
That exporter (dataAnalysis's export_quality_metrics.py) isn't running and its underlying data/processed/*.csv files haven't been regenerated in days, so it was only ever scraping a dead target.
Reduces every multi-line comment added on top of origin/main (across tracer.go, qualityscorer, metrics.go, config/types.go, cmd/main.go) down to a single line, or removes it where the content was redundant with the code. Pre-existing comments and the LLM prompt string literals in classify.go are untouched.
use_tracer used to be a plain bool; other configs (outside this repo, or
just not yet touched) may still use that form, and it broke when the type
changed to *TracerConfig. TracerConfig now has a custom UnmarshalJSON
accepting either use_tracer: true or the nested {enabled, quality_scorer}
object, with matching oneOf schema updates and a test covering both forms.
Runs Silero VAD v5 (https://huggingface.co/runanywhere/silero-vad-v5) locally over the same mic/RTSP audio ASR sensors already capture, to find how long each ASR vendor takes to return a transcript after the person actually stopped talking -- independent of the vendor's own (often later, vendor-biased) speech_end signal. - internal/vad: ONNX wrapper (onnxruntime_go, runtime-dlopened) for Silero VAD v5, plus a resampling frame segmenter with a threshold+hangover state machine emitting speech_start/speech_end events timestamped at the true silence onset. - plugins/inputs/asr: vadLatencyTracker pairs each local speech_end event (FIFO) with the next accepted transcript and appends {utterance_ended_at, transcript_at, latency_ms, provider, transcript} JSONL records to data/vad_asr_latency.jsonl. Wired into the shared asrCommon/asrSensorCore audio and transcript chokepoints, so every vendor (Google/ElevenLabs/Riva, mic and RTSP) and the parallel multi-provider sensor get it for free. Opt-in via enable_vad_latency; degrades to a no-op with a warning if the onnxruntime library or model aren't available. - Makefile: download-onnxruntime / download-vad-model targets (platform-matched, mirroring download-zenohc). Neither is a build prerequisite since onnxruntime_go dlopens its shared library at runtime, not link time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The FIFO queue paired each accepted transcript with the *oldest* unmatched speech_end. But VAD speech_end events and accepted transcripts aren't reliably 1:1: a short blip or ambient noise can trigger speech_end with no transcript ever following it (e.g. acceptASRTranscript rejects anything under 3 words), so that stale event sat in the queue and later paired with a much later, unrelated transcript -- producing wildly wrong latencies (observed: 57s for audio that round-tripped in ~3s). Track only the most recent unmatched speech_end instead of a queue; a newer event now fully replaces an older unconsumed one rather than queuing behind it. Also add a sanity bound (20s) so an implausible pairing is logged and dropped instead of written as a bogus record. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously only speech_end logged, and only at Debug (invisible without -log-level debug). To help diagnose whether reported vad-asr latency is inflated by the VAD's own timing rather than the ASR round trip, log both boundaries at Info by default, including the detected utterance duration on speech_end. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The metric was measuring vendor-reported speech-start to transcript, which conflates utterance length with actual wait time and inflates the dashboard's total-turn-latency panel. It's now fed by the local Silero VAD tracker's end-of-speech timestamp instead, matching what users actually perceive as latency. Requires enable_vad_latency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Interrupting TTS used to wait for a full vendor ASR round-trip plus a 3-word acceptance threshold, which is slow enough to feel unresponsive. The local Silero VAD tracker already runs over every mic chunk during TTS playback (when enable_tts_interrupt is on), so speech_start now triggers the interrupt directly, gated by a short confirm delay (150ms default, tunable via vad_interrupt_confirm_ms) to filter out single-frame blips the segmenter has no debounce for. VAD support is now loaded whenever either enable_vad_latency or enable_tts_interrupt is set; a missing onnxruntime/model logs an error (was a warning) but the sensor still starts, just without barge-in or latency tracking. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
origin/main independently landed the same live conversation-quality scorer feature this branch had built (internal/qualityscorer, wired via cmd/main.go). Took upstream's more complete version instead: internal/tracer/qualityscore + internal/tracer/tracer.go (renamed from internal/providers/tracer.go), wired through internal/runtime/runtime.go's Start/Stop instead of manual wiring in main. Local-only VAD-latency and TTS-interrupt work is unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Wrap increase() count queries in round() so Grafana panels show integers instead of PromQL's fractional extrapolation. - Condense comments across the VAD-latency/TTS-interrupt code to match the repo's terse, sparing comment style: trim redundant multi-line doc comments and remove changelog-style narration in the ASR vendor files in favor of stating current behavior.
Strip most remaining explanatory comments down to single lines across the VAD/ASR files and Makefile. Also fixes a dangling FrameSamples comment left mid-sentence by the previous pass.
github-actions
Bot
added
dependencies
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Implements a VAD for faster interrupt and more accurate ASR latency reporting
(If applicable, linked issue: [ ])
Type of change
Changes
Implements a VAD for faster interrupt and more accurate ASR latency reporting
Checklist
Impact
Creates a VAD system running in the background consuming a small portion of the CPU