Gate G5: speech accuracy

G5 PARTIAL

QuestionCopy Anchor Link

Is speech-to-text accuracy on noisy bid audio high enough to accept bids without a manual confirm?

ResultCopy Anchor Link

MethodCopy Anchor Link

• Cached eval: 24 clips (24 utterances) were sent to each provider once and the outputs recorded; the cached eval scores the recorded outputs through the rule parser. It runs in the voice-eval workflow (text and cached modes run on pull requests with no keys; live mode is manual).
• Clip set one: synthetic eSpeak speech, noise-free and robotic (PR #1940).
• Clip set two: the same 24 utterances spoken by one natural text-to-speech voice (PR VOI-01). Still synthetic and noise-free. The current recordings come from the option trials of PR VOI-02 (voice-eval runs 42364412751 and 42365428369).
• Providers: Deepgram nova-3 (punctuate=true, smart_format=false) and Groq whisper-large-v3-turbo with a bid-words prompt and temperature 0. The first run used Deepgram nova-2 and Groq without a prompt.
• Pass rule for the live run (npm run gate:g5 -w @gavel/e2e -- --write-gates, 20 private noisy recordings, DEEPGRAM_API_KEY): PASS needs exact plus recoverable confirm at 0.90 or more and wrong accepts at 0.05 or less; PARTIAL needs 0.70 or more; otherwise FAIL.

Numbers, from docs/GATES.mdCopy Anchor Link

Before the change (nova-2, no prompt): exact 0.615 to 0.692, wrong accept 0.083 to 0.167. Where the change came from, as exact and wrong-accept clips, Deepgram eSpeak: main 0.615 and 3 clips; the label fix alone 0.769 and 1; plus parser fixes on the old recordings 0.846 and 0; plus provider options 1.000 and 0. The 9 clips that drove a rule went from 0.286 to 1.000 (Deepgram) and 0.857 (Groq) exact; the 15 other clips were 1.000 before and after. With 24 clips one clip moves a rate by 0.042.

System level: nothing is auto-accepted on the room microphone (0 of 24 per run, by construction); with a device microphone and a second reading that always agrees, 0 of 15 auto-accepted bids were wrong. Per-clip causes: docs/evidence/G5/ERROR-ANALYSIS.md.

Parser only: the text-level runCopy Anchor Link

Reference text goes in as recognized text, so this measures the parser and nothing about speech-to-text. On 169 cases in 23 categories (commit 5d2e253, artifact voice-eval-text): exact 0.929, wrong accept 0.000, confirm 0.272. The held-out 40 cases were not used to change the lexicon or thresholds. Thresholds stay at AUTO 0.85 and CONFIRM 0.50 (issue 478): the sweep from 0.70 to 0.95 is flat because the parser emits only 0.50, 0.95 and 1.00.

EvidenceCopy Anchor Link

• 24 clips under packages/voice/fixtures/audio and recorded outputs under stt-cache (Deepgram, Groq) and stt-cache-human.
• docs/GATES.md, section G5.

LimitsCopy Anchor Link

• Synthetic audio only, noise-free. No real room, no noisy room, no accents beyond the parser's text cases.
• The 24-clip sample is small, and the parser rules and provider options were chosen after reading the failures on it.
• Latency is not measured by the text mode.
• The nightly harness skipped G5 (20 of 20 private recordings missing), so no live G5 run exists.

What GAVEL does about itCopy Anchor Link

A spoken bid is never accepted silently: it goes through the confirm gate. See voice and confirm. For demos, use the recorded replay, which makes no PayPal call.

• packages/e2e/src/gates/g5.tsthe G5 harness
• packages/e2e/src/gates/g5-cli.tsthe live run command
• packages/voice/fixturesclips and recorded provider outputs
• .github/workflows/voice-eval.ymltext, cached and live modes