Skip to content

PayPal sandbox only. No real money moves.

LearnHow it works

Voice and the confirm gate

Speech never becomes a bid by itself. The pipeline, the gate and where the accuracy gate stands.

Code-checkedread against the code it describes; nothing was run

Verified againstRead against packages/voice/src/decide/, packages/server/src/voice/bridge.ts and packages/server/src/room/confirm.ts (PR #1954). Accuracy is a separate question: see the G5 table below, read from docs/GATES.md.

Page last changed 7 Nov 2026 (from the git history)

In plain words

A bidder can say a number instead of tapping it. Three things stay true however well that goes:

  • Speech never bids by itself. A spoken bid is a suggestion until the room has checked it, and by default the bidder is asked "did you mean $150?" first.
  • The engine decides, not the microphone. Whatever speech produces goes through the same rules as a tap: price, increment, ceiling.
  • Speech can fail without stopping the auction. Manual entry and tap to bid never depend on it.

Whether speech is accurate enough to skip the question is the open part. The results so far are on synthetic, noise-free audio only, so bids still ask first; the numbers are further down.

The path a spoken bid takes

Voice pipeline and the confirm gate

G5 PARTIAL
Voice pipeline and the confirm gateAudio goes from the microphone through a segmenter, speech to text, a parser and a decision step. The decision is accept, confirm or drop. Confirm goes to the confirm gate: the bidder is asked a question, yes places the bid after it is checked again, no or expiry places nothing. Drop is logged and places nothing. Manual entry and tap to bid skip speech and enter at the engine. A dashed path shows that a clean, high confidence parse can be accepted without the gate; the default build keeps scores below that line.MICROPHONEaudioSEGMENTERspeech segmentsSTTcached by defaultPARSERamount, paddleDECIDEscore, classifyconfirmCONFIRM GATEthe bidder answersbefore the engine sees itENGINEthen ledgeraccept: skips the gate (two parsers agree, 0.85 or more)not the default build: its scores stay under 0.85dropnot a bid, or too uncertainlogged, nothing placedPrompt to the bidder"Did you mean $150for paddle 12?"yes: checked againagainst the live priceno, or 15 s passnothing is placedloggedManual entryTap to bid (QR)skip speechheldchargedreleasedneeds review

Scroll the figure sideways, or open it full size.

Read from packages/voice/src/decide/policy.ts, decide.ts, score.ts, packages/server/src/voice/bridge.ts and room/confirm.ts. The dashed red path is what the code allows, not what the default build does: with one parser the score tops out at 0.82, so a spoken bid confirms. Accuracy numbers are in the table below the figure, not drawn here.
Text version of this diagram
  1. Microphone audio is cut into segments.
  2. Speech to text turns a segment into a transcript. The default build plays cached transcripts; a live provider (Groq or Deepgram) runs only when VOICE_STT is set with its key.
  3. The parser reads amount, paddle and intent from the transcript.
  4. The decision step merges parser opinions, scores confidence and validates the bid against the room. The result is accept, confirm or drop.
  5. Confirm: the bidder is asked, for example "Did you mean $150 for paddle 12?". Yes places the bid after it is checked against the current price. No or expiry places nothing and is logged.
  6. Drop: nothing is placed, the decision is logged.
  7. Accept (dashed): a clean parse with confidence of 0.85 or more that both parsers agree on skips the gate and goes to the engine.
  8. Manual entry by the auctioneer and tap to bid skip speech and enter at the engine.
  9. The engine accepts or rejects the bid and an accepted bid is appended to the hash-chained ledger.

Gate G5 is PARTIAL: wrong-accept 0.000 to 0.000 on 24 synthetic, noise-free clips per set (limit 0.05), after fixes chosen on those same clips. No real-room or noisy-room test exists, and spoken bids always confirm.

The pipeline

  1. Segment. Audio is cut into speech segments.
  2. Speech to text. The default build plays cached transcripts. A live provider (Groq or Deepgram) runs only when VOICE_STT is set and its key exists.
  3. Parse. A rule parser reads amount, paddle and intent.
  4. Decide. The step merges parser opinions, scores confidence and validates the bid against the room. The result is accept, confirm or drop.
  5. Confirm. A question goes to the bidder (for example "Did you mean $150 for paddle 12?"). A yes is checked again against the live price before a bid is placed. A no or an expiry places nothing and is logged.

Some situations always confirm, whatever the confidence: a correction, a hedged phrase, several amounts or bidders, an unregistered paddle, a large jump, an implausible amount.

The pipeline in detail

Step through it, or hover and focus a box to see what it does and which file implements it.

Voice pipeline in detail: capture to engine

G5 PARTIAL
Voice pipeline in detail: capture to engineFour bands from top to bottom. Browser: microphone, mute controller, a 50 batch audio queue, a drop monitor and the degraded state. Server: the audio socket with a token in the first message, then speech to text through Deepgram, Groq and cached replay, then a segmenter. Decide: rule parser, score and classify into accept, confirm or drop. Room: the room gate holds low confidence or ambiguous bids for 15 seconds and asks the bidder; a yes goes to the engine, a clean bid at 0.85 or more goes straight to the engine, a drop is recorded without its transcript.heldchargedreleasedneeds reviewBROWSERMicrophoneMuteControllermute, push to talkAudioQueue50 batches, drops oldestDropMonitorover 2 s lost in 10 sCapture state: degradedmanual entry offeredkeepalive every 5 s while mutedSERVER: /audio SOCKET AND SPEECH TO TEXT/audio WebSocketauctioneer token in 5 sDeepgramfirstGroqsecondCached replaylastSegmenterdrops echo, duplicatesVOICE_STT=deepgram with a Groq key. Switch on: no event 3 s after audio, final later than 4 s,two errors in a row, fatal, closed. Deepgram is tried again after 30 s. Default build: cached replay only.DECIDE, ROOM GATE AND ENGINERule parseramount, paddleScore and classifyhard flags confirmacceptconfirmdropRoom gatebelow 0.85 or ambiguousDrop recordHeld for 15 sbidder is askedEngineholdyes0.85 or more and not flagged: straight to the engineno or 15 s: no bid, audited

Scroll the figure sideways, or open it full size.

Step 7 of 7. Arrow keys work too.

Answer. Yes is checked again by the engine at the time of the answer. No, or 15 seconds of silence, places nothing and is audited.

Hover or focus a box for what it does and which file implements it.

Read from packages/voice/src/browser (mute, queue, drop monitor), packages/server/src/host/audio-ws.ts, packages/server/src/voice/stt.ts, packages/voice/src/stt/failover.ts, packages/voice/src/decide/policy.ts, packages/server/src/voice/bridge.ts and packages/server/src/room/confirm.ts. The default build plays cached transcripts; a live provider needs VOICE_STT and its key. Speech accuracy gate G5 is PARTIAL: wrong-accept 0.000 to 0.000 on 24 synthetic, noise-free clips per set (limit 0.05), after fixes chosen on those same clips; there is no real-room or noisy-room test. A clean bid at 0.85 or more is not held by the code today, so spoken bids go through the confirm prompt by decision in docs/GATES.md, not by a code switch.
Text version of this diagram
  1. Browser: the microphone feeds a mute controller. Muted means no audio, a keepalive every 5000 ms, and the utterance is ended.
  2. A send queue holds 50 batches and drops the oldest when full. The drop monitor raises a degraded state when more than 2000 ms were dropped in 10 s; the person is told to use manual entry. It recovers after 10 s without drops.
  3. The /audio socket needs the auctioneer token as its first message within 5 s. Limits: 64 KiB per second, 100 frames per second, 30 minutes, 2 streams per token, 4 per room.
  4. Speech to text: Deepgram, then Groq, then cached replay when VOICE_STT is deepgram and a Groq key exists. Switch triggers: silent 3000 ms, late final 4000 ms, two errors in a row, fatal, closed. Deepgram is tried again after 30 s.
  5. The segmenter removes auctioneer talk and duplicates. The rule parser reads amount, paddle and intent. Decide classifies: hard flags confirm, below 0.50 drop, below 0.85 confirm, else accept.
  6. The room holds a voice bid below 0.85 or flagged ambiguous for 15 seconds and asks the bidder. A drop is stored without its transcript. A clean bid at 0.85 or more goes to the engine.
  7. A yes is checked again by the engine at the server time of the answer. A no or an expiry places nothing and is audited.

Gate G5: wrong-accept 0.000 to 0.000 on 24 synthetic, noise-free clips per set (limit 0.05), fixes chosen on the same clips. Status PARTIAL.

Which speech provider runs

With VOICE_STT unset (or replay) the cached provider plays recorded transcripts. No key is needed and nothing leaves the host.

AUTO or CONFIRM

The decision step classifies the segment. Confidence at or above 0.85 with no hard flag is accept. From 0.50 up to 0.85 is confirm. Below 0.50, or nothing that looks like a bid, is drop. Hard flags (correction, hedged phrase, several amounts or bidders, homophone, unsupported currency, fractional amount, large jump, unregistered paddle, implausible amount, model disagreement) always confirm.

The bridge hands it to the room. An accept becomes a voice bid carrying its confidence. A confirm becomes a voice bid flagged ambiguous. A drop is written to the audit trail without the transcript.

The room gate applies. A voice bid below 0.85, or flagged ambiguous, is held for 15 seconds and its bidder is asked. A clean bid at 0.85 or more goes straight to the engine, which still checks price, increment and ceiling.

The answer is checked again. A yes runs the normal bid pipeline at the server time of the answer. A no, or no answer within 15 seconds, places nothing and is audited.

Two timers, two layers

The decision step keeps its own prompt for 8 seconds (CONFIRM_TTL_MS in packages/voice/src/decide/decide.ts). The room holds a bid for 15 seconds (CONFIRM_TTL_MS in packages/server/src/room/confirm.ts). The bridge submits a confirm decision to the room, so the room's 15 second hold is the one a bidder meets.

Mute and the drop monitor

Muting is a safety control as well as a convenience. While muted, no audio is sent: the browser sends an empty keepalive every 5 seconds so the stream stays open, and muting tells the provider the utterance has ended. Push to talk starts muted and opens the microphone only while the talk key is held. If the microphone track ends (device unplugged) one automatic restart is tried before the state becomes an error.

The drop monitor watches the browser's send queue. The queue holds 50 batches and drops the oldest when it is full. If more than 2 seconds of audio are dropped inside 10 seconds, capture moves to a degraded state and the person sees "Connection is slow, voice bids may be missed. Use manual entry." Ten seconds without a new drop returns it to running. Manual entry and tap to bid are never affected.

How accurate is it?

G5 PARTIAL

Where this lives in the code

Edit on GitHub

Last updated on

Crash recovery code-checked

What happens when the process dies in the middle of a capture.

Engine and ledger verified

A deterministic auction core and a hash-chained audit trail.