How Brisvox works
Brisvox is one Rust process that runs each call as its own chain of stages: audio in, voice activity, speech to text, the turn decision, the LLM and its tools, text to speech, audio out. This page covers the parts that decide how a call feels.
Last updated:
How a call flows
A call is one session: a chain of stages connected by bounded queues. Each call has its own chain, so calls share nothing but the inference pool and the provider connection budgets. A panic or a provider failure in one cannot touch another.
- Transport inThe caller's audio arrives over a phone carrier, WebRTC or a websocket.
-
AudioInResamples the call to 16 kHz and forwards it at once. A copy goes to the VAD through a bounded, drop-oldest queue, so audio never waits on a model. -
VadWorkerSilero voice-activity detection as jobs on the dedicated inference pool. Emits backdated speech start and stop events. -
SttStageStreams audio to the speech-to-text provider. At the VAD stop it sends Finalize immediately, in parallel with the end-of-turn model. -
TurnDriverThe single authority over turns and epochs: end-of-turn decisions, barge-in judgement, the fast Clear towards the transport. -
LlmRunnerOne task per turn: streams the LLM, runs the tool loop, cuts the context to the text that was actually played. -
SegmenterStageCuts the streaming text into speakable chunks. The first clause goes to the voice after one word. -
TtsStageStreams text to the text-to-speech provider and cuts audio into 20 ms frames at the call rate. -
TransportOutPaces audio onto the transport, tracks what was actually played and executes Clear.
Turns and interruptions
A voice agent has to know when the caller has finished, tell a real interruption from a "mm-hmm", and keep the history true to what the caller actually heard. Brisvox cuts the LLM context at the last word that was played.
Knowing when the caller is done
When the VAD sees the caller fall silent, two things start at once: the STT Finalize and Smart Turn v3.2, an on-device end-of-turn model. If the model says the utterance is complete, the turn ends as soon as the STT final arrives. If not, the turn waits, with a fallback deadline so an undecided pause cannot hang the call. The added wait is the larger of the two, not their sum.
Telling a barge-in from a "mm-hmm"
An interruption policy judges every candidate: minimum words, backchannel lists per language such as "yeah" or "uh-huh", an echo guard that recognises the agent's own words coming back, and a level gate that keeps a quieter second voice from starting a barge-in.
Proven on a paused clock
Time comes from an injected clock, so a whole call runs deterministically in simulation, with mock providers, in milliseconds of real time. A context that would break the LLM's message rules is a test failure, not a runtime surprise.
A cough does not end the answer
Since 0.1.4 (November 2026) the bot pauses instead of stopping when a sound might be a barge-in, and decides
within 1.5 seconds whether the caller really interrupted. It is on by default in agent files, the same behaviour
as LiveKit's resume_false_interruption.
- Pause after 400 ms of speech. A click or a door bang shorter than that never pauses the bot. The caller hears silence about as early as an interruption would give it.
- Interrupt on a word. Any recognised word of the caller, or a command such as "stop", "wait", "dur" or "bekle", interrupts at once. The bot's own echo does not count.
- Resume when nothing was said. If the sound stopped and no word came, the bot continues the same answer from the paused point. Anything uncertain interrupts, so a real barge-in is not dropped on a guess.
- Stop pausing for a noisy room. After two resumed pauses in a minute, the bot keeps talking until it hears a word.
Measured with the built-in harness on mostly synthetic noise and a modelled recognizer: noise without speech cut
the bot off on 13 % (wideband) and 19 % (telephone) of the candidates it made, and on none with this setting.
Short commands the recognizer misses are the cost. The numbers are in
barge-in-noise.md. A second person who talks
still interrupts the bot. Set pause_then_decide = false to turn it off.
Tools never run on a guess, and never twice.
A tool is a function the LLM can call during a turn: look up an order, book a slot, transfer the call. You mark each one as read-only or as a side effect, and the engine holds it to that.
The never-twice guarantee covers one call's turn. For a rule such as one refund per order, ever, keep your own idempotency key. The exact scope is in the tool-calling contract.
| Behaviour | read_only | side_effect |
|---|---|---|
| For | Looking things up | Booking, cancelling, transferring, charging |
| On a speculative LLM start | May run | Held until the turn is committed, discarded if it is not |
| When the caller interrupts | Cancelled | Completes, is recorded, and the outcome is reported on the next turn |
| Called twice | Harmless | Refused: the idempotency key is derived from the session, the turn epoch and the call id |
| Slow to answer | After 1.2 s by default the agent says a filler phrase, so the line is not silent | |
The providers you already use. The lines your callers already call.
Providers are Cargo features, so a binary carries only the adapters you name. An agent file that names a provider that is not compiled in is refused at startup, with the feature to turn on.
| Stage | Cloud | On your own machine |
|---|---|---|
| Speech to text | Deepgram, AssemblyAI, Gladia | sherpa-onnx (CPU), any server speaking the OpenAI Realtime transcription protocol |
| Language model | OpenAI, Anthropic, Gemini, Groq, Together, Fireworks, OpenRouter | Ollama, vLLM, any OpenAI-compatible endpoint |
| Text to speech | Cartesia, ElevenLabs, Deepgram Aura | Kokoro-FastAPI, Chatterbox, any server of the OpenAI /v1/audio/speech shape |
| Speech to speech | OpenAI Realtime, Gemini Live | Not available |
| On device | None needed | Silero VAD (compiled in), Smart Turn v3.2 end-of-turn detection (ONNX Runtime) |
Transports
- WebSocket
- Generic audio protocol, built in
- Twilio Media Streams
- Built in, with transfer, DTMF and hang-up
- Telnyx media streaming
- Built in, same call control
- WebRTC
- Cargo feature webrtc, WHIP signaling, Opus
- SIP trunk
- Cargo feature sip, UDP, TCP or TLS with SRTP
- RTVI
- Pipecat's client SDKs connect unchanged
G.711 mu-law and A-law, any call rate with any provider rate. English and Turkish are supported, and an agent can switch language in the middle of a call. No media server is needed.