Why your AI voice agent interrupts callers

End-of-turn detection in most stacks starts with voice activity detection. A VAD reports whether the audio contains speech right now. That is all it reports. It has no view of what was said, so it cannot know that a caller who just said “and” or “because” is mid-sentence, one vendor guide suggests reading the transcript’s trailing words for exactly this reason.

Stack a fixed silence threshold on top of a signal that shallow and you inherit its blindness. Coval, which builds voice agent test tooling, puts the sensitivity bluntly: a 100 ms change in the silence threshold can be the difference between an agent that feels responsive and one that constantly interrupts. They also note turn detection failures are among the most reported problems in voice deployments and among the least systematically tested.

The classic trigger is someone looking something up. A caller asked for their order number pauses to find the email. Three quarters of a second of silence later, the agent starts a new sentence. The caller starts over. Now both are talking.

This is the part I think gets underweighted. On an 8 kHz phone line with imperfect echo cancellation, the agent’s own voice comes back down the line and the VAD dutifully reports speech. The agent hears itself, concludes it is being interrupted, and stops.

Picovoice names this directly in its failure-mode guide and recommends running VAD on the caller’s channel with the agent’s playback suppressed, rather than on the mixed audio (Picovoice). Another guide calls the echo leak the common bottleneck and argues that a VAD tuned on clean audio is the leading cause of false triggers on phone lines (Dograh).

So before you reach for a smarter turn model: check whether your agent is interrupting the caller, or interrupting itself. The two look identical in a transcript and need completely different fixes.

The current answer from the frameworks is to put a small language model on top of the VAD. LiveKit’s turn detector is an open-weights model fine-tuned from Qwen2.5-0.5B-Instruct that takes the running transcript and predicts whether the pause is a finished thought. Their docs are careful that it is an additional signal to VAD, not a replacement: the VAD still detects speech presence and triggers interruptions, while the model decides when to commit the turn. It needs live STT results to work, so you cannot use it with a realtime model unless you also run a separate STT plugin (LiveKit docs). It currently covers fourteen languages, Hindi among them.

Deepgram’s write-up on the same problem is the most honest about the bill: models that fuse acoustic and semantic signals bring compute cost and confidence-threshold tuning with them, and you keep a silence timeout underneath as a fallback anyway (Deepgram). AssemblyAI frames semantic endpointing as the replacement for silence-based methods (AssemblyAI), which is the stronger claim of the two; the vendor docs that actually ship the thing describe a layer, not a replacement.

Worth knowing: almost every latency and accuracy figure in this space comes from the vendor selling the component, and the ones I could trace to a primary source were unmeasured. Treat the numbers in any turn-detection pitch, including the pitches linked above, as indicative.

You can get a long way without a model, by refusing to act on the VAD alone and requiring the transcript to agree. That is the approach Wixzel Voice takes on its composed pipelines, and the rules are plain enough to copy:

• A bare VAD hit never interrupts. It arms a timer that waits for a transcript to confirm.
• A transcript interrupts only past a word count, three words on the default setting, four on low, two on high. One- and two-word utterances let the agent finish.
• Short acknowledgements are queued, not dropped. “Yes” answering a yes/no question is content. Discarding it leaves both sides waiting in silence, which is worse than the interruption you were trying to prevent.
• Explicit stop words interrupt at any length: “wait”, “hold on”, and their equivalents in Hindi, Malayalam, Tamil, Telugu, Kannada, Bengali, Gujarati and Punjabi. “Okay” and “yes” deliberately do not, in any language.
• An utterance of three or more words that matches something the agent itself said in the last five seconds is treated as echo and ignored.
• A transcript that is just the tail of the turn already dispatched is ignored for 2.5 seconds, because streaming ASR re-emits those and treating one as new speech aborts the reply in progress.

Two knobs are exposed on the agent: voice_config.turn_taking.interrupt_sensitivity (low, normal, high) and silence_wait_ms, the extra silence tolerated after an utterance that sounded unfinished, from 200 to 3000 ms with a default of 500. If callers are being cut off, raise silence_wait_ms before you touch sensitivity, it targets the mid-thought pause specifically, where sensitivity changes behaviour on every turn.

The limitation, stated plainly: those knobs only reach the composed pipelines. On the Deepgram Voice Agent engine, endpointing and barge-in are decided inside Deepgram and the client is only told after the fact. On Gemini Live, Google’s server-side VAD decides. Set interrupt_sensitivity on either of those and nothing happens. Picking an engine is therefore also picking who owns your turn-taking, which is not how an engine comparison is usually read.

Log the decision, not just the outcome. Hamming’s runbook makes the point that you have to test both directions (false positives that cut the agent off, and false negatives that make callers repeat themselves) and that the interruption decision needs to be replayable (Hamming). A transcript alone will not tell you which of the six rules above fired.

One more from Picovoice worth stealing: when an interruption is real, cancel the in-flight reasoning and the queued audio at the logic layer, not only the media layer. Stopping playback while the LLM keeps generating gets you an agent that answers a question the caller abandoned ten seconds ago.

There is a repeatable structure for this in the voice agent evaluation checklist. If you are working in Indian languages, where backchannels are frequent and the acknowledgement words are short, see the Hindi and Malayalam walkthrough covers the engine side. And if you are weighing a framework instead, turn detection is one of the things you take on yourself: how that compares.

Every claim above about someone else’s product comes from the pages below, read on the date shown. Most are published by the vendor selling the component, so their performance figures are indicative rather than independently measured.

• Turn detection. LiveKit, read 2026-10-24
• Turn detection and interruption handling. LiveKit, read 2026-10-24
• Common failure modes in voice agents. Picovoice, read 2026-10-24
• How to test turn detection in voice AI agents. Coval, read 2026-10-24
• Silence timeouts vs VAD vs semantic models. Deepgram, read 2026-10-24
• Voice agent interruption handling runbook. Hamming, read 2026-10-24
• Turn detection and endpointing. AssemblyAI, read 2026-10-24
• Agent talking over user. Dograh, read 2026-10-24

Found an error? Send a correction.