“Getting a voice agent right” isn’t about the word-recognition rate in lab tests. For a business, what matters is the share of calls the agent handled correctly: understood the request, didn’t break the dialogue, drove to the target action or handed off to a human in time. Let’s break down what goes into it and how to get it to production.
The call pipeline
For a single turn the agent goes through four steps: it recognises the caller’s speech, works out what to say, speaks the answer back and carries it all over the phone line. Error accumulates along the chain: a mishearing early on distorts understanding, and a delay at any step kills the liveliness of the conversation.
Recognition: where words are lost
Phone speech is a compressed channel — noise, accents and interruptions. It’s the details that matter most to the business that suffer most: names, addresses, numbers and amounts. What helps:
- The agent lets itself be interrupted: the caller starts speaking and the agent goes quiet, like a real interlocutor
- The agent recognises when a person has finished, without cutting them off on a pause
- Hints from your domain (a list of products, common names and formats) lift recognition of the details that matter
Response speed decides
Even a perfectly understood request is useless if the answer arrives three seconds late — the caller already thinks “the line froze”. The target is to answer in under ~1–1.5 seconds. It’s achieved by having the agent start speaking before it has fully processed the reply, and by recognising speech on the fly rather than after the person has finished.
In voice, speed matters more than a pretty phrase. The conversation stays alive while the agent replies at a human rhythm.
Script and human handoff
Getting it right also means honest boundaries. The agent should confidently run a standard dialogue, but on doubt or an unusual request it must not make things up — it hands off to a human with the context already collected. The “time to call a human” logic is wired explicitly: which signals and phrases trigger the handoff.
How to measure and improve
You can’t improve what you don’t measure. We label real calls, score the share of correctly handled dialogues per stage, and test changes on live calls. The final metric isn’t a recognition percentage but the result in client money: connect rate, conversion to conversation, share driven to the target action.