What we got wrong building our first voice agent

The first version of our voice agent was impressive in a demo and mediocre on a phone. It answered questions correctly, used a pleasant voice, and fell apart the moment a real caller interrupted it. Here is what we changed, in the order it mattered.

Latency is a product decision

Every component adds delay: speech recognition, the model, the tool call, the speech synthesis. Individually they look fine. Stacked, they produce a pause that feels like a bad international line. We stopped optimising the model and started optimising the path — streaming recognition, parallel tool calls, and a shorter prompt with fewer instructions competing for attention.

The target we hold ourselves to is under 800 milliseconds from end of speech to first audio. Above that, callers start saying hello again, and the conversation never fully recovers.

Accents are not a checkbox

Our first deployment was in Melbourne. The speech stack handled a neutral Australian accent well and struggled with the way people actually speak on a Friday night: names spelled quickly, suburbs clipped, background noise from a car. We added a confirmation step for names and phone numbers, and an explicit repair phrase the agent uses when confidence is low.

• Confirm names and numbers by reading them back, digit by digit
• Ask for the suburb rather than assuming it from the phone number
• Keep a short list of phrases that trigger a clean handover
• Log every low-confidence turn for weekly review

Barge-in changes the whole design

People interrupt. They interrupt because they are in a hurry, because they have heard enough, or because they think the agent is a recording. If the agent keeps talking, it confirms the suspicion. We rebuilt turn-taking so speech stops the moment the caller speaks, and so the agent can hold a partial answer without losing the thread.

Guardrails are cheaper than apologies

The first version would answer almost anything. That felt like capability and was actually risk. We now constrain the agent to a small set of intents with explicit tools, and everything outside that set routes to a human. The agent cannot invent a price, promise a refund, or confirm an allergen. It can look up, book, confirm and escalate.

The most useful line in the prompt is not an instruction about tone. It is the list of things the agent is not allowed to do.

Evaluate with recordings, not vibes

After every deployment we review a sample of real calls against a fixed rubric: was the intent recognised, was the answer correct, did the caller have to repeat themselves, and did the handover include context. The rubric is boring. It is also the only way to tell whether a change made the agent better or just different.

What we would do first next time

• Record and transcribe a week of real calls before writing a prompt
• Build the handover before building the personality
• Choose the POS or CRM integration before choosing the voice
• Set a latency budget and test it on a mobile handset, not a laptop

None of this is exotic. It is the difference between a demo that delights a room and a system that survives a Saturday night.