Voice Agent Testing: What's Different

Voice agent testing adds an audio layer on top of everything text-agent testing already requires: latency budgets, interruption handling, speech-recognition errors, and synthesis quality. But the dialog brain underneath is still an agent — and most voice failures are dialog-logic failures. The effective approach is layered: test the dialog logic with simulated users first, then test the audio loop.

What voice adds

A voice agent is a pipeline: speech-to-text (ASR), the dialog agent, text-to-speech (TTS), all running under real-time constraints over a phone line or mic. Each stage adds failure modes text never sees.

Latency budgets and dead air

In chat, a three-second pause is invisible. On a call, it's dead air — callers assume the line dropped, say "hello?", or hang up. Voice agents live under a latency budget measured in hundreds of milliseconds for the first audible response, which is brutal for an LLM pipeline that wants to retrieve, reason, and call tools before speaking.

Test for it: measure time-to-first-audio per turn, not just total response time, and find the turns that blow the budget — usually the ones involving tool calls. Watch for the coping mechanisms too: filler phrases ("let me check that for you…") are fine once, grating three turns in a row.

Interruptions and barge-in

Callers interrupt. They answer before the question finishes, cut off a long disclaimer, or talk over the agent to correct it. Barge-in handling is two problems stacked: an audio problem (detect speech, stop the TTS quickly) and a dialog problem (the agent's last utterance was only partially delivered — does its state reflect what the caller actually heard?).

Classic failures: the agent bulldozes on through the interruption; it stops but then resumes the interrupted sentence verbatim; or its dialog state assumes the caller heard information that was cut off.

ASR errors that cascade

Speech recognition is the noisiest stage, and its errors don't stay contained — they become the dialog agent's input. The chronic trouble spots:

• Names and spellings. "Kaczmarek" arrives as something else entirely; email addresses read aloud are a lottery.
• Numbers. Account numbers, order IDs, phone numbers — one substituted digit is a wrong-account lookup, which is a worse failure than no lookup.
• Accents and non-native speech. ASR error rates are not uniform across speakers; the callers your ASR hears worst are a segment your agent fails hardest, invisibly.
• Noise and crosstalk. Background TV, car cabins, a second person talking.

The compounding question is what the dialog agent does with a garbled transcript: confirm ("I heard order 4-3-1-2 — is that right?") or barrel ahead on the misheard value? A robust dialog brain routinely repairs ASR damage; a brittle one turns a one-word transcription error into a wrong cancellation.

TTS artifacts and prosody

On the way out: mispronounced names, product terms read absurdly, numbers voiced as "four thousand three hundred twelve" when the caller needs digit-by-digit, and flat or weirdly cheerful prosody in an apology. Also formatting leakage — the dialog model emits a bulleted list or a URL, and the TTS reads the bullets aloud.

No visual fallback

Text agents lean on the screen: the user can re-read the last message, skim a list, click a link. Voice has none of that. Long answers must be structured for ears — chunked, confirmable, repeatable on request. A response that works fine as a paragraph in chat is unusable as thirty seconds of uninterruptible monologue. Error recovery also changes: "click the link I sent" is not a move; "let me spell that" is.

Streaming and sentence-level commitment

Latency pressure forces voice agents to start speaking before the full response is generated. That means committing to sentences the model might have revised — a tool result arriving mid-utterance can contradict something already spoken aloud. Text agents can silently rewrite; voice agents can only correct themselves out loud, which needs its own graceful behavior ("actually, correction —").

What stays the same

Strip away the audio pipeline and the middle of that stack is an agent like any other: it classifies intent, tracks multi-turn state, follows policy, calls tools, decides when to escalate. Every failure mode of text agents applies unchanged — wrong intent on ambiguous phrasing, invented policy exceptions under pressure, lost context on turn eight, mishandled follow-up contacts.

And in practice, a large share of what gets reported as a "voice problem" is a dialog-logic problem wearing headphones. The caller rage-hangs not because the TTS sounded robotic but because the agent asked for the order number a third time — a state-tracking failure that would reproduce identically in a chat window. Multi-turn state, clarification behavior, and policy compliance are testable without any audio at all; that's the ground covered in Multi-Turn Evaluation.

This matters economically. Audio-loop testing is expensive and slow: real-time playback, telephony infrastructure, synthesized caller audio in many accents. Spending it to discover intent-classification bugs is using your most expensive test rig to find your cheapest bugs.

A layered testing approach

The practical structure is two layers, tested in order.

Layer	What you're testing	How
1. Dialog logic	Intent handling, multi-turn state, policy, escalation, clarification and repair behavior	Text-based simulated users, at scale, reproducibly
2. Voice layer	Latency, barge-in, ASR robustness, TTS quality, streaming behavior	Audio-loop tests: synthesized/recorded caller audio through the real pipeline

Layer 1: simulate the callers as text. Run the dialog brain against simulated users with realistic variety — different ages, language proficiency, patience, phrasing habits — over multi-turn conversations, including follow-up calls about unresolved issues. This is standard user simulation, and it's where most of the failures are. You can even approximate voice conditions in text: inject transcription-style noise (misheard names, dropped words) into the simulated user's messages and test whether the agent confirms and repairs rather than assuming.

This layer is where Synthetic Signals fits — and to be clear about scope: Synthetic Signals is text-based today and does not test voice calls. What it tests is the dialog brain, against a Census-grounded population of simulated users who remember across sessions, reproducibly enough that a failing conversation becomes a regression test. For a voice agent, that's the foundation layer — the intent, policy, and multi-turn behavior everything else sits on — not the audio layer.

Layer 2: close the audio loop. With the dialog logic solid, test the pipeline end to end: play caller audio (varied accents, background noise, real names and numbers), verify time-to-first-audio per turn, interrupt the agent mid-sentence and check both the audio stop and the dialog state, and listen to actual TTS output on your domain's vocabulary. This layer needs voice-specific tooling — and because it's slow and costly per test, it should be spent on genuinely audio-specific questions, not on re-finding dialog bugs layer 1 could have caught.

Then hold the seam. The layers interact: an agent that never confirms values is fine in clean text and dangerous behind a noisy ASR. So layer 1 should test repair behavior explicitly (garbled inputs, contradictory corrections), and layer 2 should include a few full-journey calls that exercise dialog and audio together.

Honest limits

Text-first testing cannot certify a voice agent. Latency, barge-in, ASR robustness, and TTS quality are real product surface, and no amount of text simulation touches them — layer 2 is mandatory, not optional. Conversely, audio-loop tests are too expensive to give you behavioral coverage across hundreds of caller types; run them broad-shallow and let text simulation be deep-broad. And simulated callers, in either layer, are still models of people: treat their findings as strong evidence, and calibrate against real call recordings as soon as you have them.

The one-line summary: voice adds a hard real-time audio layer, but the failures that lose customers mostly live in the dialog brain — so test the brain first, at scale, in text, then earn confidence in the audio loop on top.