Voice agent evaluation checklist before you scale

Define success as a verified business outcome. For an appointment flow, check that the correct time, service and caller details were recorded and that the spoken confirmation agrees with that record. A fluent conversation or a non-empty transcript is not sufficient evidence of completion.

Separate connection, conversation quality and task completion. A carrier rejection is a different failure from a misunderstood name, and both are different from an appointment API returning an error. Record the failure category so fixes target the correct layer.

Build a test set from the tasks and languages you expect to serve. Use consenting testers and synthetic details where possible. Keep prompts and scenario instructions versioned so you can repeat the same evaluation after changing an engine or integration.

The following cases form a starting checklist. They are proposed tests, not claims that every engine passes them. Add the caller behavior and failure conditions that matter to your application.

• A straightforward request with an unambiguous expected result.
• A corrected name, phone number or appointment time partway through the conversation.
• An interruption while the agent is speaking, followed by a different request.
• Background noise, silence, an unfamiliar accent and each supported target language.
• A question outside the knowledge base and a request the agent is not allowed to fulfill.
• An unavailable backend, duplicate event and caller disconnect before completion.
• A request for a human, tested against the escalation behavior your application actually implements.

Record the scenario, engine and prompt version, call identifier, connection result, task result, duration and cost. Have a reviewer check the underlying business record instead of letting the same agent grade its own answer.

For responsiveness, define the interval you measure, such as the end of a caller’s utterance to the start of audible agent speech. Report the distribution, sample size and test conditions. Do not turn a single fast demo into a general latency promise.

• Task completion rate = verified completions / conversations evaluated for that task.
• Connection rate = connected calls / attempted calls, reported separately.
• Cost per completion = all pilot spend / verified completions; report undefined when there are zero completions.
• Record critical errors separately so an average cannot hide a wrong booking or invented answer.

Read the call record for status, transcript and cost_micros. When a call did not connect, inspect failure_code and failure_reason before changing the prompt. Reconcile charges with usage events. Review webhook deliveries when the call completed but your application did not update.

Use idempotency keys when starting calls so a request retry does not create a second dial. Test your own event handler for duplicate deliveries and verify its stored result. Authentication and webhook signature checks belong in the integration before the pilot grows.

Expand when your predefined acceptance criteria are met, critical failures have owners and the operational path is tested. Choose thresholds for the risk and value of your use case; there is no universal completion rate that makes every voice workflow ready.

Increase traffic in controlled increments, review failed calls and compare observed spending with the budget. Keep a rollback path for prompt and model changes. Share measured results with methodology and limitations if you publish a case study.

This resource is published by Wixzel Voice. Product details come from the references below; evaluation recommendations are a proposed method, not benchmark results.

• Call errors and failure codes
• Webhook delivery and verification
• Safe retries with idempotency

Found an error? Send a correction.