Most voice AI evaluations fail before the demo starts, because the buyer walks in without a list of questions that separate a really good product from a well run demo. This guide is that list: twelve questions you can ask your solution providers, why each one matters, and what a strong answer sounds like.

1. Is the agent conversational or a flowchart?

Deterministic bots walk a tree: ask, capture, branch. They break the moment a caller says two things at once. Ask the vendor to demo a caller who interrupts, changes their mind mid-order, and gives information out of sequence. A strong answer is a live demo that survives all three, and an architecture explanation that doesn't include the phrase "dialog flow."

2. Can it complete tasks, or just talk?

"Agentic" means the system takes actions: placing orders, updating records, booking appointments, processing claims. Ask exactly which systems the agent will write to on day one, and what a completed task looks like in your stack. A strong answer names APIs and shows a task finishing with no human touch.

3. What happens on day one if our CRM has no open API?

Some deployments stall for months waiting on integration work. Get specific: what can launch with zero integration? Inbound flows often can, because the caller supplies the information the agent needs. One of our home-services customers went live while its CRM vendor kept the API closed, and Omaha Steaks exposed APIs from a homegrown AS400 stack in about a week with our engineers alongside theirs.

4. What's the realistic time to first live call?

Six months is an answer you should question. Eight days is an answer you should verify. Omaha Steaks signed a proof of concept on June 2 and took first live calls June 10; ask any vendor for their equivalent story with dates and a reference customer you can call.

5. How fast does it respond, and how does it know when to speak?

Voice tolerates far less delay than chat. Response latency around 800 milliseconds is where conversations start feeling natural; beyond that, callers talk over the agent and loops multiply. Ask what the vendor's median and p95 response times are on real calls, and whether the turn-taking and voice-activity models are their own.

6. Who owns the model stack?

If the vendor is a thin wrapper on someone else's transcription, inference, and voice APIs, their latency, cost, and peak-season capacity are all someone else's decisions. Ask which pieces they host and auto-scale themselves. If they rent everything, ask how they handled last December's peak, and pay attention to whose engineers did the work.

7. How does it hand off to a human?

Every agent needs to escalate sometimes. The difference is whether the context survives to sets the human up for success. A strong answer includes a warm transfer, a screen pop showing the human agent everything gathered so far, and partial-containment reporting so you get credit for the half of the call the AI already did. The customer should never repeat themselves.

8. What does the analytics layer show me?

You want intent categorization across 100% of calls, containment and conversion by use case, and alerting on anomalies. Mid-peak last December, our monitoring caught a spike in callers saying the add-to-cart button was broken on the Omaha Steaks website; their web team shipped the fix before the QA vendor had transcribed a single call. Ask the vendor for the story their analytics caught that nobody was looking for.

9. How do we test changes safely?

Prompt edits, voice changes, and script experiments need versioning, a sandbox, and ideally A/B testing on live traffic. Ask how long a change takes to ship, and how you'd measure whether it helped. "Hours" is a good answer. "Submit a ticket" means every experiment waits in the vendor's queue.

10. What's the security and compliance posture?

SOC 2 Type II at minimum. HIPAA if health information can come up. Real-time PII redaction in transcripts. Ask where audio and transcripts are stored, for how long, and who can access them. A strong vendor answers without checking with legal.

11. What does it cost, and what makes the invoice grow?

Per-resolution pricing puts the vendor in charge of deciding what "resolved" means. Usage pricing is more predictable but demands volume estimates. Whatever the model, ask for the invoice math on your actual call volume, peak month included, and get the overage terms in writing.

12. Who works with us after launch?

The deployments that hit 70% containment and hold it are the ones where vendor engineers meet with the customer team weekly and treat the roadmap as shared. Ask who your named engineers are, how often you'll meet, and what the last customer shipped with them in month six. If the answer is "documentation and a support portal," plan accordingly.

Scoring the answers

No vendor aces all twelve. Weight the ones tied to your economics: if your inbound line sells, questions 1, 2, 5, and 9 decide revenue. If you're regulated, question 10 is a gate. If your stack is homegrown, question 3 will decide your timeline.

Make every vendor answer the same twelve questions in the same order. By the third demo, you will be able to sense which ones were rehearsed.