How to test an AI receptionist before it answers a real customer

Why should you test an AI receptionist before going live?

Because the demo is not the product. A vendor's demo line is tuned to show the best case. Your callers bring accents, bad connections, weird requests and interruptions. The only way to know how an agent handles your phones is to call it yourself, on your number, with your scenarios.

The good news: you do not need weeks. Twelve scripted calls, done in one or two sessions, will surface most of what matters — how it greets, how it books, how it fails, and how it hands off. We built this protocol from the scenarios that break AI receptionists most often, and from the trial terms vendors actually publish.

One note on us: customerbot.io will publish its own test-call results for each vendor as we complete them. We have not made those calls yet, so nothing in this guide is a claim about how any specific vendor performed. This is the protocol we will use, and you can use it too.

What do you need before the first test call?

Set up first, or your results will mislead you.

• Configure the agent fully. Connect your calendar, load your FAQs, set your transfer number and business hours. Testing a half-configured agent tells you nothing.
• Write down your answers. For every scenario below, note what the correct outcome is before you call. 'Good' is only measurable against an expectation.
• Use a real phone line. Call from a mobile, then from a different phone with background noise. Do not test only from a quiet desk.
• Get a log sheet ready. Every call gets a row: scenario, what you expected, what happened, score.
• Check what the trial meters. Some vendors count test calls or minutes against your plan. Know that before you burn 12 calls (more on trial terms below).

The 12-call test script

Run all twelve. Each row: the scenario, what good looks like, and what to log.

Calls 1-4: the basics

1. After-hours call. Call outside your stated business hours. Good: the agent answers, says you are closed, offers to take a message or book a slot, and does not promise a callback time you cannot keep. Log: did it state hours correctly, and what did it offer?
2. Wrong number. Say you were trying to reach a different business. Good: a short, polite close, no long script, no attempt to book. Log: how many seconds before it ended the call cleanly.
3. Urgent request. Say it is an emergency (a burst pipe, a same-day legal deadline — whatever fits your trade). Good: the agent recognizes urgency, follows your escalation rule, and either transfers or flags the call immediately. Log: whether the escalation path fired, and how fast.
4. Book a slot. Ask for an appointment. Good: it checks your real calendar, offers times that are actually free, books one, and confirms by text or email. Log: whether the booking appeared on your calendar, and whether the confirmation arrived.

Calls 5-8: the stress tests

5. Reschedule. Call back and move the appointment you just made. Good: it finds the existing booking, changes it, and confirms the new time. Log: whether the old slot was freed on your calendar.
6. Caller interrupts. Mid-sentence, cut the agent off and change the subject. Good: it stops talking, listens, and follows the new topic without restarting its script. Log: whether it talked over you or recovered.
7. Background noise. Call from a street, a car, or next to a radio. Good: it still captures the key details (name, number, request) accurately. Log: transcription accuracy on the call summary.
8. Ask for a human. Say 'I want to talk to a person.' Good: it follows your configured handoff — transfer, message, or callback — without arguing. Log: which handoff fired and whether the transfer reached the right number.

Calls 9-12: the edge cases

9. Long address. Dictate a long, multi-part address with an apartment number and a confusing street name. Good: it repeats the address back correctly or confirms it by text. Log: character-level accuracy in the summary.
10. Pricing not in the knowledge base. Ask about a price you deliberately did not configure. Good: it says it does not know and offers to take a message or connect you — it must not invent a number. Log: exactly what it said. A hallucinated price is an automatic fail.
11. Spam call. Have a friend run a robocall-style pitch or an obvious sales script. Good: the agent ends it quickly and, ideally, does not count it against your allowance. Log: call duration and whether it hit your quota.
12. Second language. If the vendor claims multilingual support, call in that language. Good: it detects the language and continues fluently, or switches when you do. Log: whether it stayed in-language for the whole call. Skip this call only if the vendor makes no language claim.

How do you score the results?

Score each call 0-2: 0 = failed, 1 = partial, 2 = passed. Maximum 24 points. We would treat anything under 18 as not ready for live callers, and any single 0 on calls 3, 8 or 10 (urgent, human handoff, hallucinated pricing) as a fail regardless of total.

Double-weight the four calls that cost you money or trust when they fail. Then re-run any failed call after adjusting the configuration — a failed call that a settings change fixes is a setup issue, not a product flaw.

Trial terms to check before you start

Your 12 calls will consume something — minutes, calls, or a trial clock. Here is what the vendors we track publish:

• Goodcall offers a free demo agent: build it, test it, hear it answer, and pay only when you connect a number. Note that AI test-run usage is metered by plan allowance and topped up $5 at a time once you are on a paid plan.
• Rosie has a 7-day free trial with all features, cancel anytime. Plans are minute-based (250 minutes a month at $49/month), so budget your test calls against the clock.
• Upfirst offers a 14-day free trial with full access to every feature in the selected plan, no credit card required. Plans count calls (30 calls on the $24.95/month Starter), and extra calls bill per call.
• Smith.ai AI Receptionist has a free forever plan: $0/mo with 25 live calls a month, though a credit card is required at signup. Every answered call counts toward quota by default, with spam removals capped at 10% per cycle — relevant for your spam test.
• Dialzara offers a 7-day free trial on every plan, no contract. Billing is per minute ($29/month with 60 minutes on Business Lite), so keep test calls short.
• My AI Front Desk advertises a 7-day free trial on Business-in-a-Box. Its free-plan messaging conflicts — the homepage FAQ mentions a free plan with limited usage while the pricing page says there is no permanently free plan — so confirm before you rely on one.

Two checks before you dial: whether the trial needs a credit card, and whether test calls consume the same allowance as live calls.

What to do after the 12 calls

Total the scores. If the agent passed, run one more round: have two or three people who were not involved in setup call it cold, unscripted. Fresh callers find the gaps your script missed.

If it failed, separate setup failures from product failures. Wrong hours, missing calendar connection, no transfer number — those are configuration. Hallucinated prices, inability to handle interruptions, broken language switching — those are the product, and no settings menu will fix them quickly.

Then compare what you saw against what you would pay. Our AI receptionist cost guide breaks down per-minute, per-call and flat-monthly billing, and the AI receptionist vs answering service comparison covers the human alternative if no agent clears your bar. When you are ready to shortlist, the AI receptionist directory lists every vendor we track, and get matched narrows it to your call volume and budget.