An ai chatbot testing checklist is a repeatable list of checks that shows whether a support bot is ready for real customers. It covers correct answers, safety, escalation, privacy, and the mobile experience. Without a checklist, testing becomes a few random chats and a hope that nothing breaks. With a checklist, you can prove what you tested, what passed, and what still needs fixing.
This guide is for beginners and small teams. You do not need special software to start. You need a set of test questions, a simple scoring sheet, and time to run the tests before launch and after every change. The checklist below is built for that workflow.
This ai chatbot testing checklist covers scenarios, answer quality, safety, mobile and re-testing, so your bot is checked before customers meet it.
Why Test Before Customers Do
Customers will test your bot whether you do or not. They will ask vague questions, misspell product names, ask two things at once, and ask for a person when they are frustrated. If the bot has not been tested on those cases, it will fail in front of a real customer, and that customer may not come back.
Testing also protects your team. A bot that gives wrong refund dates or invents a policy creates extra work for agents, who must correct the answer and calm the customer. A bot that asks for a password or shares the wrong order details creates a privacy problem. A short test run before launch catches many of those problems while they are still cheap to fix.
An ai chatbot testing checklist matters because customers remember the one wrong answer far longer than the ten right ones.
AI Chatbot Testing Checklist: Build a Scenario Set
A scenario set is a list of test conversations, each with a question, the expected answer, and the source for that answer. Build it from real customer questions first. Look at your inbox, your help desk, and your past chats. Pick the questions that appear most often, the questions that cause the most confusion, and the questions where a wrong answer would cause the most harm.
Include several types of questions. Start with common questions that have a clear source, such as delivery times, return windows, and setup steps. Add exception questions, such as a return after the window or a delivery to a different address. Add vague questions, such as "it is not working" or "where is my order?" Add questions with misspellings, short forms, and everyday phrasing. Add questions with two topics in one message, and questions that are outside your approved topics.
Also include safety and privacy cases. Test what the bot does when a customer asks for a password reset, shares a card number, or asks for another customer's order details. Test what it does when a customer is upset or asks for a manager. Test what it does when it has no source for the question. Those cases are easy to miss in a test set built only from common questions, but they are the cases that matter most.
Aim for thirty to fifty scenarios for a first launch. That is enough to cover the main topics without taking a whole day to run. Write each scenario in plain language, with the expected answer and the source article it should come from. If a scenario has no source, write that down too, and note that the expected result is for the bot to admit it does not know and offer a person.
Keep the scenario set in one document or sheet that the whole team can use. Each row should have the scenario number, the test message, the expected answer, the source, and a space for the result. When you add a new product or policy, add scenarios for it at the same time. When a real chat reveals a failure, add that case to the set so it is tested again next time.
For the questions to test with, read our guide to writing FAQ answers for an AI chatbot.
Most testing features live in the chatbot plugin, so compare them in the official WordPress plugin directory.
Score Answer Quality
Scoring turns a feeling ("the bot did well") into evidence. Use a simple score for each test so anyone on the team can apply it the same way.
For questions with a source, use four scores. Correct means the facts match the source and the answer is complete. Partly correct means the main fact is right but a detail is missing or wrong. Wrong means a key fact does not match the source. No answer means the bot said it does not know when the source had the answer. For questions with no source, the correct result is different: the bot should say it does not know and offer a person or a useful next step. A guessed answer for a no source question is a fail, even if the guess sounds plausible.
Score more than facts. Check tone, clarity, and next steps. Is the answer easy to understand? Does it use plain language? Does it offer a useful next step, such as a link to a help article, a follow up question, or a person? Does it stay in bounds, or does it promise something it cannot do? A factually correct answer that is confusing or rude is still a problem.
Record the results in your test sheet. Note the date, the scenario number, the bot reply, and the score. Add a short note for any fail, stating what went wrong and what needs to change. That note is the start of your fix list. Do not delete old test runs. Keep them so you can compare this month with last month and see whether the bot is improving.
What Counts as a Pass
A pass is not a perfect score on every question. It is a clear bar that you set before the run and apply the same way each time. For a small support bot, a useful bar is: no wrong answers on money, safety, or account topics; no guessed answers on no source questions; and at least nine out of ten common questions scored correct. You can set a stricter bar once the bot is stable, but start with a bar your team can meet and defend.
If a test fails on a money or safety topic, do not launch until it is fixed. If several common questions are partly correct, fix the source articles before you change the prompt. Partly correct answers often come from a source that is vague or incomplete, not from the bot itself. Fix the source, run the test again, and check whether the score improves.
Your ai chatbot testing checklist should score answers on accuracy, tone and whether the bot cited a real source.
For the metric behind quality scoring, read how to measure AI chatbot resolution rate.
Test Safety, Escalation, and Privacy
Accuracy is only one part of testing. A bot can give correct answers and still fail on safety, escalation, or privacy. Test those areas with the same care.
For safety, test questions about refunds, charges, account access, and any topic where a wrong answer could cause harm. The bot should not approve a refund, change a price, or give advice on safety, health, or legal rights beyond your approved articles. It should escalate those topics to a person. Test that it does so, even when the customer pushes for an answer.
For escalation, test the handoff itself. Ask for a person in plain language, ask for a manager, and ask for a person after two failed answers. Check that the bot offers a person quickly, uses a calm and clear message, and passes the transcript and key details to the agent. Test the after hours path too. Send a message outside business hours and check that the bot says the team is offline, saves the message, and gives a clear reply window.
For privacy, test that the bot never asks for a password, a full card number, or other sensitive details. Test what happens if a customer shares sensitive information by mistake. The bot should not repeat it back in full, and your team should have a rule for removing it from the transcript. Check that test cases and shared examples do not contain real customer data. Use made up order numbers and email addresses in tests, and give access to test results only to team members who need them. Store API keys securely, never in a test file or a prompt that is shared widely.
A complete ai chatbot testing checklist includes safety — the bot must refuse personal data requests and hand off sensitive cases.
Test the Mobile Experience
Many customers will meet your bot on a phone. A test that only runs on a desktop misses the experience most customers will have. Run the core scenarios on a mobile phone as well as on a desktop.
Check that the chat button is easy to find and tap, that the chat window opens without covering the whole page, and that messages are easy to read on a small screen. Check that buttons and quick replies are large enough to tap without hitting the wrong one. Check that long answers are still readable, and that the bot does not ask for a long form inside a small chat window. A form with many fields is hard to complete on a phone, and many customers will abandon it.
Check typing as well. Customers on mobile type shorter messages and use more short forms. Run a few scenarios with short, misspelled, or abbreviated messages and see whether the bot still understands the intent. Check that the bot does not send several long messages in a row, which can feel overwhelming on a small screen. Short, clear replies work better on mobile.
Most visitors chat from a phone, so your ai chatbot testing checklist must include a full mobile test on a real device.
Tools like Chatbase let you preview the bot on mobile before launch — see Chatbase’s official site for how testing works there.
Re-Test After Updates
Re-testing after updates is where most small teams slip. A change that looks small, such as a new help article or a reworded prompt, can change how the bot answers an old question. Without a re-test, you will not know until a customer reports the problem.
Set a simple rule: no change goes live without running the core test set first. The core set is a smaller version of your full set, with ten to fifteen scenarios that cover the most common questions, the riskiest topics, and the cases that have failed before. Run the core set after every change, even a small one. Run the full set once a month and after a major change, such as a new product, a new policy, or a new bot version.
Keep a change log. For each change, note the date, what changed, why it changed, and the test result. For example: "Updated return article to new window, ran core set, all passed." A change log prevents the same problem from returning and helps a new team member understand why a rule exists.
When a re-test fails, stop and fix the cause before publishing. Do not publish a change that breaks a previously correct answer and plan to fix it later. Later usually means after a customer has already seen the wrong answer. Fix the source, the prompt, or the routing rule, run the test again, and only publish when the core set passes.
Related reading in this set: How to Reduce AI Chatbot Hallucinations in Support, How to Set Up Human Handoff for an AI Support Bot, and How to Handle Questions Your AI Chatbot Cannot Answer.
Every ai chatbot testing checklist needs a re-test step, because a new page or a plugin update can quietly break answers.
A Ready to Use Checklist
Use this checklist for each test run. Copy it into your test sheet and tick each item as you complete it.
- Scenario set covers common, exception, vague, and out of scope questions.
- Each scenario has an expected answer and a source, or a note that there is no source.
- Safety cases tested: refunds, charges, account access, and risky topics.
- Escalation tested: direct request, request after failed answers, and after hours.
- Privacy tested: bot never asks for passwords or full card numbers.
- Mobile tested: chat opens, buttons are tappable, answers are readable.
- Answers scored: correct, partly correct, wrong, or no answer, with notes.
- No wrong answers on money, safety, or account topics.
- Core set re-run after the last change, with a pass recorded.
- Failures logged with a cause and a fix, and re-tested after the fix.
| Area | Test | Pass looks like |
|---|---|---|
| Accuracy | Common questions | Facts match the source |
| Safety | Refund and account cases | Bot escalates, does not approve |
| Escalation | Ask for a person | Quick, calm handoff with context |
| Privacy | Sensitive data cases | Bot never asks for passwords |
| Mobile | Core scenarios on a phone | Readable, tappable, no long forms |
| Regression | Core set after a change | No previously correct answer breaks |
Keep this ai chatbot testing checklist next to the bot settings and run it after every change, not just at launch.
Limitations
A checklist cannot prove that a bot will never make a mistake. It can only show that the bot passed the cases you tested, on the day you tested them. New questions, new products, and new phrasing will always appear. That is why the checklist includes a monthly full run and a re-test after every change, and why the bot should always have a clear path to a person.
Do not treat a pass as a promise of perfection. Treat it as evidence that the bot is ready, with a human team watching real chats. A bot tested regularly will stay useful. A bot tested once and left alone will drift.
Frequently Asked Questions
How many test questions do I need?
Start with thirty to fifty test questions for a first launch. That is enough to cover common questions, exceptions, vague messages, and out of scope cases without taking a whole day to run. Keep a smaller core set of ten to fifteen questions for quick re-tests after each change. Add new questions when a real chat reveals a failure, when you add a product or policy, or when a topic starts to appear often in your inbox. A test set that grows from real customer questions is more useful than a large set built from guesses.
What is a pass?
A pass is a clear bar you set before the run. For a small support bot, a useful bar is no wrong answers on money, safety, or account topics, no guessed answers on questions with no source, and at least nine out of ten common questions scored correct. A pass also means escalation, privacy, and mobile checks passed. If a money or safety case fails, do not launch until it is fixed. Record the bar, the scores, and the date so the next run can be compared fairly.
Conclusion
An ai chatbot testing checklist turns testing from a hope into a routine. Build a scenario set from real customer questions, score answers with a simple and consistent rule, test safety, escalation, privacy, and mobile experience, and re-test after every update. Keep a log of results and fixes, and add new cases when real chats reveal a failure.
Your next step is simple: write thirty test questions from your inbox and past chats, run them through the bot this week, and fix the failures before you publish. That one run will tell you more about your bot than a month of guessing, and it will give you a baseline to improve from.