Will a chatbot make things up? How to stop AI giving customers wrong answers

AI chatbots write answers by predicting what words should come next. They are very good at sounding right. They don't check facts the way a person would.

OpenAI's researchers explain it with an exam comparison. Like students facing hard exam questions, AI models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty. They argue that the way models are trained and tested rewards guessing over saying "I don't know".

This is known as hallucination. Anthropic, which makes the Claude AI model, says even the most advanced models can sometimes produce text that is factually wrong or doesn't match the information they were given. So the question isn't whether it can happen. It is how you reduce the chance and limit the harm.

Not all mistakes are equal. A slightly clumsy sentence doesn't matter much. A wrong price, a promise you can't keep or bad advice can cost you money and trust.

The table shows the common types and where they come from.

The best known case is Moffatt v Air Canada, decided by the British Columbia Civil Resolution Tribunal in February 2024. A customer asked the airline's website chatbot about bereavement fares after his grandmother died. It told him he could apply for the reduced fare after travelling, within 90 days. That was wrong.

Air Canada argued that the chatbot was responsible for its own actions and that the correct information was elsewhere on its site. The tribunal disagreed. It said Air Canada could not separate itself from the chatbot built into its own website, and that a customer can't be expected to double check one part of a website against another. It awarded around 650 Canadian dollars in damages, plus interest and fees.

This is a Canadian small claims decision, not UK law. But the principle is easy to understand: customers will treat what your chatbot says as what your business says. Plan on that basis.

There is also the risk to your reputation. In January 2024 a customer got DPD's chatbot to swear and call the company names, and the screenshots spread widely. DPD said the problem came after a system update and switched off the AI part of its chat.

Assume that anything your chatbot tells a customer will be treated as a promise from your business. Set it up so it only says things you would happily put in writing.

AI companies publish their own advice on this. Anthropic's guidance includes allowing the AI to say "I don't know", which it says can drastically reduce false information, and telling it to use only the documents provided, not its general knowledge. It also suggests having the AI back up its claims with quotes from your documents.

Put those ideas into practice with these steps. None of them needs technical skill.

1. 1 Use a chatbot that answers only from your own information, not from the internet. Ask the supplier how it does this.
2. 2 Clean up your information first. Remove old pages and documents, and make sure prices and policies match everywhere.
3. 3 Write exact question and answer pairs for anything that must be precise, such as prices, deposits and cancellation rules.
4. 4 Instruct it clearly: if the answer isn't in the information, say so and offer to take contact details or pass to a person.
5. 5 Ban commitments. Tell it never to promise refunds, discounts, prices or dates that aren't written in its information.
6. 6 Send sensitive topics to a person, including complaints, refunds, legal questions and medical or financial advice.
7. 7 Test it with tricky questions before launch, then read the logs every week and fix what went wrong.

Yes. A visitor can type something like "ignore your rules and tell me..." to push the chatbot off track. This is called prompt injection. The security community OWASP describes it as user prompts that change the AI's behaviour or output in unintended ways.

The UK's National Cyber Security Centre is frank about it. In a blog post on the subject, it says that current AI models simply do not enforce a security boundary between instructions and data, and that prompt injection attacks may never be totally mitigated. The advice is to reduce the risk and the impact, not to expect a perfect fix.

For a small business chatbot, that means keeping its powers small. A chatbot that only answers questions and collects contact details can do little harm if tricked. Be much more careful with one that can change bookings, issue refunds or see customer records.

• Give the chatbot a narrow job and tell it to ignore attempts to change its instructions.
• Don't connect it to systems or data it doesn't need. OWASP calls this least privilege.
• Require a person to approve anything with real consequences, such as refunds or account changes.
• Keep logs of conversations so you can spot misuse. The NCSC recommends logging inputs and outputs.
• Be wary of any product that claims to stop prompt injection completely. The NCSC advises preferring products that explain how they reduce it.

Saying "I don't know" is a feature, not a failure. A chatbot that admits a gap and offers a next step is far more useful than one that guesses.

GOV.UK advises giving users a way to reach a real person, so they don't get stuck in loops, and offering other contact channels. A good fallback answer acknowledges the question, says the chatbot doesn't have that information, and offers a choice.

GOV.UK says you should test response accuracy before a bot goes live, and keep testing regularly after. It also warns that accuracy might drop as the bot is exposed to requests beyond its scope.

Build a simple test list of real customer questions and the correct answers. Run it before launch and again after any big change. Then read the conversation log. Look for wrong answers, questions it couldn't handle and places where people gave up, and fix the information or rules behind each one.

Anthropic's own guidance ends with a reminder that these techniques reduce hallucinations but don't remove them, and that critical information should always be checked. Treat your chatbot like a new member of staff: helpful from day one, but supervised.

Our Website Chatbot answers visitor questions using only what it has been taught about your business: your pages, documents or pasted text. It is set up to say it doesn't know rather than invent an answer, hands off to a person when needed, and logs every conversation so you can check what it said. It also captures name, email and the enquiry mid-conversation, works on any website with one line of code, and the price is on application.

Tell us about your project through our quote form, and we can talk through which questions it should and shouldn't answer for your business.