When Your AI Doesn't Know, But Answers Anyway: The Hidden Cost of Confident Confabulation

AI hallucination isn’t a technical problem. It’s a trust problem. And we’re all already living inside it.

There is a particular moment in every conversation with an AI chatbot that most people have experienced but few have properly examined.

You ask a question. The AI responds immediately, fluently, with complete confidence. The answer is structured, plausible, and delivered in the authoritative tone of someone who knows what they’re talking about. You accept it. You move on. You build on it. You make decisions based on it.

And then, later, you discover it was wrong. Not wrong in the way a human expert might be wrong — wrong in the way a person who has never been to a city but gives you directions anyway is wrong. Confidently. Convincingly. Without a trace of hesitation or uncertainty.

This is hallucination. Not the kind where the system crashes or returns an error message. The kind where it fabricates information and presents it as fact, with the same fluency and conviction it uses for things it actually knows. And the latest data suggests the problem is far worse than most people realise.

What the numbers actually say

Artificial Analysis recently published the AA-Omniscience benchmark, which measures AI models on two dimensions: accuracy (how often they get the answer right) and non-hallucination rate (how often they admit they don’t know, rather than inventing an answer).

The results are sobering.

The best-performing model, Claude Fable 5, achieves a non-hallucination rate of 45%. That means 55% of the time it is uncertain about something, it fabricates an answer rather than admitting ignorance. And this is the best model on the market. Every other model tested performs worse.

At the other end of the spectrum, DeepSeek V3 Flash hallucinates 96% of the time when uncertain. Command A+, the most honest model in the benchmark, still fabricates 14% of the time.

Most of the models people actually use for content generation, research, and decision support sit somewhere in the 30-60% hallucination range. That means a third to half of the time these tools are generating information they are not confident about, they are presenting it as established fact.

Let that settle for a moment. The tools we are increasingly relying on to write our emails, summarise our research, answer our questions, and inform our decisions are confidently wrong about a significant percentage of what they tell us. And they never signal that they are uncertain.

Why this is not a technical problem

The instinctive response to hallucination data is to treat it as an engineering challenge. Build better models. Improve training data. Add guardrails. Develop detection tools.

All of that is happening. None of it addresses the real problem.

The real problem is not that AI models hallucinate. Every information source has a reliability threshold. Academic papers contain errors. Wikipedia has inaccuracies. Journalists get things wrong. Humans have always navigated an information landscape where not everything presented as true actually is.

The real problem is that we do not treat AI output the way we treat other uncertain information sources. We treat it differently. We treat it better. We extend to it a level of trust that we would never extend to a human stranger, a Wikipedia article, or a social media post.

And the reason we do this is not because we have evaluated the evidence and concluded that AI is reliable. It is because of how the information is delivered.

The psychology of misplaced trust

When a human being tells you something with visible uncertainty — hedging their language, qualifying their statements, pausing to think — your brain processes that uncertainty as a signal. It activates your critical evaluation systems. It says: this person is not sure, so I should verify before I rely on this.

When an AI tells you something with absolute confidence — clean sentences, authoritative structure, no hedging, no pause — your brain processes that confidence as a signal too. It says: this source knows what it is talking about.

This is not a rational evaluation. It is a cognitive shortcut. Behavioural scientists call it the authority bias: our tendency to assign greater credibility to information delivered with confidence, regardless of whether that confidence is warranted. It is the same bias that makes us trust a doctor who speaks decisively more than one who says “I’m not entirely sure, but I think…”

Dual Process Theory, one of the seven pillars of the STAR Framework, explains the mechanism. Our System 1 processing — the fast, intuitive, automatic mode of thinking — evaluates fluency and confidence as proxies for accuracy. A well-structured sentence feels more true than a hesitant one, even when the content is identical. AI output is almost always fluently structured. It almost always sounds confident. So System 1 accepts it, and System 2 — the slow, deliberate, analytical mode — never gets engaged to check.

The result is what researchers call automation bias: the tendency to favour computer-generated outputs over contradictory information, even when the human has reason to doubt the machine. Studies have shown that people will override their own correct judgements to accept an incorrect AI recommendation, simply because the machine presented its answer with certainty.

This is not stupidity. It is a design feature of human cognition that served us well for most of evolutionary history. In a world where confident speakers were usually confident because they had reason to be, trusting fluency was a reasonable heuristic. We no longer live in that world. We live in a world where machines can generate fluent, confident, authoritative-sounding text about things they have no knowledge of, at a scale no human could match.

The compounding problem

The danger of AI hallucination is not just that individual outputs are sometimes wrong. It is that the errors compound.

When a student asks an AI assistant “What is the best student accommodation in Leeds?” and the AI confidently recommends a property that does not exist, or quotes a price that was never accurate, or invents an amenity that was never built — that is not a single error. That is the beginning of a decision chain built on false premises.

The student visits the website of the recommended property. They compare it to alternatives. They make a booking decision based on information that was fabricated. They tell their friends. The fabricated information enters their social network’s shared knowledge base. It becomes, for all practical purposes, something that everyone “knows.”

This is the compounding problem of hallucination in an AI-mediated world. The error does not stay contained within the interaction where it was generated. It propagates. It becomes reference material for other queries. It enters the training data for future models. It becomes harder to correct with each repetition, because each repetition adds another layer of apparent confirmation.

In the language of Cognitive Bias Theory, this is confirmation bias operating at industrial scale. Once a piece of information is accepted as true — whether by a person or by a system that indexes person-level beliefs — subsequent encounters with that information are processed as confirmation rather than new claims requiring verification. The hallucinated fact becomes increasingly “true” with each repetition, not because evidence accumulates, but because the repetition itself creates the illusion of evidence.

What this means for brands

If you are a brand, the implications are immediate and uncomfortable.

The “Invisible First Click” framework, which I have written about in the context of AI-mediated discovery, argues that the moment of brand selection is increasingly happening before a human ever visits a website. AI assistants are forming consideration sets, making recommendations, and pre-selecting options before the prospective customer has any direct interaction with the brand.

If those AI recommendations are based on hallucinated information — fabricated reviews, invented pricing, non-existent amenities — then the brand’s first impression is being formed by a machine that is confidently wrong.

This is not a hypothetical. It is happening now. Ask any AI assistant to recommend student accommodation in a major UK city and compare its output to reality. The discrepancies are not minor. They are structural. Properties that do not exist. Prices that were never offered. Locations that are wrong. Amenities that were never built.

For the brands being recommended, this is a double-edged sword: a hallucinated recommendation might drive traffic, but it drives traffic based on expectations that cannot be met. The student who arrives expecting a rooftop terrace and finds a standard room does not blame the AI. They blame the brand.

For the brands being overlooked — because the AI did not know about them, or hallucinated a competitor into their space — the cost is invisibility. Not a penalty. Not a punishment. Just an absence from a consideration set that was formed without human agency.

The trust asymmetry

There is a deeper psychological dynamic at work that Appraisal Theory helps to explain.

When a human expert gives you wrong information — a doctor who misdiagnoses, a financial adviser who recommends a losing investment — your emotional response is shaped by how you appraise the situation. If you believe the expert acted in good faith, with the best available information, you are likely to feel disappointed but not betrayed. If you believe they were careless, or were not actually expert, the emotional response shifts to something closer to anger.

With AI, the appraisal is different. We do not attribute intention to the machine. We do not think it was careless or malicious. But we do attribute reliability to the system that deployed it. When an AI hallucinates, the emotional response lands not on the model but on the brand, platform, or organisation that put the AI in front of us.

This creates an asymmetry: the trust we extend to AI output is high (because of fluency and authority bias), but the trust we withdraw when that output proves false is directed at the human institution behind the AI (because we cannot be angry at a machine, so we are angry at whoever told us to trust it).

The consequence is that brands and organisations deploying AI tools are absorbing a trust liability they may not fully understand. Every confident, fluent, authoritative-sounding AI output that turns out to be wrong does not just fail to inform. It actively erodes trust in the institution that delivered it.

What to do about it

The practical response is not to stop using AI. That ship has sailed, and the tools are genuinely useful when they work.

The practical response is to treat AI output the way you would treat a first draft from a talented but unreliable junior colleague: useful as a starting point, dangerous as a final answer.

Concretely:

Verify before you publish. If AI generates content that will be seen by customers, students, or stakeholders, check the facts. Not the tone, not the structure, not the plausibility — the facts. Does this property exist? Is this price accurate? Was this study actually conducted?

Choose your models deliberately. The AA-Omniscience benchmark shows that hallucination rates vary dramatically across models. Model selection is a risk management decision, not just a cost decision. A cheaper model that hallucinates 60% of the time will cost you more in credibility than a more expensive one that hallucinates 15%.

Build uncertainty into your AI interfaces. If you are deploying AI in customer-facing contexts, design for uncertainty. Give the AI permission to say “I don’t know.” Show confidence indicators. Make it clear that the output is generated, not verified.

Invest in being accurately represented. If AI engines are forming consideration sets for your products and services, invest in making sure those engines have access to accurate, structured, up-to-date information about your brand. Schema markup, llms.txt files, structured data, and consistent citation sources are not SEO tactics. They are trust infrastructure.

The uncomfortable truth

We are living inside a trust experiment at civilisational scale.

We have deployed systems that generate confident, fluent, authoritative-sounding text about everything, and we have made those systems available to billions of people. We have done this knowing that those systems fabricate information a significant percentage of the time. We have done this knowing that human psychology is poorly equipped to distinguish between AI outputs that are grounded in knowledge and AI outputs that are grounded in nothing at all.

The consequences are not dramatic. They are not catastrophic in the way that a bridge collapse or a plane crash is catastrophic. They are quiet. They are cumulative. They are the slow erosion of the shared informational foundation that civilised decision-making depends on.

The question is not whether AI will get better. It will. The question is what happens to trust in the meantime — and whether we will still have enough of it left to make the transition worthwhile.

David Chadderton is the creator of the STAR Framework and the author of three books: The STAR Framework: Rewriting the Rules of Consumer Engagement (NYC Big Book Award 2025), The STAR Operating System: Decode Mindset, Understand Motivation, Transform Human Behaviour, and Dear Algorithm, It’s Not Me, It’s You. He spent his twenties and thirties as a military aviator and instructor, studying how people make decisions when the stakes are highest. He now applies those principles as a Chief Marketing Officer, bringing behavioural science to performance marketing at scale. He writes about human behaviour, AI, and the psychology of decision-making on The Unoptimised Human.