Why does AI so convincingly lie to us?

Ever had chatGPT confidently tell you something completely wrong, but do it so convincingly that you start to question everything you know? As smart as AI is, it can be spectacularly good at talking nonsense.

In my first 'tuition' blog (here) I explained how Large Language Models (LLMs) work. Put simply, they predict the next token (or word), then the next, then the next, until they've constructed an answer.

This usually works pretty well due to the absolutely vast amount of training data it's been given previously - until it doesn't. It invents something completely fictional with sheer unadulterated confidence, provides an impressively detailed justification, cites a study that doesn't exist and then when you challenge it confidently informs you that you are, in fact, mistaken. These are generally known as 'hallucinations'.

The world's most confident b*ll-sh*tter

Let's start with a really important point: an LLM isn't fundamentally a 'truth machine'. It's an incredibly good guessing machine. Training means it gets extraordinarily good at producing responses that look like the sorts of things extremely clever people would say.

But the model doesn't consult a 'database of truth' once it's finished writing the sentence. It doesn't have a fool-proof internal 'alarm' that goes off once it's finished writing that says "what on earth have you just written? that is complete nonsense".

A good analogy is if I forced you to speak as intelligently as you could on a topic that you have absolutely no clue about - but you were absolutely not allowed to stop saying words. You must keep talking.

I bet you'd do a pretty impressive job.

This is fundamentally why an LLMs incorrect answers sound pretty much as convincing as its correct ones. It's chosen a bad token, that token then helps to inform the next token (thus compounding & propagating the error) - and before you know it it's trying to convince you that there are no letter 'E's in 'seventeen' (watch this for a laugh: link).

Grovelling apologies

This doesn't mean AI's are incapable of recognising mistakes after-the-fact. But this typically requires the user to call it out. It's then, again, up to the AI to decide whether to issue a grovelling apology and backtrack, or to double down and fight its corner. There's even a behaviour called 'sycophancy' where the AI can backtrack & apologise for an answer in which it was genuinely correct in the first place - instead deciding to agree with what the user is saying.

So why can't an AI simply say "I'm not sure?".

Generally in training these models there is a bias to ensuring a question gets answered. People like it when their questions are answered. Good things come to responsive LLMs.

And that's the stance that a lot of frontier models take - they accept that sometimes AIs will get things spectacularly wrong - but it's a small price to pay if it happens infrequently enough and it means their models are always responsive & answering anything the user wants. Make what you will of the ethical ramifications of that.

All of this shouldn't stop you from trusting your local friendly AI. But it should make you always vet & question what they're saying. When a hallucination happens in a harmless chatGPT conversation, it's usually not too big a deal.

But it gets a bit scary when agents are starting to be entrusted with more information, potentially sensitive, potentially important decisions - what's to say an AI can't hallucinate when it's summarising those meeting notes for you? Did you read & verify them...? 😄