AI Hallucination

Also Known As

TL;DR

Explanation

LLMs predict the most probable next token given context — they do not retrieve facts from a database. When the model has insufficient training data for a query, it generates plausible-sounding completions that may be fabricated. Hallucinations are more common with: obscure topics, recent events after the training cutoff, precise numerical data, citations and URLs, and code for niche APIs. Mitigations: RAG (provide source documents), constrained output (JSON schema), chain-of-thought, and output validation.

Diagram

flowchart TD
    PROMPT[User prompt] --> LLM[LLM generates response]
    LLM -->|confident but wrong| HALL[Hallucination<br/>plausible-sounding falsehood]
    subgraph Causes
        TRAIN[Not in training data]
        AMBIG[Ambiguous prompt]
        LIMIT[Knowledge cutoff]
    end
    HALL --> TRAIN & AMBIG & LIMIT
    subgraph Mitigations
        RAG[RAG: ground in real documents]
        CITE[Require citations]
        TEMP[Lower temperature<br/>less creative = more factual]
        VERIFY[Human verification<br/>for critical outputs]
    end
    HALL -.->|reduce with| RAG & CITE & TEMP & VERIFY
style HALL fill:#f85149,color:#fff
style RAG fill:#238636,color:#fff
style VERIFY fill:#238636,color:#fff

Watch Out

Common Misconception

Why It Matters

Common Mistakes

• Treating LLM output as ground truth without validation — always verify factual claims from external sources.
• Using LLMs for real-time data without RAG — the model's training cutoff means recent facts are fabricated.
• No output validation for structured data — a hallucinated JSON field name causes a production crash.
• Prompting for citations without providing source documents — the model fabricates plausible-looking but fake references.

Avoid When

• Trusting LLM output for factual, legal, medical, or financial claims without verification.
• Using a model without retrieval augmentation for domain-specific or time-sensitive facts.
• Presenting LLM output directly to users without human review in high-stakes contexts.
• Relying on a model to accurately cite sources — citation hallucination is common even in capable models.

When To Use

• Always implement retrieval-augmented generation (RAG) when facts matter — ground the model in verified sources.
• Use structured output and validation to catch impossible or malformed responses.
• Add a verification step that checks generated claims against a trusted knowledge base.
• Set system prompts that instruct the model to say 'I don't know' rather than fabricate.

Code Examples

// Trusting LLM output without validation:
$answer = $llm->complete('What is the current PHP 8.4 memory_limit default?');
echo $answer; // May be completely fabricated — training data may be outdated
// Worse: generating code with fabricated function names that don't exist

// RAG with source citation — grounded, verifiable:
$docs = $vectorDb->search('PHP 8.4 memory_limit default', topK: 3);
$context = implode("\n", array_column($docs, 'content'));
$prompt = "Answer using ONLY the provided documentation. If unsure, say so.\n\nDocs:\n$context\n\nQ: {$query}";
$answer = $llm->complete($prompt);
// Also validate structured outputs with JSON Schema

Tags

🧠 Logic Engine

References

• https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)