Module 3.1 · Topic 3
Why AI Gets Things Wrong
Bottom Line Up Front: Generative AI fails in predictable, systematic ways: hallucinating facts, confusing pattern-matching tasks with knowledge tasks, running into knowledge cutoffs, and producing confident-sounding…
3.1 Understanding Hallucinations: Confident Fabrication
When you ask a fast/chat model for a 2020 ruling and that specific case doesn't exist in training data, the model provides details (case numbers, judge names) that sound authentic and may even resemble real cases, but are mere fabrications nonetheless. On its own, the model has no way to distinguish fabrication from real patterns. Both feel identical. It outputs the most likely continuation, which may be fabricated. Hallucinations are systematic outputs of a pattern-matching system with no grounding in facts and no linked database to enable validation. When patterns include the true answer, it's still just statistics, it just happens to be have been statistics that correspond with truth.
3.2 The Difference Between Knowledge and Pattern Matching
| Task Type | Pattern-Matching Tasks (More Reliable) | Knowledge Tasks (Higher Hallucination Risk) |
|---|---|---|
| Example 1 | Reviewing a contract for clause consistency — extracting structure from text you provide | Researching whether a specific statute was amended in 2025 — requiring factual knowledge AI may not have |
| Example 2 | Summarizing a document's main arguments — recognizing rhetorical patterns in provided text | Identifying the current legal standard for a novel fact pattern — requiring synthesis of case law |
| Example 3 | Rewriting a paragraph for clarity — understanding grammar and style patterns | Citing a specific case holding — requiring the AI to "recall" accurate information |
| Why the difference? | All the information is in your prompt. AI doesn't need to retrieve facts — it needs to recognize patterns within provided text. | The AI must retrieve facts from its training data. If facts are rare, contradictory, or missing, the model guesses and hallucinates. |
3.3 Training Data Boundaries and Knowledge Cutoffs
Logic behind this approach:
Testing the risk of reliance on model outputs beyond knowledge cutoffs is a fairly simple task. Ask about recent factual events to identify what the model can answer. Calibrate verification effort: post-cutoff events need external research; older events may be in training data.
Sample prompt:
What to expect:
A good response provides factual answers with appropriate confidence or acknowledges uncertainty. The AI should not guess about post-cutoff events, but it still might. Use responses to identify which topics need external research.
3.4 Recognizing When AI Is Guessing vs. Knowing
Probably Reliable
- Document-specific details: AI citing or summarizing documents you provided is likely accurate because it will be central to its context and therefore weigh heavily in text prediction.
- Pattern recognition: AI identifying structures is more reliable than factual recall.
- Conditional language: "Suggests" or "appears" indicates patterns, not facts. Safer than definitive claims.
Probably Guessing
- Unsourced citations: Verify before relying. Hallucination zone.
- Vague references: "Courts generally hold..." without citing case suggests pattern-matching.
- Absolute statements: "This is a violation" is higher-risk than "This may violate."