2.4 Strengths, Limits, and Hallucination

Trusting AI too little can waste useful leverage. Trusting it too much ships the mistake. This lesson calibrates you: where models often help, where they reliably stumble, why they produce plausible falsehoods — and the verification habits that let you move responsibly.

What you will learn

• Map the terrain: tasks where models excel vs. tasks where they quietly fail.
• Explain hallucination mechanically — why it happens and why it looks so confident.
• Apply a right-sized verification habit for different stakes.

Builder principle

Trust is not the goal. Calibration is. Know exactly how much to check, and checking becomes fast.

Where models are genuinely strong

• Transformation — rewrite, summarize, translate, reformat, convert. Often strong because the source is in context, but omissions, mistranslations, changed emphasis, and formatting loss still require review.
• Drafting — first versions of pages, plans, emails, and code. A draft can reduce blank-page work, but the useful starting point varies and still needs review.
• Code — the standout domain. Code has tight patterns and a built-in truth test (run it), which is exactly why agents work so well: the verify step is mechanical.
• Breadth on demand — instant working knowledge of almost any topic, invaluable at the "what should I even be asking?" stage.
• Repeatable execution — agents can apply the same procedure across many files without ordinary human fatigue. Attention and context effects still make quality uneven, so sample throughout the run rather than checking only the first result.

Where models reliably stumble

• Fresh facts. Training data has a cutoff; the model doesn't know last week unless a tool (web search, your files) puts it in context.
• Precise arithmetic and counting. Pattern-predictors, not calculators. Good agents know this about themselves and reach for code to compute — a habit you should reinforce, not fight.
• Niche specifics. The 30-person subfield, your company's internal API, the plugin with 12 users. Thin training signal → confident guessing. This is hallucination's favorite neighborhood.
• Knowing what they don't know. The model has no reliable internal flag for "I'm unsure here." Fluency stays constant while accuracy varies. That asymmetry is the entire reason verification exists.
• Very long-horizon coherence. Over marathon sessions, drift happens (Lesson 2.2). Scoped tasks, fresh sessions, and inspected project memory reduce drift; none guarantees perfect attention.

Hallucination, mechanically

Why does a model invent a book title, a court case, a Python package?

Recall Lesson 2.1: the model generates plausible next tokens from learned patterns and current context. That process can produce truth or a truth-shaped error — a realistic title, a properly formatted but nonexistent citation, or a package name that follows conventions. Fluent form is not evidence that the underlying claim was retrieved or checked.

Two consequences:

1. Confidence tells you nothing. The tone is identical either way. Judge claims by checkability, never by vibe.
2. Grounding can reduce it. Put authoritative material in context — the actual document, codebase, or a current primary source — and require citations or executable checks. The model can still misread, omit, combine, or miscite grounded material, so grounding improves the evidence path; it does not replace verification.

Right-sized verification

Verification isn't paranoia — it's a dial you set to the stakes:

Stakes	Example	Verification
Low	Brainstorm, draft for your eyes	Skim. Move on.
Medium	Content you'll publish, personal tools	Read fully; spot-check any factual claim you'd repeat
High	Client work, anything with money or reputation	Test it yourself, on another device; independent source for key facts
Critical	Legal, medical, financial, security	AI drafts, qualified human signs. No exceptions.

For code, verification is beautifully concrete: run it. Chapters 4 and 9 build this into your workflow so thoroughly it stops costing willpower — the agent runs tests automatically and you review evidence, not promises.

Responsible-use gate

Accuracy is only one kind of failure. Before AI-assisted work affects another person, ask:

1. Data and consent: Do you need this personal, customer, or confidential data? Are you allowed to provide it, and do you know how the selected service retains it?
2. Rights and provenance: Do you have the right to use the source material, and can you trace important claims, quotes, media, and generated assets?
3. Uneven impact: Who could be misrepresented, excluded, or harmed? Test with representative users and cases rather than assuming one successful path generalizes.
4. Disclosure and recourse: Would AI involvement materially matter to the recipient? Name the human who can review, correct, or appeal the result.
5. Misuse: How could this output or automation be abused? Reduce capability, scope, or access before adding a warning label.

If you cannot answer the gate at the stakes involved, stop and get the qualified owner rather than delegating the uncertainty.

Try it now

Calibration is a felt skill. Build some now:

1. Ask Claude about something you know deeply — your field, your city, your craft. Grade it: where was it excellent, where was it shallow, did anything smell invented?
2. Ask about the most obscure corner of that topic you can think of. Watch the quality change as training data thins.
3. Journal one line: what does "thin ice" look like now that you've seen it?

Check your understanding

• Name three strength areas and three stumble areas from memory.
• Explain hallucination in one sentence using the word "plausible."
• Why does grounding (putting truth in context) reduce hallucination?
• What verification level does client work get? What about a brainstorm?
• Run the responsible-use gate on one task from Lesson 1.1. Which question changes your plan?