Hallucinations and Grounding

A model can write a smooth answer that includes a wrong date, an invented detail, or a quotation that does not appear in the source. This is often called a hallucination: an answer that is false or unsupported but presented as if it were known. The risk is not limited to obscure trivia. A model can misread a document you supplied or fill in a missing fact because it makes the answer sound complete.

You can reduce this risk by giving the model relevant evidence, telling it what to do when evidence is missing, and checking each important claim against the source. These steps make an answer easier to audit. They do not guarantee that every answer will be correct.

Why plausible answers can be wrong

Language models generate likely continuations from patterns in their training and the current context. They do not automatically look up a fact or verify that a sentence is true before writing it. If the prompt asks for a detail that is absent, the model may supply something that fits the surrounding pattern. If two facts are easy to confuse, it may blend them. If a question includes a false premise, it may answer as though the premise were true.

This can happen even when the model is capable of answering similar questions. A source might be incomplete, outdated, ambiguous, or irrelevant. A long document may contain a small exception that changes the answer. A retrieval system may return a near match instead of the correct policy. A citation may point to a real source but fail to support the sentence beside it.

The AI Agents lesson on how language models work explains next-token generation in more detail. For this lesson, remember the practical point: fluency is a property of the wording, not evidence that its claims have been checked.

Ground the answer in material you can inspect

Grounding means tying an answer to information that is available and relevant for the task, such as a policy, a report, a database record, or a trusted website. The source should be close enough to the answer that you can check whether each statement follows from it.

For a question about a supplied document, tell the model to use that document as the evidence for its answer. Say what it should do if the document does not contain the answer. For example:

1Answer using only the policy excerpt below.
2For each factual claim, include a short exact quote and the section label that supports it.
3If the excerpt does not answer the question, say “The provided policy does not say.”
4Do not fill gaps with general knowledge or assumptions.
5
6Policy excerpt:
7[paste the policy and section labels]
8
9Question:
10[ask the specific question]

This changes the task from “produce a plausible answer” to “answer from named evidence, or say the evidence is missing.” The model may still choose the wrong quote or misread it. You need to inspect the quotation and its surrounding context, especially when the decision matters.

Anthropic’s current hallucination guidance recommends allowing uncertainty, extracting direct quotes for document tasks, and verifying claims with citations. These are techniques to reduce unsupported content, not proof that a cited answer is true. The exact prompt details may differ between providers, but source checking is useful across models.

Allow a useful “I don’t know”

If an answer is not in the source, guessing should not be the only way to satisfy the prompt. State what the model should say when it cannot find the requested fact. A useful instruction might be: “If the source does not state the delivery date, say it is not provided. Do not estimate one.”

Be specific about the boundary. “If you are unsure, say so” can help, but it does not explain what evidence is sufficient or which kind of uncertainty matters. Tell the model whether to answer only from supplied text, whether it may use an approved external source, and whether it should ask a follow-up question when the input is incomplete.

Also decide what happens after the model abstains. A user may need to provide another document, an application may retrieve a current record, or a person may need to review the question. An “I don’t know” message without a next step can be honest but still unhelpful.

Ask for quotes before conclusions

For document questions, requesting relevant quotes before asking for a summary or conclusion can make omissions easier to spot. Ask the model to return short exact passages with page, section, or record identifiers when available. Then ask it to answer using those passages, or produce the quote and answer together so a reviewer can compare them.

Quotes have limits. A sentence can be copied exactly but taken out of context. A quote may support one part of a claim and not another. A section label may be wrong. Read enough surrounding material to see whether a later exception or qualification changes the meaning. The lesson on long documents and context discusses source labels, staged summaries, and quote-first workflows in more detail.

Retrieval helps find evidence, but it can miss

If the needed information is not in the prompt, a system may retrieve it from a collection or query an external source. Retrieval-augmented generation (RAG) combines retrieval with generation: a retriever finds passages relevant to the user’s question, and those passages are supplied to a model to help form an answer.

Retrieval is useful when the knowledge base is too large to include in every prompt, when facts change, or when different questions need different documents. The original RAG research paper describes combining a model with explicit retrieved memory for knowledge-intensive tasks. Its reported results apply to the systems and benchmarks in that study. A retrieval step can provide evidence, but it cannot ensure the right passage was found or that the model used it correctly.

Common retrieval problems include missing or stale documents, a passage that matches the wording but not the actual question, and results that omit an important exception. Check the source’s date and authority. If retrieval returns no useful passage, the answer should say the available sources did not resolve the question rather than improvise.

The AI Agents course covers the retrieval system itself in RAG and vector databases. This lesson focuses on the prompt behavior around retrieved material: identify the source, answer from relevant evidence, and report when the evidence is insufficient.

Check claims after the answer is written

Grounding can happen before generation, by providing sources and constraints, or after generation, by checking each claim. In a post-generation check, split a response into factual statements and ask what source supports each one. Then open the source and verify that it says what the answer claims.

For example, imagine (this is an illustrative case, not a recorded model reply) a policy excerpt says: “Most items may be returned within 30 days. Items marked final sale cannot be returned.” A user asks whether a final-sale item can be returned. A careless answer might quote “returned within 30 days” and say yes. The quote is real, but it does not support the answer for this item. The exception controls the case.

A careful check asks:

1. What exact claim is the answer making?
2. Which source and passage does it rely on?
3. Does the source support the whole claim, including conditions and exceptions?
4. Is the source authoritative and current for this question?
5. If support is missing, should the claim be removed, qualified, or sent for review?

Do not ask only “Are all these claims correct?” That can invite another unsupported judgment. Give the checker the source, ask it to point to supporting text, and independently inspect the important references. For high-impact answers, use a person or a reliable deterministic check as well.

Choose a method that matches the information gap

Use the smallest method that supplies the needed evidence. If a short excerpt answers the question, there is no need to build a retrieval system. If current account status matters, a static prompt cannot provide it; you need an up-to-date source or a human response.

Test grounding as a behavior

Include cases where the answer is directly supported, where one key fact is missing, where an exception changes the result, and where two sources disagree. Check not only whether the final answer sounds right, but also whether its cited support matches the claim and whether it handles missing evidence correctly.

For a small test set, you can manually mark the supporting passage and expected answer. For structured tasks, code can verify that every claim has a source identifier or that the answer abstains when a required field is null. A model grader can help assess whether a quote supports a sentence, but validate its decisions with people. The lesson on comparing prompts with graders explains how to compare these graders and preserve unknown results.

Avoid scoring a response as grounded simply because it has citations. A citation can be fabricated, unrelated, or incomplete. Track citation support and answer correctness separately. If the task allows general knowledge in addition to provided sources, label which claims come from which source type so reviewers can tell the difference.

Mistakes beginners make

1. Treating fluent wording as checked knowledge. Verify factual claims against evidence suited to the question.
2. Telling the model to “be accurate” without supplying information. Add a trusted source, retrieval step, or a rule to ask for missing details.
3. Assuming “I don’t know” will happen automatically. Define when to abstain and what next step is useful.
4. Accepting a citation without opening it. Check that the source exists and supports the full claim.
5. Using a real quote as proof of a conclusion. Confirm the quote’s context, exceptions, and applicability.
6. Assuming retrieval guarantees grounding. Check that the retrieved passage is relevant, authoritative, and current.
7. Asking only for a final answer when evidence matters. Request short quotes or source identifiers and inspect them.
8. Treating a self-check as independent verification. Use source checks, code, or human review where the stakes require it.

Recap and try it yourself

Models can produce unsupported details because plausible continuation is not the same as fact lookup. Ground document answers in supplied or retrieved sources, allow the model to say when the information is missing, and ask for quotes or source identifiers that a person can inspect. Retrieval can help find evidence, but it can miss or return poor evidence. Check important claims against authoritative, current sources and test abstention as part of the task.

Try this with a non-sensitive paragraph that contains a rule and an exception:

1. Ask a direct question that can be answered from the paragraph.
2. Ask for the answer plus a short exact quote and section label.
3. Change the question so the paragraph does not contain the answer. Check whether the model admits the gap instead of guessing.
4. Add an exception that changes the answer and ask again.
5. Verify every quote and conclusion against the paragraph, including its surrounding context.

If the answer cites a real sentence but draws the wrong conclusion, revise the task criteria or add a clearer check. Grounding makes unsupported claims easier to find; the final responsibility for verification still belongs to the person or system using the answer.