A RAG chatbot can give a wrong answer even when the correct document is in your knowledge base. The document may not be indexed, the search may retrieve the wrong section, or the model may misread the evidence it receives. Those failures look similar to the user but require different repairs.
RAG means retrieval-augmented generation: the system finds source material and passes it to a language model before generating an answer. That extra evidence is useful. It does not automatically make the answer correct, current, or appropriate for the person asking.
Start with one failing question and follow it through the system. The aim is to find the earliest point where the correct evidence disappears or changes meaning.
The Reddit question behind this diagnostic
In an r/Rag discussion about confidently wrong answers, the poster described trying model and prompt changes before discovering problems with stale data, filters, and apparently relevant passages that did not answer the question.
The comments separated retrieval failure from answer faithfulness. u/hannune suggested checking whether the right document reached the model at all. u/Unhappy_Finding_874 emphasized version information because an unchanged document identifier can hide changed source content. Another participant described using a reference question set but acknowledged that it did not resolve freshness by itself.
This is a practitioner discussion, not a benchmark of how often each failure occurs. It also includes a vendor-affiliated commenter and a vendor article linked by the original poster. The useful contribution is the debugging question: what evidence did this particular answer actually use?
A separate municipal-chatbot discussion raised another practical issue: real users combine topics and use wording unlike the documents. Improving the demo questions would hide that problem rather than solve it.
Begin with the answer the system should have given
Before changing retrieval settings, ask a subject-matter owner to establish an acceptable answer and the sources that support it. Some questions do not have one answer until the user supplies missing context.
For example, “Can we offer weekend installation?” may depend on the service, region, staffing, and contract. A general installation guide might explain working hours but contain no live availability. A correct chatbot should not turn that guide into an appointment promise.
Record whether the expected outcome is an answer, a clarifying question, a live-system lookup, or a handoff. Include essential qualifications. Otherwise, an evaluation can reward a fluent but incomplete response and penalize an appropriate request for more information.
Do this before showing the reviewer the model's response where practical. Seeing a confident answer first can make the review revolve around its wording instead of the underlying evidence.
A worked example: the right policy exists, but loses
Consider this hypothetical internal support question: “Does the revised installation policy cover Saturday visits for existing customers?” The approved July policy says Saturday visits require a separate availability check. An archived March guide describes a previous arrangement.
The knowledge base contains both. The chatbot answers from March and links to a real document. There is no invented URL, yet the answer is wrong for the current policy.
Use a trace like this to locate the failure:
| Stage | What to inspect | What the example might reveal |
|---|---|---|
| Source storage | Approved file and effective date | July policy exists |
| Ingestion | Extracted text and indexed version | July file was uploaded but indexing failed |
| Filtering | Product, status, date, user access | Archived documents remain eligible |
| Retrieval | Candidate passages | March appears; July cannot appear |
| Context assembly | Material passed to the model | Only the obsolete rule is supplied |
| Answer | Claims and citations | Model faithfully repeats the wrong source |
In this version, a more capable model still lacks the current rule. Repair indexing and the document-status policy first. In a different trace, July might reach the prompt and the model might still choose March. That would require a different investigation into conflicting evidence and generation.
This example is a diagnostic exercise, not a reported customer incident.
Check ingestion before tuning search
Verify that the needed passage survives document processing. A file can be present in storage while its tables, footnotes, scanned pages, or appendices are absent from the indexed text.
Open the extracted representation beside the original. Check headings, table labels, units, and qualifications. A row that reads “30” is not useful without knowing whether it means days, minutes, or a quantity limit. A heading can determine which customer group a paragraph applies to.
Track source identity, version, extraction status, and indexing time. A green upload indicator should not be treated as proof that every page is searchable. Failed or partial processing needs an owner and a retry path that preserves the original evidence.
For frequently changing documents, define what happens between approval and completed indexing. Depending on the task, the system might temporarily use a direct lookup or state that the latest policy is not yet available. Silently using a known stale copy is a business decision that should not happen by accident.
Diagnose filters without weakening access controls
A filter can correctly hide irrelevant material or incorrectly remove the only useful source. Check product, location, effective date, document status, and permission filters separately.
Replay the failure using the same user context. An administrator's successful query does not demonstrate that a customer or employee sees the same evidence. Likewise, removing access filters to improve retrieval is not an acceptable production fix.
Use controlled test identities to verify both allowed and denied cases. A user should not receive private facts through a summary simply because the link itself is inaccessible. The permission boundary needs to apply to the material used to generate the answer.
Also inspect cached results. If the system reuses a response, confirm that its cache key and validity rules account for the relevant access context and document updates. A fresh search path can work correctly while a cached answer continues to serve an obsolete result.
Decide whether keyword search would help
Vector search finds conceptual similarity. Exact identifiers need particular care: a part number, policy code, or customer's name may distinguish the correct passage from several near matches.
Microsoft's Azure AI Search documentation describes hybrid search as combining keyword and vector queries, then merging their results. It specifically notes the usefulness of keyword matching for items such as product codes, specialized terms, dates, and names. That supports testing a hybrid approach when semantic similarity misses decisive details. Microsoft hybrid search overview
Do not add components without a failure they can address. If the right document is excluded by a date filter, hybrid search will still not find it. If the answer depends on a live CRM status, searching yesterday's export more effectively may not help.
Keep a baseline run, change one retrieval choice, and compare the same questions. Record whether the evidence improved and whether response time remained acceptable.
What retrieval research actually measures
Anthropic's contextual-retrieval experiments provide a useful example of a measured improvement with a defined scope. In its reported evaluation, combining contextual embeddings and contextual keyword retrieval reduced the top-20-chunk retrieval failure rate from 5.7% to 2.9%. Adding reranking reduced it to 1.9%. Anthropic contextual retrieval
These are retrieval results from the company's tested datasets and configuration. They are not a guarantee that a business chatbot will answer 98.1% of customer questions correctly. Finding a relevant chunk and producing a complete, applicable answer are different outcomes.
The practical lesson is to define the metric before quoting the improvement. Ask a supplier whether “accuracy” refers to retrieval, answer correctness, citation support, or an end-to-end task. Request the test set's relationship to your documents and the cases it excludes.
Check whether citations support individual claims
Once the right evidence reaches the model, inspect the answer sentence by sentence. Does each material statement follow from a supplied source, or has the model filled a gap with a plausible assumption?
A citation can be real and still be inadequate. It may point to a broad document whose relevant section says something narrower. It may support the first half of a sentence while the second half adds a commitment. It may refer to the right policy but the wrong effective date.
For high-consequence answers, ask the system to retain the supporting passage internally and show a useful section link where available. Reviewers should be able to reach the evidence without searching an entire document.
Do not force a citation onto an unsupported claim. If no source answers the question, the useful outcome is to say what is known, identify the missing information, and offer the correct next step.
Bring a question your knowledge-base assistant gets wrong. We can trace the source, retrieval, and business-system integration before recommending changes.
Build an evaluation set around actual work
Collect questions from support cases, internal requests, and permitted conversation logs. Include the wording people use, not only questions written by someone who knows the document titles.
Group them by the kind of evidence required: one passage, several documents, an exact identifier, a current policy, or live data. Add unanswerable and ambiguous questions deliberately. Include changed permissions and superseded documents in controlled tests.
A useful scorecard separates these outcomes:
- Required evidence reached the model.
- The answer was correct and complete enough for the task.
- Citations supported the material claims.
- Access restrictions were respected.
- Missing information triggered clarification or handoff.
- The response arrived within the workflow's usable time.
Human review remains necessary for the reference answers and a meaningful sample of outputs. An automated judge can help organize review, but a model agreeing with another model is not independent confirmation that the business policy was applied correctly.
Test the same question in the language users use
For a few important tasks, create equivalent questions with different wording. Compare an exact policy title, an everyday description, a short follow-up, and a question containing the wrong product name. Have a subject-matter reviewer confirm which versions genuinely ask the same thing.
The expected behaviour may differ. An exact identifier can justify a direct lookup; conflicting product details may require clarification. Record those distinctions instead of expecting every variation to produce identical text.
This reveals whether the assistant works only when users already know the answer's vocabulary. It also tests conversation context: a follow-up such as “does that apply to existing customers?” needs the preceding topic, while a new conversation should not inherit an unrelated customer's context. Keep these as separate evaluation cases.
Read the numbers without hiding failure modes
Suppose a hypothetical pilot contains 100 questions. Twenty require clarification; the other eighty are answerable from approved documents. The system retrieves sufficient evidence for 72 of those eighty and answers 65 correctly.
Retrieval success on answerable cases is 90%. Correct answers among answerable cases are 81.25%. Neither number describes whether the twenty ambiguous cases were handled properly. Report that separately, along with any unauthorized disclosures.
This decomposition prevents an apparently strong average from hiding a specific defect. If retrieval is good but answers drift, tune and test generation. If retrieval fails mainly on product codes, investigate exact matching. If failures cluster around new documents, investigate ingestion and freshness.
Keep question categories stable between comparisons. Replacing difficult questions with easier ones can improve a score without improving the assistant.
When an agent is justified
Some questions require several dependent lookups. The answer to the first search determines the next question, so a controlled agent may be useful. Other questions need a deterministic connection to a system of record rather than more document searching.
For a booking question, retrieve the service policy, then query the actual scheduling system under appropriate permissions. Do not infer availability from a policy PDF. For a document-location question, a maintained catalogue may be simpler than an agent deciding among multiple search strategies.
The AI agent versus workflow guide helps separate those choices. Add autonomy when it solves a demonstrated task requirement, and test the new failure paths it introduces.
What a useful repair deliverable looks like
A completed diagnosis should include the original failing question, approved evidence, the stage that failed, the change made, and the before-and-after result on both the failing case and related cases. Preserve any unresolved limitations.
That record makes the fix maintainable when a document, model, or retrieval service changes. It also prevents the next incident from restarting the same argument about whether the model is “smart enough.”
Begin with one wrong answer you can reproduce. Establish the correct evidence, follow where it goes, and repair the first broken step. A chatbot earns trust through traceable answers and well-handled uncertainty, not through confidence in its wording.