Why company knowledge assistants hallucinate—and what to test

A knowledge assistant gives the wrong answer and the model is blamed. Sometimes the model invented a claim. Often it was given the wrong policy, an incomplete paragraph, two conflicting versions or no useful evidence at all.

“Hallucination” is convenient shorthand, but it hides where the system failed. If every wrong answer is treated as one problem, teams tune prompts while broken ingestion, retrieval and ownership remain untouched.

The practical question is not whether a model can ever be wrong. It is whether the system can find the right evidence, express uncertainty, preserve permissions and fail safely for the decision at hand.

Separate the failure modes

The source is wrong or stale

The assistant faithfully cites an obsolete procedure. This is a content-governance failure. No generation technique can decide which policy is authoritative if the source estate does not encode approval and effective dates.

The content was damaged during ingestion

OCR corrupts a number, a table loses its columns or chunking separates an exception from the rule. The answer may be grounded in a fragment that no longer carries the source's meaning.

Retrieval misses the evidence

The correct passage is indexed but does not appear in the candidate set. Acronyms, exact identifiers, domain language and metadata filters often expose the limits of pure semantic similarity.

Context is insufficient or conflicting

The model receives related passages but lacks the governing section, or receives two versions without lifecycle metadata. It resolves ambiguity confidently because the application did not give it a reliable way to abstain.

Generation exceeds the evidence

The correct source is present, but the answer adds unsupported detail, merges separate claims or ignores an instruction to limit itself to evidence.

The citation does not support the claim

A linked document may be relevant while the specific claim is absent. Citation presence is not citation correctness.

Test the layers independently

Microsoft's RAG evaluators distinguish retrieval quality from groundedness and response quality. AWS also provides retrieval evaluation for knowledge bases. That separation matters: an excellent answer judge can mask a retrieval failure if the model answers from prior knowledge.

Build tests for:

1. Content readiness: correct version, parser output, metadata and permission.
2. Retrieval: expected evidence in the top candidates; irrelevant or prohibited evidence excluded.
3. Context sufficiency: enough information to answer, with conflicts visible.
4. Grounded generation: each material claim supported by supplied evidence.
5. Citation correctness: the cited location supports the adjacent claim and opens for the user.
6. Abstention: the system refuses or asks for clarification when evidence is missing, ambiguous or unauthorised.
7. Task outcome: the answer helps a person complete the real job without unacceptable risk.

One aggregate “accuracy” score cannot show which component to repair.

Build the evaluation set from real work

A useful set includes ordinary questions, not only polished demo prompts:

• vague language and misspellings;
• acronyms with multiple meanings;
• exact clauses, numbers and table cells;
• questions spanning documents;
• conflicting and superseded sources;
• questions with no answer;
• sensitive questions across roles;
• malicious instructions in user input or documents; and
• high-impact decisions that require human review.

Domain experts should establish expected evidence and acceptable answers. Production queries, failures and user corrections should add cases over time. Keep a stable regression core so a model or index change cannot improve one slice while quietly breaking another.

Human and model grading play different roles

Model-based evaluators scale and can apply structured rubrics, but they are not an independent source of truth. They can be inconsistent, share blind spots with the system under test or reward fluent answers.

Calibrate automated graders against expert review. Use deterministic checks for permissions, citation links, required fields and unsupported formats. Sample disagreements. For high-risk domains, retain human approval or escalation at the point of use.

The evaluation method itself needs a version, owner and evidence trail.

Design for honest uncertainty

An assistant should be able to say:

• I could not find an approved source;
• these sources conflict;
• the available document is out of date;
• you do not have access to the supporting material;
• the question needs a jurisdiction, product or date; or
• this decision requires a qualified reviewer.

Abstention is not failure when the alternative is confident fabrication. Measure whether refusals are appropriate: too little abstention creates risk; too much makes the system useless.

Production monitoring needs context

Log enough to reconstruct a failure: application and prompt version, model, retrieval configuration, source identifiers, policy decision, latency, tool calls and evaluator results. Protect that telemetry because it can contain sensitive content.

Monitor shifts in query mix, retrieval scores, no-answer rate, citation opens, user corrections, cost and latency. A stable benchmark does not guarantee stable production behaviour when the corpus and users change.

Create a feedback route that leads to an owner. A thumbs-down stored in a dashboard is not an improvement loop.

Risk should set the release bar

An assistant suggesting internal reading can tolerate a different error profile from one interpreting a contract or changing a customer account. Classify use cases by impact and reversibility.

For higher-risk tasks, require stronger evidence, narrower scope, human review, traceability and safer fallback. Sometimes the right product is search with highlighted evidence, not a generated answer. Sometimes the right answer is a structured workflow rather than open conversation.

What to measure

• retrieval recall and ranking on expected evidence;
• grounded-claim and citation-support rate;
• correct abstention and false-refusal rate;
• permission and cross-tenant failures;
• performance by document type, domain and risk class;
• expert/model evaluator agreement;
• user corrections that become regression tests;
• time from reported error to source or system fix; and
• outcome quality for the business task.

Do not publish a single accuracy percentage without the test population, rubric and uncertainty.

The counterargument: a stronger model will solve most of this

A stronger model can improve reasoning, instruction following and synthesis. It may reduce some errors. It cannot retrieve a file that was never indexed, reconstruct a destroyed table with certainty, know a user's missing permission or make an obsolete policy current.

Model selection matters. It is one controlled variable in a larger system.

Adovo’s view

The phrase “the AI hallucinated” often lets the organisation off too easily. The system may have done exactly what its data, retrieval and product design encouraged.

Test the chain, not only the sentence. The goal is not a demo that always answers. It is a service that knows when its evidence is strong enough to deserve action.

Primary documentation and further reading

• Microsoft RAG evaluation guidance
• NIST AI Risk Management Framework