RAG misses exact IDs and retrieves stale policy: fix retrieval before prompting

A support assistant gives a polished answer with a citation, but the cited policy is superseded or belongs to a similarly named product. A recent RAG practitioner discussion describes failures around exact identifiers and stale documents. These are retrieval problems; polishing the answer prompt will not reliably put the right document into context.

Separate exact lookup from semantic similarity

Build a diagnostic set with real product codes, version numbers, policy IDs, aliases and near matches. Record the expected source document and the wrong candidates the system retrieved. Microsoft's hybrid search guidance explains why keyword retrieval helps with codes, specialized terms, dates and names that vector similarity can miss. Combine lexical and vector signals where the corpus and query warrant it.

For exact identifiers, normalize the key at ingestion and query time. Preserve the original identifier in a searchable field. A fuzzy title match should not outrank an exact active policy ID merely because its prose is semantically closer to the question.

Make validity explicit in metadata

Store effective date, superseded-by relationship, product scope, locale, owner and approval status as structured fields. Microsoft's filter documentation describes exact-match filters that can restrict retrieval. Apply eligibility filters before generation where supported, then verify that the top cited document actually matches the requested scope and date.

Do not delete historical policies simply to prevent a wrong answer. They may be needed for an “as of last year” question. Let the assistant clarify or refuse when the user does not specify the date and multiple policy versions are valid for different periods.

Evaluate the failure, not only the answer

Score whether the correct document was retrieved, whether an obsolete document was excluded, whether the answer used the right version and whether the citation supports the claim. Test current and historical questions separately. The acceptance set should include a close-looking wrong ID; easy examples are a poor measure of retrieval quality.

Adovo’s view

Trustworthy answers begin with trustworthy selection. An assistant cannot reason its way out of a missing document or an unmarked obsolete one. Give retrieval the same discipline as a database query: keys, validity, scope and evidence. Then the language model has something worth explaining.