The same RAG evidence ranks differently when retrieval paths change order

Two routes through a retrieval graph can find the same document. If one route gives it a strong score and the other a weak score, which score should survive deduplication? A LlamaIndex issue reproduces a case in RecursiveRetriever where the first encounter wins, so reversing path order changes the retained score for the same node. The reported version is llama-index-core==0.14.25.

Inspect retrieval before generation

For a query whose answer changed, log node ID, route, score and final rank before sending context to the model. Run the same query with deterministic test retrievers in both route orders. If the evidence list or score changes with traversal order, a prompt rewrite will not repair the retrieval contract. Separate retrieval quality from answer wording in evaluation.

Define what duplicate means

Deduplication by node ID assumes that both paths point to the same underlying evidence. If scores are comparable, retaining the strongest score may be sensible. If scores are produced by different retrievers on different scales, taking the maximum can also mislead. Decide whether to normalize scores, rerank after merging, or keep route-specific evidence before modifying application logic. Test the policy against a small set of real questions with known relevant documents.

Release with a regression set

Keep cases where a document appears through multiple parents, including contradictory or stale evidence. Track retrieved IDs, rank, citations and final answer quality. A library upgrade should pass this set before production rollout, because a seemingly internal deduplication change can alter which documents the model sees first.

Adovo’s view

The model can only reason over the evidence the system chooses to show it. Retrieval order is a product decision when it changes that evidence. Make the ranking rule explicit and testable.