Azure AI Search semantic ranker made retrieval worse: test the candidate set

The team enables semantic ranking and its RAG answers become less useful. A Stack Overflow question reports lower measured recall after adding the reranker. Treat that as a measurement problem to localize, rather than assuming every reranker helps every corpus.

Freeze a small query set

Collect representative queries with known relevant documents, including exact product names and less obvious paraphrases. Run the same index and filters with vector-only, text, hybrid and hybrid-plus-semantic configurations. Record relevant-document recall before reranking and the final rank after reranking. A single answer-quality score hides whether retrieval or generation failed.

Inspect the candidate boundary

Microsoft says semantic ranking reranks an existing top-50 result set; it does not search the full index again. Its text-only reranking relies on semantically rich fields in the configured order. If a relevant chunk was outside the candidate set, change retrieval, chunking or hybrid input first. If it entered the set but fell after reranking, inspect field mapping and reranker behavior.

Read the scores correctly

Azure AI Search's hybrid ranking guide distinguishes base @search.score from @search.rerankerScore and documents a query debug option for subscore inspection. Compare rankings, not raw scores across different query types. Adjust candidate count and field configuration against the fixed query set; measure latency and cost before rollout.

Adovo’s view

Relevance is a chain of gates. A sophisticated last gate cannot restore a document discarded at the first one. Spend the first hour proving where the answer disappeared, then tune only that stage.