Enterprise RAG fails at ingestion before it fails at the model

The demo uses a clean handbook and answers beautifully. Production receives scanned contracts, nested SharePoint pages, duplicate policies, spreadsheets, diagrams and documents whose permissions changed yesterday. Retrieval deteriorates, citations become vague and the team upgrades the model.

The model is the most visible component, so it receives the blame. But an answer cannot restore a table that parsing flattened, a heading that chunking detached, a superseded policy that indexing left active or an access rule the pipeline never captured.

Enterprise RAG is a knowledge-ingestion system before it is a chatbot.

Ingestion is not file upload

A production pipeline has to discover, classify, parse, normalise, enrich, authorise, index, update and delete content. Each verb has failure modes.

The source estate may include PDFs, office files, HTML, tickets, wikis, databases and images. Some content has clean text; some requires OCR; some depends on layout. A single parser and a universal chunk size are attractive because they are simple. They are also a reliable way to create plausible nonsense.

Route documents by type and quality. Preserve source, title, headings, page or section, dates, owner, version and access metadata. Keep the original and a traceable transformation record so a bad answer can be followed back to what the pipeline actually saw.

Parsing quality is retrieval quality

Tables, multi-column layouts, footnotes and diagrams routinely lose meaning during extraction. A financial table converted into an unordered stream of cells may contain all the words and none of the relationships.

Sample and score parser output. Test whether:

• text is readable rather than OCR noise;
• reading order is correct;
• headings and sections survive;
• tables preserve row and column meaning;
• figures have useful surrounding context;
• references and page locations remain traceable; and
• protected or unsupported files are reported, not silently skipped.

Use specialist processing only where it improves measured outcomes. Expensive vision processing for every document is not automatically better than a routed pipeline.

Chunking is an information-design decision

Microsoft's RAG guidance treats chunking as a phase with choices such as fixed size, structure-aware and semantic approaches. The right boundary depends on the question and document.

A short policy clause needs precision. A design rationale may require the surrounding section. A table may need a structured representation plus a textual explanation. Parent-child retrieval can return a precise passage with enough context to interpret it.

Evaluate chunking against real questions. Useful measures include whether the correct passage is retrievable, whether it contains sufficient context, how much irrelevant text travels to the model and whether citations land where a reviewer can verify them.

There is no enterprise-wide magic number of tokens.

Metadata is part of the answer

Embeddings capture useful similarity, but enterprises ask filtered questions: the current UK policy, the approved product specification, the contract for one account, the procedure owned by finance.

That needs governed metadata:

• document type and business domain;
• jurisdiction, product or organisation;
• effective and expiry dates;
• approval and lifecycle state;
• authoritative versus reference status;
• sensitivity and access policy;
• source, owner and lineage; and
• relationships to versions or referenced documents.

Use deterministic source fields where possible. Model-generated enrichment can help, but it should be validated and identified as derived rather than silently becoming authoritative metadata.

Freshness includes change and deletion

Many pilots prove that content can enter an index. Production must prove that it can leave.

Define how the pipeline handles:

• an edited section;
• a moved or renamed file;
• a new approved version;
• a revoked permission;
• source deletion;
• legal hold and retention; and
• connector failure or partial reprocessing.

Idempotent jobs, source identifiers, content hashes and tombstones are less glamorous than a model demo. They prevent two versions of policy from competing in the same answer.

Permissions must travel with content

Ingestion should capture the source's access boundary and keep it enforceable at retrieval. Indexing everything into one unrestricted collection and filtering after generation is not an acceptable shortcut.

Identity, group membership and document-level policy change over time. Test revocation latency and negative cases: can a user retrieve a fragment they are not allowed to open? Can a citation reveal a confidential title? Can logs or caches leak retrieved text?

Security is covered more deeply in the companion article, but the foundation starts here. An ingestion pipeline that discards permissions cannot bolt them back on later.

Evaluate the pipeline in layers

Do not grade only the final answer. Keep separate tests for:

1. Discovery: did the connector find the expected source items?
2. Parsing: was the relevant content extracted accurately?
3. Indexing: are version, metadata and permissions correct?
4. Retrieval: does the expected evidence appear in the candidate set?
5. Generation: is the answer grounded, useful and appropriately uncertain?
6. Citation: can a user verify the statement at the source?

Create a representative evaluation set with domain experts, including tables, scans, conflicting versions, ambiguous questions and access-denied cases. Production query feedback should expand it, not replace it.

The economics are mostly operational

Ingestion cost includes connectors, parsing, OCR, enrichment, embedding, storage, reprocessing, review and incident handling. A cheap per-token model can sit behind an expensive and fragile content operation.

Prioritise high-value collections. Measure query demand before indexing every archive. Process incrementally, cache deterministic work and apply expensive extraction selectively. Put an owner and service expectation on each source.

Sometimes the correct decision is not RAG. Structured data may belong behind an API or query layer. A small, curated search corpus may outperform an enterprise-wide knowledge promise. Content cleanup may create more value than another retrieval technique.

What to measure

• source discovery and ingestion success by collection;
• parser-quality failures by document type;
• freshness and permission-revocation lag;
• duplicate and superseded-content rate;
• retrieval recall on the evaluation set;
• citation validity and source-open success;
• unanswered queries caused by missing content;
• cost per processed document and per successful answer; and
• content domains without a named owner.

Adovo’s view

If the assistant fails on a policy, the fastest reaction is often to tune the prompt. First inspect what entered the index. In enterprise systems, the boring layers—document quality, ownership, metadata, permissions, updates and deletion—decide whether the answer deserves trust.

The model can only reason over the evidence we preserved.

Primary documentation and further reading

• Microsoft RAG evaluation guidance
• Azure AI Search document-level access control