Unused metadata blocks a RAG document from being chunked

A document fails ingestion with a metadata-length error even though the node splitter was configured with include_metadata=False. An open LlamaIndex issue includes a minimal reproduction for SentenceSplitter and TokenTextSplitter on a specific code revision. The report says metadata is still counted against the chunk budget; it does not prove every installed version behaves the same way.

Reproduce with your package and payload

Record the exact LlamaIndex version, splitter, chunk size, overlap and metadata fields. Retry one rejected document in isolation with a short text and the original metadata. Then remove or shorten the largest metadata field and compare. This distinguishes a splitter budget problem from tokenizer, parsing or embedding failures.

Treat metadata as part of the ingestion contract

Inventory which fields must reach retrieval and which only belong in the source system. Keep a bounded metadata size, especially for copied descriptions or full HTML placed in a field intended for a title or ID. Do not silently discard source identifiers needed for provenance or access control.

Gate a fix with a regression corpus

Use representative large-metadata documents in a staging ingestion test. If a package update resolves the issue, prove that chunk text, metadata and retrieval results remain acceptable. Until then, a controlled metadata normalization step is safer than repeatedly raising chunk size for every document.

Adovo’s view

Retrieval quality starts before the model sees a question. A pipeline that quietly excludes awkward documents creates an answer gap that looks like model failure. Ingestion must be measurable at the document boundary.