RAGingestionparsingchunkingpdfmetadata

Ingestion: Parsing & Chunking

Turning raw files into clean, well-bounded, well-labelled chunks is where most RAG quality is won or lost.

Progress

Parsing: getting text out without destroying it

Ingestion starts before any embedding: you must extract text from the source format while preserving the structure that carries meaning. A PDF is a set of positioned glyphs, not a document; a naive extractor produces two-column papers interleaved line by line, headers repeated on every page, and tables flattened into word soup.

Each format needs its own parser. PDFs need layout-aware extraction (detect columns, reading order, headers/footers) and OCR for scanned pages. HTML needs boilerplate removal — navigation, cookie banners, footers — and conversion of headings to a structure you keep. Tables are the hardest: convert them to Markdown or CSV rows with the header repeated, or store each row as its own chunk with the column names inlined ("Region: EMEA, Q3 revenue: 4.2M").

Cleaning follows parsing: normalise whitespace and Unicode, drop repeated page furniture, fix hyphenation across line breaks ("retriev-\nal"), and strip content that should never be retrievable (tracking pixels, internal comments). Garbage in the chunk becomes garbage in the embedding and eventually garbage in the prompt.

PDF: layout-aware extraction, OCR fallback, dedupe headers/footers, keep page numbers.
HTML: readability-style boilerplate removal, keep heading hierarchy, resolve relative links.
Tables: row-per-chunk with column names inlined, or Markdown tables with header rows repeated.
Code: split on function/class boundaries, keep file path and language.

Chunk size and overlap

A chunk is the unit that gets embedded and retrieved. Too small, and a chunk lacks the context needed to be understood or to match a query ("it expires after 30 days" — what does?). Too large, and the embedding averages over several topics and matches none of them sharply, while each retrieved hit eats a big slice of the token budget.

Typical starting points: 200–500 tokens per chunk, 10–20% overlap. Overlap repeats the tail of one chunk at the head of the next so a sentence straddling a boundary survives intact in at least one chunk. Overlap is not free — it inflates the index and produces near-duplicate hits that a reranker or deduplication step must handle.

There is no universal best size. The right value depends on the corpus (FAQ entries vs legal clauses vs API reference), the query style (short keyword lookups vs long natural-language questions), and the reranker. Treat chunk size as a hyperparameter and pick it with a retrieval eval, not by intuition.

Semantic and structural chunking

Fixed-size splitting ignores the document. Structural chunking splits on the document's own boundaries — headings, paragraphs, list items, table rows, functions — so each chunk is a coherent unit an author intended. This is almost always better than a fixed window and is usually where you should start.

Semantic chunking goes further: embed consecutive sentences and start a new chunk when the similarity between adjacent sentences drops below a threshold, so topic shifts become chunk boundaries. It costs more at ingestion and helps most on long unstructured prose; on well-structured docs the headings already give you the same boundaries for free.

A common and effective hybrid: split structurally, then apply a maximum size with overlap to any section that is too long, and merge tiny sections (a heading with one sentence) into their neighbour. Prepend the heading path ("Billing > Refunds > Timing") to every chunk so the chunk is self-describing even when read in isolation.

Structural split with a size cap and heading path prepended.

1def chunk_markdown(doc: str, max_tokens: int = 400, overlap: int = 60) -> list[dict]:
2    chunks, path = [], []
3    for section in split_on_headings(doc):          # yields (level, title, body)
4        level, title, body = section
5        path = path[: level - 1] + [title]
6        prefix = " > ".join(path)
7        for window in sliding_windows(tokenize(body), max_tokens, overlap):
8            chunks.append({
9                "text": f"{prefix}\n\n{detokenize(window)}",
10                "heading_path": prefix,
11            })
12    return chunks

Chunk metadata

Every chunk should carry metadata that is not part of the embedded text but travels with it: doc_id, source_url, title, heading_path, page, char_start/char_end, updated_at, tenant_id, acl groups, language, doc_type. Metadata powers Metadata Filtering, Citations (you need offsets to quote spans), deduplication, and re-ingestion (delete every chunk with this doc_id when the document changes).

Store the original text and the embedded text separately if they differ — for instance when you prepend a heading path or a summary for embedding but want to show the user the raw passage.

Identity: doc_id, chunk_id, version — needed for updates and deletes.
Location: page, char_start, char_end, heading_path — needed for citations.
Access: tenant_id, acl — needed for security filtering.
Freshness: updated_at, source_updated_at — needed for recency ranking and staleness checks.

Worked example: a boundary that breaks a fact

Source text from a policy document: "Refunds are available for annual plans. Requests must be made within 14 days of purchase. For monthly plans, the window is 48 hours." Suppose a fixed 20-token splitter without overlap cuts after "within". Chunk A ends "Requests must be made within" and chunk B begins "14 days of purchase. For monthly plans, the window is 48 hours."

Query: "How long do I have to request a refund on an annual plan?" Chunk B contains "14 days" but no longer mentions "annual"; chunk A mentions "annual" but has no number. Dense retrieval ranks B highest because it is about refund windows — and the model, reading only B, sees two numbers and no plan attached to the first one. It may answer "48 hours". This is a retrieval-side failure caused by ingestion, and no prompt fixes it.

The fixes, in order of preference: split on paragraph boundaries so the three sentences stay together; add overlap so "annual plans. Requests must be made within 14 days" appears intact in some chunk; and prepend the heading path "Refund policy > Annual plans" so even a fragment is labelled.

Key points

Parse with format-specific tools; naive text extraction destroys tables and reading order.
Chunk size is a hyperparameter: 200–500 tokens and 10–20% overlap is a starting point, not an answer.
Prefer structural boundaries (headings, paragraphs, rows) over fixed windows.
Prepend the heading path so every chunk is self-describing.
Attach metadata for identity, location, access, and freshness to every chunk.
A bad chunk boundary is a retrieval failure that no prompt engineering can repair.

When to use — and when not to

Use it when

Any corpus that must be retrieved passage-by-passage.
Documents with clear structure — use structural chunking first.
Long unstructured prose — consider semantic chunking.
Tables and code — use format-aware chunkers, never plain text windows.

Avoid it when

Documents short enough to embed whole (FAQ entries, tickets) — one chunk per document.
Data that is really structured records — load it into a database instead of chunking it.
Semantic chunking on well-headed docs — the headings already give the boundaries.

Failure modes

PDF columns interleaved line by line; chunks are unreadable and embed as noise.
A fact split across two chunks; neither chunk answers the question alone.
Tables flattened so numbers lose their column labels.
Chunks lack doc_id, so updating a document leaves stale chunks in the index.
Boilerplate (nav menus, footers) embedded thousands of times and retrieved for everything.
Chunk size chosen once and never re-evaluated after the corpus changes.

← RAG Overview Embeddings →