Chunking strategies that survive a document reformat

The chunking strategy that worked fine until it didn’t

Here’s a scenario that’s probably happened to you if you’ve run a RAG pipeline for more than a few months. You built your chunking logic against a set of PDFs. It worked. Retrieval was solid, answers were grounded, everyone was happy. Then someone on the docs team switched export tools, or a vendor changed their API response format from HTML to markdown, or legal asked for headers to be renumbered. Nothing about the underlying content changed. But your chunks did, and retrieval quality quietly dropped for two weeks before anyone noticed, because nobody was watching chunk boundaries, they were watching answer quality, and that lags.

This is the part of a rag chunking strategy that doesn’t get talked about enough. Most writeups compare chunk sizes and overlap percentages as if the source document is a fixed, stable artifact. In production it isn’t. Documents get reformatted constantly: PDF to markdown conversions change, CMS exports add or drop wrapper tags, someone “cleans up” a Confluence page and the heading levels shift. If your chunking strategy depends on the document’s exact structure staying put, you’re signing up to re-chunk and re-embed every time someone touches formatting upstream, and you often won’t know it happened until retrieval degrades.

Why fixed-size and page-based chunking break first

Fixed-size chunking (split every N tokens, done) and page-based chunking (one chunk per PDF page, or split on page breaks) are the two strategies teams ship first because they’re the least code. They’re also the most format-dependent, for different reasons.

Fixed-size chunking looks format-agnostic on the surface since it just counts tokens. But it isn’t immune to reformatting, because it doesn’t respect sentence or paragraph boundaries at all, and small formatting changes shift where sentences start and end in the raw text stream. A paragraph that used to end exactly at your token boundary now bleeds into the next chunk, or gets cut mid-sentence differently than before. The chunks are still correct, but they’re different chunks than last time, with different embeddings, which means your vector store now has two generations of overlapping content unless you fully re-index.

Page-based chunking is worse under reformatting because page boundaries are a property of the rendering rather than of the content. Re-export the same PDF with different margins or a different font and your page breaks move. A table that used to fit on one page now spans two, and your “one chunk per page” logic just split a table in half in a way it didn’t before.

Structure-aware chunking with a fallback

The fix that’s held up best for me is recursive, hierarchy-based splitting: try to split on the biggest structural unit available (section headers, then paragraphs, then sentences, then words), and fall back down that list only when a unit is still too big for your target chunk size. The key word is fallback. If headers exist, use them. If they don’t (because someone flattened the markdown, or a PDF-to-text converter stripped them), the splitter holds up, falling back to paragraph and sentence splitting instead of producing one enormous chunk or garbage output.

This matters specifically for the reformat case because it means your chunker doesn’t have a single point of failure tied to one structural signal. A strategy that only splits on ## headers is fragile in a very specific way: the moment headers disappear or get renamed to <h2> tags or bolded text, that chunker either produces one giant chunk per document or throws an error. A strategy that treats headers as the first preference but sentences as the last resort degrades gracefully instead of failing outright.

The tradeoff is that fallback chunking produces less uniform chunk sizes than fixed-size splitting. You’ll get a 200 token chunk next to a 480 token chunk. If your downstream reranker or context window budget assumes uniform chunk sizes, you need to account for that variance, usually by capping max chunk size and letting the minimum float.

Semantic chunking as a hedge, not a default

Semantic chunking, where you embed sentences and split at points where cosine similarity between adjacent sentences drops, is the strategy most resistant to reformatting because it reads meaning rather than document structure. A reformat that changes tags, headers, or page layout has zero effect on it, because it never reads those signals in the first place.

The cost is real, though, and I want to be direct about the tradeoff rather than pretend it’s free. You’re running an embedding call per sentence (or a sliding window of sentences) just to figure out where to cut, before you’ve even embedded the final chunks for retrieval. For a large corpus that’s a meaningful chunk of your embedding budget spent on chunking decisions rather than on the index itself. It’s also slower to run at ingestion time, which matters if you’re processing documents on a schedule versus in a batch job overnight.

I use semantic chunking selectively: for content that’s genuinely unstructured (support tickets, transcripts, freeform notes) where there’s no reliable header hierarchy to lean on anyway. For structured docs (API references, contracts, technical manuals) the recursive structure-aware approach with a sentence-level fallback covers the reformat risk without paying for an extra embedding pass on every ingest.

Overlap that survives boundary drift

Whatever splitting method you use, overlap between chunks (repeating the tail of one chunk at the head of the next) is what absorbs small boundary shifts from a reformat. If your split point moves by half a sentence because a paragraph got re-wrapped, a 10 to 20 percent overlap usually means the meaning that would’ve been orphaned on the wrong side of the cut is still present in the neighboring chunk. Without overlap, a small formatting change can silently drop a sentence’s context into a gap between two chunks where neither one retrieves well for it.

Overlap isn’t free either. It inflates your total token count and your storage, and pushes up embedding cost proportionally, so there’s a real ceiling on how much overlap is worth buying for a given corpus size. I’ve settled on overlap sized as a fraction of chunk length rather than a fixed token count, so it scales down for a lot of small chunks and up for fewer large ones instead of being tuned once and forgotten.

Give chunks identity that isn’t tied to location

The mistake that costs the most time after a reformat is using page numbers, line numbers, or character offsets as chunk identifiers. Every one of those shifts the instant the document is reformatted, even if the content itself hasn’t changed a single word. That means a routine reformat looks identical to your pipeline as a full rewrite, and you re-embed the whole document for no reason, paying the token cost of the entire corpus over content that’s byte-for-byte the same.

Hash the chunk’s normalized text content instead, and use that hash as the stable ID. Normalize whitespace and casing before hashing so trivial formatting noise (extra blank lines, tab versus space indentation) doesn’t register as a content change. When a document comes back through the pipeline after a reformat, you diff the new hashes against the old ones. Chunks with unchanged hashes get their existing embeddings and vector store entries left alone. Only genuinely new or altered content gets re-embedded. This is the single change that’s saved me the most in API spend, because most reformats change presentation, not content, and there’s no reason to pay to re-embed prose that didn’t move.

Test for reformat resilience before you ship

The practical test I run before trusting a chunking strategy in production: take a document, run it through the pipeline, then deliberately mangle its formatting (strip headers, convert to a different markup, re-wrap paragraphs) and run it through again. Diff the chunk sets. A resilient strategy produces mostly the same chunks with a handful of boundary shifts. A fragile one produces a wildly different chunk set from content that a human would read as identical. That diff is the actual measure of whether your rag chunking strategy will hold up once your document sources start drifting, which they will, because upstream formatting is never something you fully control.

None of this is exotic. Structure-aware splitting with a sentence fallback, modest overlap, content-hash chunk IDs, and a reformat diff test as part of your pipeline checks. The point is to stop paying, in dollars and in silent retrieval degradation, every time someone upstream touches a header tag. Finding the perfect chunk size was never the goal.

If you’re building or debugging a RAG pipeline and want more of this kind of grounded, no-hype breakdown, check out the rest of what we cover at AI Tool Gazette.