Context windows explained: how big is big enough

Every AI chat tool has a limit on how much text it can hold in its head at once. That limit is called the context window, and it’s measured in tokens, not pages or words. Ignore it and you’ll hit the same wall eventually: paste in a long contract or a big codebase, ask a question about something near the start, and the model answers as if that part never existed.

I run into this constantly comparing AI tools for this site. Vendors love to headline “1 million token context window” the way phone makers used to headline megapixel counts, but the number on the spec sheet doesn’t tell you whether the tool will actually work for what you’re trying to do. Here’s what a context window is, how it works under the hood, and what to actually check before you pick a tool based on it.

what it is

A context window is the total amount of text a model can process in a single request, counting your prompt, any documents or chat history you’ve included, and the response it generates. All of that shares one fixed budget.

That budget is measured in tokens. A token is roughly four characters of English text, or about three-quarters of a word, though this varies by tokenizer and language. So a 100,000-token window is roughly 75,000 words, or a decent-sized novel.

The size is set by the model itself and varies a lot between vendors and even between models from the same vendor. OpenAI’s GPT-4o handles a 128,000-token context window, per OpenAI’s own model documentation. Anthropic’s Claude models run at 200,000 tokens on the standard tier, according to Anthropic’s docs. Google’s Gemini 1.5 Pro shipped with a 1-million-token window at general availability, with a 2-million-token version previewed, per Google’s own Gemini documentation. These numbers shift as vendors ship new models, so treat any figure here as a snapshot and check the vendor page before you build something that depends on it.

how it works

Everything that goes into a request, your system prompt, your chat history, any uploaded file, the results of any tool calls, and everything the model writes back all draw from the same token budget. Once you’re close to the ceiling, whatever’s in front of the model, the chat app, the API client, has to decide what to do about the overflow. Some interfaces summarize older turns to save space. Others just drop the earliest messages and hope you don’t notice. Either way, once something falls out of the window, the model can no longer see it, even if you referenced it three messages ago.

Under the hood, the reason context windows aren’t just “make the number bigger” is a mechanism called self-attention. The model compares every token in the window against every other token to decide what to weight when generating the next word. That comparison cost grows roughly with the square of the token count, so doubling the context window doesn’t double the compute cost, it roughly quadruples it. That’s part of why long-context requests are slower and pricier per call, and why researchers have had to invent workarounds like sliding-window or grouped-query attention just to make million-token windows practical to serve at all.

why it matters

It decides whether your task even fits. If you’re asking a model to review an entire codebase, summarize a stack of PDFs, or work through a full legal contract, all of that has to fit inside one window alongside your question and its answer. A 32,000-token window, about 24,000 words, runs out fast on a 60-page document plus a back-and-forth conversation about it.

It drives your bill. Most vendors charge per token, both in and out. If your workflow re-sends a large document on every single question, you’re paying to retransmit page one every time, even when the question only concerns page 40. Chunking documents or using retrieval instead of dumping the whole file in every request is often the cheaper move, not just the tidier one.

A bigger window doesn’t mean the model reads it evenly. Research out of Stanford, “Lost in the Middle: How Language Models Use Long Contexts” by Liu et al., found that models are measurably worse at retrieving facts buried in the middle of a long context than facts placed near the start or end, even when everything technically fits inside the window. A million-token context window doesn’t guarantee the model actually uses all million tokens well.

It’s a privacy decision, not just a performance one. Big context windows invite people to paste in more than they should, entire customer databases, internal wikis, unredacted contracts, because it’s technically possible. What happens to that data afterward, whether it’s logged, retained, or used to train future models, is set by the vendor’s data policy, not by the window size. If you’re pasting anything sensitive into a long context, it’s worth reading the vendor’s data handling terms first; theprivacywire.com’s blog has a rundown of what to actually check before you do.

common misconceptions

“A bigger context window means a smarter model.” These are separate specs. Context window size tells you how much text a model can hold at once, not how well it reasons about that text. A smaller-context model with better training can beat a bigger-context one on quality, even though it can’t hold as much material.

“The model remembers everything I’ve ever told it.” Not unless the product layers on a separate persistent memory feature outside the context window itself. Once a conversation runs past the window’s limit and gets truncated or summarized, earlier detail is gone unless something saved it elsewhere.

“If it fits in the window, the model uses all of it equally.” The lost-in-the-middle research above says otherwise. Content placed at the very start or very end of a long context tends to get better recall than content buried in the middle, even when the whole thing technically fits.

“Tokens and words are basically the same thing.” Close enough for rough budgeting, but not exact. Tokenization depends on the model’s tokenizer, and code, non-English text, and unusual punctuation often eat more tokens per word than plain English prose does, which can burn through a budget faster than a word count would suggest.

where to go from here

Context windows are one piece of picking the right AI tool for a job. A few places to go next on this site: our explainer on what AI tokens are and why they cost money if the token math above raised more questions than it answered, our breakdown of RAG vs. fine-tuning if you’re hitting context limits on a recurring workflow and need a workaround instead of just a bigger window, and our piece on why AI models hallucinate since long-context accuracy problems and hallucination are related but not the same failure mode. You can browse everything else we’ve published at the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-13.