Sending screenshots to a vision API without burning tokens

The screenshot habit that quietly wrecks your bill

If you’re building anything that looks at a screen (a coding agent checking its own UI output, a QA bot verifying a webpage rendered correctly, a support tool reading a user’s bug report screenshot) you’ve probably noticed your vision API costs climb faster than your text costs. Screenshots are the worst-case input for a vision model: they’re big, they’re mostly whitespace and chrome, and they get sent over and over in a loop because the whole point is to check “did this change.”

The fix isn’t a vendor trick or a paid tier. It’s understanding how these APIs actually turn pixels into tokens, and then not sending pixels you don’t need.

How image tokens actually get counted

Every major vision API turns an image into tokens by tiling it, not by treating it as one flat blob. The exact math differs by provider and changes as they ship new models, so treat any specific number here as “how the mechanism works,” not a locked-in price you should hardcode into a cost calculator without checking current docs.

The general pattern:

1. The image gets resized to fit inside some maximum dimension. If your screenshot is 3840x2160 (a 4K capture) and the model’s cap is, say, 1568px on the long edge, it gets downscaled server-side before any tokenization happens. You’re paying to upload pixels that get thrown away.
2. The resized image gets cut into tiles, commonly somewhere in the 512x512 to 768x768 range depending on the model.
3. Each tile costs a roughly fixed number of tokens, plus a small base overhead for the image as a whole.

The consequence: token cost scales with tile count, not with file size or “how important” the image is. A screenshot with a tiny error message in the corner of an otherwise empty browser window costs the same as a screenshot packed with dense UI, because the model doesn’t know in advance which tiles matter.

This is the part most people miss. You’re not paying for information, you’re paying for area.

Resize before you upload, not after

The single highest-leverage move is downscaling on your side before the image ever leaves your machine. If the API is going to resize your 3840x2160 screenshot down to something like 1568px wide anyway, sending the full-resolution PNG accomplishes nothing except a slower upload and, depending on the provider’s billing model, a bigger source-size penalty before the internal resize kicks in.

Concretely: if you’re capturing full-desktop screenshots for a debugging agent, resize to the model’s documented input ceiling (check the current vendor docs for the exact number, it moves) before the API call, using something as basic as Pillow’s resize() or a sips/ffmpeg call in a pre-processing step. You lose nothing the model would have kept anyway, and you cut upload time and, on providers that meter by source resolution before their own resize step, real token cost.

A second lever: JPEG over PNG for anything that isn’t line art or text-heavy UI. Vision models tokenize the resized bitmap, not the compressed file, so file format doesn’t change token count directly. But smaller files upload faster, and if you’re running this in a loop hundreds of times a day, the wall-clock savings add up even when the token math doesn’t move.

Crop before you send

This is the trick that actually changes tile count, and therefore actual token cost. If your agent only needs to check whether a “Save successful” toast appeared in the top-right corner of a 1920x1080 screen, sending the full screen means the model tokenizes empty page background along with the three words you care about.

Crop to the region of interest first. A 1920x1080 screenshot might tile out to a dozen or more tiles at typical model resolutions. A 400x150 crop of just the toast area fits in one or two tiles. That’s not a marginal saving, it’s an order of magnitude, and it compounds every time the loop runs.

The catch: cropping requires you to know where to look, which means either a fixed layout (fine for testing your own app) or a first low-cost pass to locate the region before a second, tighter pass reads it. That two-pass pattern is worth building into anything that screenshots the same layout repeatedly.

Use low-detail mode when you don’t need to read fine print

Several vision APIs expose an explicit detail or resolution parameter that skips the tiling step entirely and charges a small flat token cost for a heavily downscaled version of the image. This is meant for cases where the model needs gist, not detail: “is this a login page or a dashboard,” “does this chart trend up or down,” “is there an error banner visible at all.”

The tradeoff is real and you should name it to yourself before flipping the switch: low-detail mode will not reliably read small text, distinguish similar icons, or catch a one-pixel layout shift. If your use case is “verify this screenshot doesn’t have a visible crash dialog,” low detail is enough and dramatically cheaper. If your use case is “read the stack trace in this screenshot and tell me the line number,” you need full detail, tiles and all, and trying to save tokens there just gets you wrong answers that cost more to debug than the tokens would have.

Match the mode to the actual question you’re asking, not to habit.

The agent-loop trap: re-sending the same screenshot

Coding agents and browser-automation loops have a specific failure mode worth calling out separately: they screenshot after every action, and if the screen barely changed, they’re paying full image-tokenization cost to tell the model “still on the same page, one button moved two pixels.”

A cheap mitigation that doesn’t require any API feature: diff the screenshot against the previous one locally (a basic perceptual hash or even a pixel-difference threshold) before deciding to send it at all. If nothing meaningfully changed, skip the vision call and reuse the last description, or send a much smaller crop of just the changed region. This isn’t a vendor optimization, it’s just not calling the API when you already know the answer.

Some providers also support prompt/context caching that can reduce cost on repeated context around an image, but the image tokens themselves are typically not what gets cached cheaply, since the image is usually the part of the input that changes between calls. Don’t assume caching solves this for you; check what your specific provider actually caches before you count on it.

When paying for full resolution is the right call

None of this is an argument for always shrinking everything. If you’re building something that reads dense spreadsheets, small UI labels, or code in a screenshot, aggressive downscaling will cost you accuracy, and a wrong answer from a cheap call is more expensive than a right answer from a full-resolution one once you count the retry. The goal isn’t minimum tokens, it’s not paying for resolution the task doesn’t need.

A reasonable default for a screenshot-heavy pipeline: crop to the relevant region first, resize to the model’s stated input ceiling, use low-detail mode for existence/gist checks, and reserve full-detail, uncropped sends for the cases where you genuinely need to read small text or catch a subtle visual diff. That combination is what actually moves the bill, not switching API providers or waiting for a cheaper model.

If you want more breakdowns like this on the tools and tradeoffs behind actually shipping with LLMs, check out the rest of what we cover at AI Tool Gazette.