Testing LLM output when the answer is different every time

I broke a production pipeline for six hours because a test passed that should not have. The test called an LLM, checked that the response contained the string “success”, and moved on. The model had returned “This was not a success, but here’s what happened,” and my substring match happily lit up green. That is the moment I stopped writing tests for LLM output the way I write tests for a sorting function.

If you ship anything that calls a model in a loop, in a pipeline, or in a customer-facing feature, you already know the problem. Run the same prompt twice and you get two different strings back. Traditional unit tests assume f(x) always equals y. LLM calls give you f(x) equals “something in the neighborhood of y, probably.” Testing that requires different tools, not more of the same tool.

Why assertEqual doesn’t work here

A standard unit test asserts exact equality because the function under test is deterministic. Even with temperature set to 0, most hosted LLM APIs are not guaranteed to be bit-for-bit reproducible across calls, because of batching, hardware scheduling, and floating point non-associativity on GPUs. You can get close to deterministic with temperature 0 and a fixed seed where the API supports one, but “close” is not “equal,” and your test suite needs to know the difference.

The instinct is to relax the assertion into a substring or keyword check, which is how I got burned. The next instinct is to switch to fuzzy string matching, like a Levenshtein distance threshold. That fails too, because two responses can be nearly identical in wording and completely different in meaning (“approve this request” vs “do not approve this request” are one word apart), or wildly different in wording and identical in meaning. String distance is not a proxy for correctness.

What you’re actually testing

Before picking a testing strategy, separate what you’re validating into three buckets, because each one needs a different check:

Structure. Did the output parse as valid JSON? Does it have the fields your downstream code expects? Is the enum value one of the allowed values? This is deterministic even when the content isn’t, and it’s where most of your test coverage should live.

Properties. Is the output under your token or character budget? Does it avoid banned words (competitor names, profanity, PII patterns)? Does a generated SQL query only touch the tables it’s allowed to touch? These are yes/no checks you can write as plain code, no model needed to grade them.

Semantic quality. Does the summary actually reflect the source document? Is the customer support reply on-topic and polite? This is the hard bucket, and it’s where teams either skip testing entirely or reach for another LLM to grade the first one’s output.

Most of the non-deterministic-output pain people complain about is actually a bucket-three problem being handled with a bucket-one tool. Fix the bucket mismatch first and the rest gets easier.

Structural and property tests: write these first

These cost nothing extra to run and they don’t need a model to grade them. If your pipeline expects the model to return JSON with category, confidence, and summary fields, write a test that calls the real API (or a recorded response, more on that below), parses the result, and asserts the schema. Use something like Pydantic or a JSON Schema validator so the failure message tells you exactly which field broke, not just “test failed.”

Property tests go further: assert the confidence field is a float between 0 and 1, assert summary is under 500 characters, assert category is one of your five allowed labels and nothing else. None of this requires the output to be identical between runs. It requires the output to obey the contract your code depends on.

This is also where you catch prompt regressions early. If you change a prompt and the model starts wrapping its JSON in a markdown code fence, your structural test catches it in CI before it hits a customer. A vibes-based manual check of “read the last ten outputs” won’t catch it reliably, because you’ll get unlucky ten times in a row eventually.

Snapshot tests: useful, but you have to accept drift

Snapshot testing (recording a known-good output and diffing future runs against it) works for LLMs only if you loosen what “diff” means. A raw text diff against a saved snapshot will flag a change every single run, because the wording moves even when the meaning doesn’t. That trains your team to stop reading diffs, which defeats the purpose.

What works better is snapshotting the structured fields, not the prose. Snapshot the category and confidence and leave summary out of the diff, or snapshot summary separately with a human review step instead of an automated pass/fail. I keep a small set of snapshot cases (15 to 20 real inputs, not synthetic ones) that get manually reviewed on prompt changes, because that’s the number a person can actually read through in one sitting without going numb. Past that, review quality drops and the snapshot becomes theater.

If you want automated snapshot diffing on free text, run the diff for a human to read, not for CI to fail on. A red CI check on prose drift with no human judgment attached just gets ignored or the test gets deleted, and then you have no test.

Using an LLM to grade another LLM’s output

This is the “LLM-as-judge” pattern, and it’s the tool most teams reach for on bucket-three problems. It works, with two conditions you need to be honest with yourself about.

First, the judge needs a narrow, checklist-style rubric, not an open-ended “is this good?” prompt. “Does the reply address the customer’s stated question? Yes/No. Does it contain a refund amount if one was requested? Yes/No.” is gradeable. “Rate this reply’s quality from 1-10” produces a number that varies run to run for the exact same input, because you’ve just moved the non-determinism problem one level up instead of solving it.

Second, a judge model has real cost and real latency, and it can be wrong in ways that are hard to notice because a wrong grade doesn’t throw an error, it just quietly lets a bad output through or flags a good one as bad. Treat judge-model verdicts as a signal to sample and review, not as a ground truth you can trust blindly in CI without ever looking at what it flagged. I run judge checks on a subset of test cases in CI and log every verdict, then spot check a handful weekly. I have not run a controlled study on how often the judge disagrees with a human reviewer on my specific rubrics, so I won’t hand you a false-positive rate. What I can tell you is that when I’ve spot-checked, disagreements cluster on ambiguous inputs, the same inputs a human reviewer would also flag as borderline. That’s a reason to keep a human in the loop, not a reason to trust the automation unattended.

Recorded responses beat live calls in most of your suite

Calling the real API on every CI run means your test suite has API costs, network flakiness, and rate limits baked into it. Record real responses once (a cassette, in the VCR-testing sense) and replay them for the structural and property tests that make up the bulk of your suite. Reserve live API calls for a small, separate suite that runs less often, on a schedule or before a release, specifically to catch cases where the provider changed model behavior under you without a version bump. That happens. Providers update models behind the same endpoint name, and your recorded cassette won’t tell you about it because it’s frozen in time. A live check on a schedule is your alarm for that.

What this actually buys you

None of this makes your model deterministic. It doesn’t need to be. The goal of a test suite for LLM output is to catch the failure modes that are actually deterministic even when the text isn’t: a broken parser, a prompt change that drops a required field, a response that blows your token budget, a reply that mentions a competitor by name. Those are bugs you can catch with code, not vibes, and they’re most of what breaks in production. The genuinely hard cases, whether a summary is faithful to a five-page source document, still need a human reading a sample of outputs on a schedule. No amount of clever test tooling replaces that, and anyone telling you otherwise is selling something.

If you’re building or maintaining an LLM-powered pipeline and want more of this kind of grounded, no-hype breakdown, check out the rest of what we cover at AI Tool Gazette.