Why a 100 percent pass rate usually means a broken eval
I ran a quality checker across roughly two hundred generated files and every one of them passed. Zero failures. I looked at that for a couple of hours before it occurred to me that a perfect score on two hundred machine generated drafts is not a result, it is a symptom. Real batches do not come back clean. They come back mostly fine with a handful of genuinely bad ones buried in the middle.
The checker was broken. It had been broken the whole time I was using it.
The bug was two spaces
The checker reads a piece of writing and flags patterns that make text read as machine generated: uniform paragraph lengths, the same three item list repeated through a piece, a closing paragraph that restates the opening. It prints each finding on its own line with a severity label, then prints a summary line at the bottom with the counts.
I had a wrapper around it, because I was checking hundreds of files rather than one. The wrapper ran the checker per file, searched the output for lines beginning with a severity label, counted them, and passed the file if the count was zero.
The findings were indented by two spaces, because the checker prints them as a nested list under the filename. That is a sensible thing to do for a human reader. My wrapper anchored its search to the start of a line, so it matched nothing, on every file, forever. Every file scored zero. Every file passed.
What tipped me off was not clever. I opened a file by hand, read two paragraphs, saw a repeated structure that should have been flagged, and ran the checker on that single file in a terminal where I could see its actual output. The finding was right there. The checker had been working correctly the entire time. The only broken component was the thing sitting between it and me.
Both programs behaved exactly as written. Neither crashed, neither logged a warning, neither did anything you would call an error. The failure lived in the assumption connecting them, and assumptions do not raise exceptions.
A false pass survives in a way a crash never does
A checker that crashes gets fixed in four minutes, because the breakage is visible and it blocks you. A checker that returns green when it should return red can live for months, because green is the answer you were hoping for. Nobody interrogates a good result as hard as a bad one.
That asymmetry is the entire risk. The failure was silent, and it pointed in the direction I was least likely to question.
It cost me. Two files in that batch had real defects the checker would have caught, and one of them was in work I had written myself and would have wanted flagged. I found them later with a fixed version. When I widened the re check to an earlier run, twenty six more pieces turned up carrying real problems that had also passed. So the exposure was not one afternoon, it was however long that wrapper had been in place, which I cannot tell you precisely.
Your eval is load-bearing code that nobody tests
This is worth generalising, because it is the eval problem and it is the most under tested code in most AI projects I have looked at.
An eval in an LLM application is whatever inspects non deterministic output and scores it. It might check a required field, run a regex, validate against a schema, or call another model and ask whether the answer is any good. Whatever the mechanism, it is software whose entire job is to tell you the truth about your system.
Application code gets tests. Prompts get versioned and diffed. The eval usually gets neither, and stays a script somebody wrote in an afternoon that has never been verified against a case where it should have said no.
That is backwards. If the eval lies, every decision downstream of it is built on the lie, and the lie is shaped like good news, so nothing prompts you to look.
Stop parsing output that was written for humans
The specific bug class here comes up constantly: I was scraping another program’s human readable output. Indentation, alignment, a friendly summary. None of that is a contract. The tool’s author can add a space, rename a label, or reorder columns without breaking anything from their perspective, and silently break every script parsing it.
If a tool offers structured output, use it. A JSON flag, a machine readable mode, an exit code that means something. Take the boring interface over the pretty one. Where none exists, parse the most stable thing available. In my case that was the summary line with the counts, not the individual finding lines that were subject to formatting decisions.
A few smaller things belong on the same checklist. Confirm whether the tool writes to stdout or stderr, because capturing only one can throw away the half where errors live. Check the exit code separately from the text, and do not assume zero means success until you have confirmed that tool sets it meaningfully, since plenty return zero regardless. Watch encoding, particularly on Windows, where one unicode character in a subprocess call can mangle text and quietly break matching.
Write the case your checker has to fail
This is the recommendation I would push hardest, and it is cheap.
Every eval needs an input it is guaranteed to fail. Seed a known bad case: a file that is deliberately, obviously terrible by whatever standard you are measuring, and run it through the checker as part of the check. If it passes, your checker is broken and you know immediately rather than two hundred files later.
Make the bad case properly bad, not subtly bad. Mine is a few paragraphs of the most obvious machine writing I could produce on purpose: identical paragraph lengths throughout, a three item list in every section, a closing paragraph restating the opening. I want it to trip every rule at once. A subtle case tests the sensitivity of your checker, which is a harder and separate question. The negative control answers one thing only, whether the wiring is connected, so make it loud enough that a miss is definitive rather than debatable.
Negative controls are standard practice in other fields. You do not only run the instrument on the samples you care about. You run it on something whose answer you already know, so you can tell whether the instrument works before trusting what it says about everything else. An eval that cannot fail is not measuring anything, it is a function returning true.
The related discipline is harder because it is not a code change: be suspicious of the number you wanted. When a result comes back better than expected, that is the moment to verify the measurement. I now treat a pass rate above roughly ninety five percent on generated content as a reason to go check the checker. Sometimes it is genuinely a good batch. Verifying costs a few minutes, and not verifying a fake result costs everything built on top of it.
The same failure with a model as judge
If you use an LLM as a judge, all of this applies with the volume turned up.
A judge that has drifted into approving everything looks identical to a judge working correctly on a strong batch. You cannot distinguish those two states from the pass rate, which is the only number most teams watch. Feed the judge known bad cases on a schedule and confirm it still rejects them. A judge that never says no is not a judge, it is a rubber stamp with a token bill attached.
What I changed
The wrapper parses the summary line now instead of the finding lines. A deliberately terrible file runs through the checker on every batch, and if that file passes, the run halts and tells me. I watch the pass rate as a signal about the instrument rather than as a score.
I will be straight about the limits. I have not re audited everything that ever passed through that wrapper. I fixed it, re checked the batch in front of me, swept the previous run, and found what I found. There may be more. The negative control means it cannot fail the same way again, but I am not going to claim the corpus is clean, because I have not measured it.
The takeaway is small and cheap: write the failing case first, and make sure your checker can say no before you trust it saying yes. A hundred percent is not a grade, it is a question.
For more breakdowns like this on how AI tooling behaves once you put weight on it, head back to the AI Tool Gazette homepage.