What to log in an AI application

The log line for the request a customer complained about read: status 200, 1.4 seconds, 812 tokens, no errors. Everything a log line is supposed to carry.

It told me nothing whatsoever.

I replayed the question through the same code, against the same model, and got a sensible answer back. The code had no bug in it that I could find because the code had no bug in it. The bad answer came out of text that had been assembled at request time and thrown away at response time.

That is the part that catches people. Every debugging habit you have transfers except the one that matters.

Why the usual log line stops working

On an ordinary service, code is truth. Something took a branch it should not have taken, and you can read the branch. Replay the request, watch the same failure, patch the line, watch it stop. I run a proxy business with billing and provisioning in it and every incident I have ever had there ended that way.

An LLM call snaps that chain at one point. The surrounding code is deterministic and generally innocent. The thing that produced the output is a string, and that string was built at runtime out of a template, some database rows, whatever the user typed, and anything else your code decided to stitch in.

Once the response comes back, that string is gone. Nowhere in memory, nowhere on disk, not in your metrics, not in your APM trace.

So if you did not record it, the incident is unresolvable. I use that word deliberately. You can guess, you can bolt on a check that would have caught this specific case, but you cannot go and look at what actually happened. The ticket closes as “could not reproduce” and both of you know that is not the same as fixed.

The fields

Six things, per call.

The rendered prompt. Every variable interpolated, the system message included, exactly the bytes that went over the wire.

The model, with the resolved version string rather than the alias you called.

The sampling parameters. Temperature, top p, max tokens, stop sequences, seed if you set one.

The raw response, before anything in your code touches it.

An id shared by every call in one user request, plus a step number.

And the boring metadata: finish reason, token counts in and out, wall clock duration, timestamp. Have the timings sitting there so that when somebody claims the product got slower last week you can answer from data rather than from feel. What to do about slow is a separate problem and not this one.

The template trap

Logging the template instead of the rendered text is the mistake I see most, and it is the one that feels most responsible when you make it. Template name plus a content hash. Compact, deduplicated, tidy on a dashboard.

The template is almost never the problem. It has been sitting in the repo working correctly for two months. What broke is what got poured into it.

A null that rendered as the literal string None. A support message with 4,000 words of forwarded email quoted underneath the actual question. A date arriving as a Unix timestamp because someone altered a column type. A customer name containing an apostrophe that terminated a section of your instructions three lines early.

None of those are visible from a name and a hash. All of them are visible instantly from the rendered string, usually before you have finished reading it.

Yes, it is more storage. It is worth every byte, and the rest of this piece is about what to give up instead.

The version string deserves its own note because it fails silently. Aliases are pointers and pointers move, which is the entire point of them. You pinned a friendly name in March, the provider rolled it to a newer build in June, and your outputs shifted on a day when nobody on your team deployed anything. If your logs hold the alias, correlating those two dates is impossible. If they hold the resolved version, it takes a minute. I have needed that exactly once and it settled an argument that would otherwise have run for days.

Whatever you retrieved is input

If there is retrieval anywhere in the request, the chunks that came back are part of the prompt in every sense that matters. Log the document ids, the chunk ids, the scores, and enough text to recognise them.

Whether your retrieval is any good is a different subject with its own long answer. The narrow point here is that a RAG answer cannot be debugged from the answer alone. Two completely different failures produce the same shape of complaint: the model got the right passage and misread it, or the model never saw the right passage and filled the gap. One is a prompt problem. One is an index problem. The fixes have nothing in common.

With the chunks in your log you can tell which one in about thirty seconds. Without them, what happens next is that somebody adds another paragraph of pleading to the system message and ships it, and now you have a system prompt with a fossil record in it.

The obligation you just signed up for

Turn on full prompt logging and your log store now holds personal data. Names, addresses, order numbers, whatever the customer decided to paste in, and once in a while a password they pasted by accident.

Which means retention periods, an access rule, and an answer for the day someone asks you to delete everything you hold about them. If you cannot locate a person inside your logs, you do not have a compliance gap, you have an engineering gap that presents as a compliance gap.

Here is what I settled on. Failures kept 90 days, successes kept seven. Obvious patterns redacted on write. The log store is a separate database from the application, with its own credentials, because the blast radius of a leak here is worse than the blast radius of leaking the app database.

I am not a lawyer and none of that is jurisdiction advice. It is the shape of the thing, and the shape is enough to tell you this gets decided before the logs exist rather than after somebody notices them.

Where to make the cut

Full prompt logging at volume is genuinely expensive, and anybody who tells you otherwise is running a toy. A chatty retrieval application writes 50KB per row once the chunks are in there. Do a few hundred thousand calls a month and you have a storage line item somebody will raise in a meeting.

So sample the successes. One percent, five percent, whatever keeps the bill sane. You need enough of them to see the distribution and to answer questions about what normal traffic looks like, and a thin sample does that perfectly well.

Keep every failure. All of them.

Sampling failures is a false economy

I will take an argument on this one, because the alternative looks so reasonable when it is proposed.

Somebody sets a uniform sample rate. Five percent across the board, applied evenly, no special cases, which is what the tracing library does out of the box and what you would do with HTTP logs. Then a failure gets reported and there is a 95% chance the exact call you need was discarded.

Failures are the only entries anybody ever reads. Nobody has opened a log of a successful call at two in the morning. Successes exist for aggregates and for the occasional what does normal look like question, and one in a hundred answers both.

Failures are why the system exists at all. Keep the retries, the timeouts, the responses that failed to parse, the ones a user marked as wrong, the ones a downstream validator rejected. That set is a tiny fraction of your traffic by volume and it is where every hour of debugging you will ever do gets spent.

If the storage bill has to come down, drop the success rate. Leave the failure path alone.

What I got wrong

I wrote the redaction pass before I wrote the logging, because I was nervous about storing user text, and I made it far too aggressive. It stripped anything resembling an email address, a phone number, or a long run of digits, and replaced each with a marker.

The first real bug I tried to debug with it involved an order reference. My log read redacted, twice. I knew an identifier had gone wrong and I could not tell which one, or whether the two markers were even the same value.

I hash identifiers now instead of masking them, and I log the length alongside, so at minimum I can see when the same value appears in two places. That is a weaker privacy posture than what I started with and a much stronger debugging one, and I am not confident I have the trade sitting in the right place.

The other thing I would change is timing. Turn this on before launch. Every field you add after an incident helps with the next one, and the one you are being asked about today is already gone.

The full field list I use, the redaction rules, and the retention settings that go with them are here.