On call for an AI feature at three in the morning

The pager doesn’t know the difference between a crash and a bad answer

Traditional on-call has a clean mental model. The service is up or it’s down. You get a 500 error rate spike, a latency graph that falls off a cliff, or a health check that stops responding, and you know something is broken. AI features break that model completely. Your RAG endpoint can return a clean 200 status code and still hand a user a confidently wrong answer, a hallucinated citation, or a response that ignores half the retrieved context. Nothing about that trips a standard health check. The service is “up.” It’s just lying to someone.

That’s the first thing you have to unlearn when you start carrying a pager for an AI feature. Uptime and correctness stopped being the same question. You can have 100% availability and a genuinely broken product at the same time, and your existing alerting probably won’t tell you.

What actually wakes you up

For a real AI feature in production, the alerts that fire at 3am usually fall into a few buckets: error rate on the completion endpoint itself, p95 latency creeping past your timeout, token spend per hour crossing a budget threshold, retry rate climbing, or a canary eval that runs every fifteen minutes suddenly scoring lower than its baseline.

A typical page looks like this: latency on the chat endpoint went from a steady two seconds to eighteen seconds over ten minutes, and your error rate on that same endpoint climbed from under half a percent to double digits. That’s enough to wake someone up, but it tells you almost nothing about the cause yet. It could be the model provider, your own retry storm, a context window overflow from one user pasting in a huge document, or a bad deploy that changed the prompt template an hour earlier and nobody noticed because the change looked harmless in the diff.

Check the status page before you touch anything

The first move, before editing a single line of code, is checking the model provider’s status page and cross-referencing the actual HTTP status codes in your own logs. A wave of 503s or elevated latency across every request, regardless of which prompt or which user, usually means the problem is upstream, not yours. Provider status pages update on their own timeline, which is sometimes slower than you’d like, but the pattern in your logs will usually tell you the story before the status page catches up.

If it’s a provider-wide issue, there’s often nothing to fix. Your job becomes damage control: flip to a fallback path, tell your team what’s happening, and wait. If the errors are scoped to one prompt template, one region, or one customer segment, that’s a strong signal the bug is yours, not theirs, and you can start narrowing from there instead of staring at a dashboard hoping it resolves itself.

Retries turn a blip into a bill

A five-minute upstream hiccup becomes a real incident when your retry logic doesn’t have backoff and jitter. Naive retries hammer the same endpoint the moment it starts struggling, which does two things at once: it makes rate limiting kick in harder, and it means you’re now paying for every failed attempt on top of the successful ones. A small provider slowdown can turn into a self-inflicted spend spike purely because your code kept dialing the same number faster instead of backing off.

The fix is exponential backoff with jitter, a hard cap on retry count, and a circuit breaker that stops calling the model entirely once the error rate crosses a threshold for a given window. There’s nothing clever about it. That circuit breaker is what keeps a rough ten minutes from turning into a rough hour and a surprising line item on next month’s invoice.

The kill switch is the most important line of code you’ll write

At 3am, you don’t want to be reasoning about prompt engineering. You want a switch. A feature flag that disables the AI feature and falls back to something simpler, deployed independently of your normal release pipeline so it can flip in seconds, not through a full build and deploy cycle. Maybe the fallback is a cached response, a simpler rules-based version of the feature, or routing to a different, more boring model while you diagnose the real issue in daylight with a coffee instead of a phone screen.

The teams that handle AI incidents well almost always have this in place before they need it. The teams that don’t are the ones editing a prompt template in production at 3am, hoping the next deploy fixes it, with no way to just turn the thing off.

Logging enough to diagnose without keeping too much

You can’t debug a bad response you can’t see, but you also can’t log raw prompts and completions in plaintext forever, especially when users are pasting things they’d reasonably assume are private. The practical middle ground is structured logging: token counts in and out, latency per call, which model version served the request, cost per request, and a redacted or hashed version of the input, with the full unredacted version retained for a short, defined window and behind access controls. That’s usually enough to spot the pattern, and it’s enough to hand to someone doing a proper investigation, without becoming a second incident of its own later.

Silent model updates are their own category of incident

One of the stranger failure modes with AI features is the incident with no diff. Nothing in your codebase changed, no deploy went out, and behavior still shifted, because the provider updated the model sitting behind an alias like “latest” and your prompts now behave slightly differently against it. There’s no commit to bisect. Your regression tests, if you have any, were written against the old behavior.

The defense here is boring but effective: pin to a specific model snapshot rather than a moving alias, and treat moving to a new version as a deliberate change you test against your own eval set, not something that happens to you on a Tuesday night. If you’re on an alias-based endpoint, you’ve implicitly agreed to let someone else decide when your product changes.

Write a runbook that assumes you’re half asleep

A good AI on-call runbook is written for the version of you that just woke up and can barely read a dashboard, not the sharp version of you writing it on a Tuesday afternoon. It should say, in order: check the provider status page, check error rate and latency by endpoint, check token spend against the hourly budget, check whether the circuit breaker has already tripped, and know exactly where the kill switch lives and who to tell once you’ve flipped it.

It should also cover incident types that don’t look like outages at all: a spike in hallucinated numbers reported by users, a jump in refusals or “I don’t know” responses, or an eval score drop on the canary set. Those deserve their own postmortems, separate from the usual “service was down for eleven minutes” writeup, because the fix is usually a prompt change, a retrieval fix, or a model version pin that needs its own testing before it ships, rather than a rollback.

What good on-call for AI actually looks like

None of this is glamorous. It’s mostly timeouts, feature flags, backoff logic, and log retention policies, which is the same unglamorous plumbing that keeps any production system standing. The one genuinely new piece is that “wrong but confident” is now a real incident class, and your alerting has to be built to catch it, because uptime metrics alone will tell you everything is fine while the product quietly isn’t.

If you’re building or maintaining an AI feature that real users depend on, the pager for it deserves the same seriousness as any other production system, plus a little extra humility about the fact that the thing you’re paging on can fail in ways a health check was never designed to see.

For more breakdowns of what it actually takes to run AI tools, LLM features, and RAG pipelines in production, head back to the AI Tool Gazette homepage.