How to ship an LLM feature that survives real users

I shipped my first AI feature in 2023, a chatbot support widget bolted onto a SaaS product. It worked great in the demo I built for the founder. Three weeks after launch it was quoting made-up refund policies to real customers and running up an OpenAI bill nobody had budgeted for. That gap, between “works in the demo” and “survives real users,” is where most LLM features die.

This is for operators, PMs, or solo builders who are past the prototype and about to ship an LLM-backed feature to actual users, whether that’s a support bot, a content generator, a search assistant, or anything that calls a model like GPT-4o, Claude, or a self-hosted Llama in response to a user action. It isn’t a prompt engineering guide. It’s the plumbing around the prompt, the part that decides whether the feature still works, and is still affordable, six months from now.

By the end you’ll have a rollout process: an eval set built from real usage, cost ceilings that can’t be blown through, a fallback path for when the model API is slow or down, and a gradual rollout with a working kill switch. None of this is exotic. It’s the same discipline you’d apply to any other external dependency, just applied to a component that’s nondeterministic and priced per token.

what you need

API access to at least one frontier model provider (OpenAI, Anthropic, or Google) with billing set up and a spend alert configured
a second model or provider as fallback, even a cheaper or smaller one
a place to log prompts and responses that isn’t just your terminal: a Postgres table, a tool like Langfuse, or a jsonl file you tail
a staging environment with its own API keys, so a staging bug can’t burn your production budget
20-50 real user inputs to build a first eval set, pulled from support tickets, beta testers, or your own dogfooding
a feature flag system, even something as simple as an environment-based percentage rollout
budget: expect $50-300/month in API spend for a low-traffic feature during rollout, more once you’re past a few thousand calls a day

step by step

1. write down what failure looks like before you write a prompt

Action: before touching the model, list the concrete ways this feature fails a user. Not “the AI is wrong,” but specific: it invents a discount code that doesn’t exist, it answers in the wrong language, it returns malformed JSON your frontend can’t parse, it takes 40 seconds and the user closes the tab.

Expected output: a short list, 5-10 bullets, each one testable.

If it breaks: if you can’t name specific failure modes, you don’t understand the feature well enough to ship it. Go watch a few people use the non-AI version of this workflow first.

2. build an eval set from real usage, not your imagination

Action: collect 20-50 real inputs and run them through your prompt. Grade each output pass or fail against the failure list from step 1.

Expected output: a pass rate. If you’re below 90% on obvious cases, the prompt isn’t ready.

# quick eval loop against the OpenAI API
for f in eval_inputs/*.txt; do
  curl -s https://api.openai.com/v1/chat/completions \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg input "$(cat "$f")" '{model:"gpt-4o-mini", messages:[{role:"user", content:$input}]}')" \
    >> eval_outputs.jsonl
done

If it breaks: if the pass rate won’t move above 90% no matter what you change in the prompt, the task may not fit a single LLM call. Break it into smaller steps, or add a retrieval step so the model has facts to ground on instead of guessing.

3. wrap every call in a timeout, retry, and fallback

Action: no LLM call should hang your request indefinitely or fail your whole feature because the API had a bad five minutes. Set a hard timeout (8-15 seconds for most user-facing calls), one or two bounded retries with backoff, and a fallback response for when both fail.

from openai import OpenAI, APITimeoutError, APIError

client = OpenAI(timeout=12.0, max_retries=2)

def generate_with_fallback(prompt, fallback="Sorry, I couldn't process that right now."):
    try:
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.choices[0].message.content
    except (APITimeoutError, APIError):
        return fallback

If it breaks: if fallback responses fire more than 1-2% of the time, check the provider’s status page, Anthropic’s is at status.anthropic.com, before assuming your code is the problem. Provider outages happen more often than the marketing suggests.

4. set a hard cost ceiling per user and per feature

Action: decide the max you’re willing to spend per user per day, and enforce it in code, not just in a dashboard you’ll check later. A confused user sending 200 messages in a loop can burn through a monthly budget in an afternoon.

Expected output: a counter, Redis or even an in-memory cache at low traffic, that caps calls per user per time window with a friendly message once they hit it.

If it breaks: if legitimate users hit the ceiling, that’s a product decision to revisit the limit. If a single user or IP blows past it fast, you’ve found either abuse or a retry loop in your own frontend code, check both.

5. log every prompt and response, and know what’s in them

Action: log the full prompt, the full response, latency, token count, and cost per call, tied to a user or session ID. Strip or hash anything you don’t need to keep.

{"ts":"2026-07-20T09:14:02Z","user_id":"u_881","model":"gpt-4o-mini","tokens_in":412,"tokens_out":180,"latency_ms":1840,"cost_usd":0.0021,"status":"ok"}

If it breaks: if you’re logging raw user input and it might contain names, emails, or financial details, you now have a data handling problem, not just a debugging one. theprivacywire.com/blog covers this from a practical angle if you want more than “don’t log PII” as advice. This isn’t legal advice, check with a lawyer if you’re in a regulated space.

6. ship it behind a flag to a slice of real traffic

Action: don’t flip it on for 100% of users on day one. Start at 5-10%, chosen by a stable hash of user ID so the same users stay in the test group across sessions.

# simple percentage rollout by hashing user id
ROLLOUT_PCT=10
echo -n "$USER_ID" | sha256sum | awk '{print $1}' | \
  awk -v pct="$ROLLOUT_PCT" '{print (strtonum("0x" substr($0,1,4)) % 100) < pct}'

If it breaks: if errors spike right after the flag flips, that’s the whole point of doing it gradually. Roll back to 0% and diagnose before touching the percentage again.

7. watch for the failures your error logs won’t show you

Action: a 200 status code doesn’t mean the answer was good. Check for empty responses, responses that don’t match your expected format, and repeated identical outputs that suggest the model is stuck.

Expected output: a “silent failure” rate, tracked separately from your HTTP error rate.

If it breaks: if silent failures climb after a prompt change, diff the prompt against the eval set from step 2 before rolling further. That’s exactly what it’s for.

8. alert on cost, latency, and error rate, not just uptime

Action: set three alerts: daily spend above X, p95 latency above Y seconds, error rate above Z%. Send them somewhere you’ll actually see.

If it breaks: if an alert fires and you can’t tell within 5 minutes whether it’s the provider, your code, or a traffic spike, your logging from step 5 isn’t granular enough. Add the missing dimension, provider, endpoint, or user segment, and move on.

9. run an adversarial pass before full rollout

Action: spend 30 minutes trying to break your own feature: prompt injection (“ignore previous instructions and…”), off-topic requests, other languages, extremely long inputs. The OWASP Top 10 for LLM applications is a decent checklist if you want structure instead of winging it.

Expected output: new failure modes added to step 1’s list, and fixes for the worst ones.

If it breaks: if someone gets the model to say something you wouldn’t want screenshotted, that’s not hypothetical. Fix it before 100% rollout, not after.

10. roll out the rest of the way, keep the kill switch live

Action: move from 10% to 50% to 100% over days, not hours, watching the same dashboards each step. Keep the flag wired so you can drop back to 0% instantly, and keep it wired after full rollout too.

If it breaks: if disabling a misbehaving AI feature at 2am needs a full deploy, that’s the gap to close before the next one ships.

common pitfalls

shipping the demo prompt as the production prompt. the prompt that impressed a founder on three cherry-picked examples usually falls apart on the fifth real input. run it through the eval set from step 2 first.
no fallback for provider downtime. both OpenAI and Anthropic have had multi-hour outages. if your whole product goes down when one API does, that’s a design choice, not bad luck.
treating token cost as someone else’s problem until the invoice arrives. set the ceiling from step 4 before launch, not after a surprise bill.
logging raw user input indefinitely with no retention policy. it’s convenient for debugging and a liability for everything else. decide retention, 7, 30, or 90 days, up front.
confusing “the model responded” with “the model responded correctly.” a fluent, confident, wrong answer is worse than an error message, because users trust it.

scaling this

At roughly 10x your launch traffic (a few hundred calls a day), the manual checks from steps 7 and 8 are fine to do by eye. One provider, one model, a spreadsheet-grade eval set. Cost is a rounding error.

At 100x (thousands of calls a day), you need actual infrastructure: a queue in front of the model calls so bursts don’t spike latency, response caching for repeated or near-duplicate queries, and a fallback provider you’ve actually tested, not just configured. This is also where prompt caching, both Anthropic and OpenAI support it and document how it interacts with your usage tier in their rate limit docs and OpenAI’s equivalent, starts meaningfully cutting cost if your prompts share a long, stable prefix.

At 1000x (tens of thousands of calls a day and up), the calculus changes again. You’ll likely be routing across multiple providers by cost and latency, tracking cost per feature and per customer segment rather than just total spend, and evaluating whether some fraction of traffic can move to a cheaper or self-hosted model without hurting quality. This is also the point where one bad prompt change can cost real money in hours, so the eval set from step 2 needs to run automatically on every prompt change, not just before big launches.

where to go next

If you’re trying to control spend as you scale, how to cut your LLM API bill in half goes deeper into caching and routing than step 4 above does. If the feature needs to call external tools or take multi-step actions rather than just answer a question, how to build an AI agent that uses tools covers the extra failure modes that come with letting a model act. If step 2’s pass rate won’t clear 90% because the model keeps guessing at facts it doesn’t have, fine-tuning vs. RAG: which do you actually need walks through how to decide between grounding it with retrieval and fine-tuning it on your own data. For the rest of what we cover here, see the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-20.