Debugging a chain of twenty model requests

Somewhere around step fourteen, the agent starts hallucinating a tool argument that doesn’t exist in the schema. By step seventeen it’s retrying the same malformed call three times. By step twenty you get a final answer that’s confidently wrong, and you have no idea which of the previous nineteen requests planted the bad assumption. If you’ve shipped an agent that chains more than a handful of model calls, you’ve lived this. This is what llm agent tracing is actually for, not a dashboard to admire, but a way to answer “which of these twenty requests broke the chain” without re-running the whole thing and staring at logs.

Why a chain of twenty is a different problem than one call

Debugging a single request is easy. You look at the prompt, look at the completion, done. A chain breaks that model in three ways.

First, state accumulates. By request fifteen, the context includes the outputs of the previous fourteen calls, whatever got summarized or dropped along the way, and any tool results that got stuffed back in. A bug at step three doesn’t show up until step eleven, once the model has had enough turns to build on the bad premise.

Second, failures are often silent. The model doesn’t throw an exception when it misreads a tool result or when your context window truncates the middle of a long document. It just produces a plausible-sounding answer based on incomplete information, and that answer becomes the input to the next call. Nothing crashes. You just get a wrong answer that looks fine until a user catches it.

Third, cost and latency compound in ways a single call never reveals. If each request averages, say, 2,000 input tokens because you’re re-sending accumulated context, twenty requests is 40,000 input tokens before you count outputs. Add one retry loop that fires three times because of a malformed tool call, and you’ve quietly tripled the cost of that one run. None of this shows up if you’re only looking at your API bill in aggregate at the end of the month.

What a trace actually needs to capture

A trace is not a log line. A log line tells you a request happened. A trace tells you where that request sits inside a tree of requests, what came before it, and what depended on it.

At minimum, useful llm agent tracing needs:

A trace ID that ties all twenty requests together as one logical run, plus a span ID per request so you can address any single hop.
Parent-child relationships, because agent chains aren’t always linear. A planning step might fan out into three parallel tool calls before converging back into a synthesis step.
The full input and output payload per span, not a truncated preview. The bug is usually in the part that got cut off in your terminal.
Model name and version per call. If your chain calls a cheaper model for routing and a larger one for synthesis, you need to know which one produced which output when something goes wrong.
Tool call arguments and the raw tool response, separately from the model’s interpretation of that response. This is the single most useful field for catching the “model misread the tool output” class of bug.
Token counts and latency per span, so you can see the cost and time distribution across the chain instead of one lump sum.
Timestamps that let you reconstruct ordering when calls happen concurrently.

That’s it structurally. It’s basically distributed tracing, the same idea used for microservices, applied to model calls instead of service calls. If you’ve worked with OpenTelemetry before, the shape is familiar. The GenAI semantic conventions that have been circulating extend the same span model with fields for prompts, completions, and token usage instead of HTTP status codes.

Where these chains actually break, in practice

A few failure patterns show up over and over in agent chains long enough to matter.

Context window creep is the most common. Every step appends more to the running context. Nobody notices until step twelve or so, when the oldest, most important instruction from step one gets silently truncated or summarized out, and the agent starts drifting from the original task. A trace with per-step token counts makes this visible immediately, because you’ll see the input token count climbing steadily until it plateaus at your context limit, right around where behavior went sideways.

Tool schema drift is the second. You update a tool’s return shape, forget to update the prompt that tells the model how to parse it, and the model starts guessing at fields that no longer exist. This looks like a model problem but it’s a contract problem, and the only way to catch it is to log the raw tool response next to the model’s interpretation of it, so you can diff the two.

Retry amplification is the third, and it’s the one that hits the API bill directly. If step nine fails validation and your orchestration code retries with the same prompt, you can end up paying for the same 3,000-token context three or four times over before something downstream even notices. Without per-span tracing, this shows up as an unexplained spike in your monthly spend, not as a debuggable event.

Cross-step poisoning is the hardest one. A model at step four makes a small, confident error, an entity gets misattributed, a number gets rounded wrong, and nothing downstream re-verifies it because every subsequent step trusts the accumulated context. The final failure surfaces at step nineteen, but the actual bug is fifteen steps upstream. This is the exact case a flat log file can’t help you with, because you need to walk the tree backward from the failure to find where the bad value was introduced, and that only works if every intermediate span was captured with its own inputs and outputs intact.

The tooling landscape, without a ranking

There are a handful of tools built specifically for this: LangSmith, Langfuse, Helicone, and Arize Phoenix are the names you’ll run into most often, alongside teams who just build the same span model on top of raw OpenTelemetry collectors they already run for the rest of their infrastructure. I’m not going to tell you one is better than the others, because I haven’t run a controlled comparison across them and won’t pretend otherwise. What I can tell you is the shape of the tradeoff you’re actually choosing between.

Hosted tracing tools get you a working UI fast, usually with a decorator or a wrapped client so you’re not hand-rolling span creation. The cost is that your full prompts and completions, which often include user data, are now sitting on someone else’s infrastructure. For a lot of internal tooling that’s a non-issue. For anything handling customer PII, it’s a real question you need to answer before you wire it in, not after.

Self-hosted or open source options give you control over where the data lives, at the cost of running and maintaining another service. Langfuse, for example, ships a self-hostable version specifically for this reason. That’s a real ops line item, not a checkbox.

Building it yourself on OpenTelemetry means you’re not tied to a vendor’s schema and you can feed the same collector you already use for your regular service traces. It also means you own every rough edge, and the GenAI conventions are still young enough that you’ll be filling gaps yourself.

None of these choices is free. Pick based on where your data is allowed to live and how much operational surface area you’re willing to own, not on which one has the nicest landing page.

The overhead nobody mentions upfront

Full trace capture means storing every prompt and completion for every span, for every run. If your chain averages 40,000 tokens across twenty steps and you’re running this a few thousand times a day, your trace storage bill starts looking like a second API bill. Most tools let you sample, capturing full detail on a percentage of runs and metadata-only on the rest, but sampling means the one run a user complains about might be the one you didn’t capture in full. Some tools mitigate this by capturing everything by default and letting you set retention windows instead of sampling rates, which trades storage cost for completeness.

There’s also a latency question. If your tracing calls are synchronous with the model calls, you’re adding a network round trip to every single hop in a twenty-step chain. Most serious tools batch and ship spans asynchronously for exactly this reason, but if you’re rolling your own, this is an easy thing to get wrong the first time.

What actually works day to day

The workflow that holds up in practice is boring. Every run gets a trace ID. Every model call and every tool call gets its own span with full input and output. When something breaks, you pull the trace, sort spans by step, and read the tool responses next to what the model claimed the tool said. Most of the time the bug is right there, a tool result the model misread, a truncated context, or a retry that fired silently. The value isn’t in a fancy dashboard, it’s in having the actual data from the actual run sitting in front of you instead of trying to reconstruct what happened from an API bill and a vague memory of what the chain was supposed to do.

If you’re building a chain longer than four or five steps, wire in tracing before you need it, not after the first confusing production failure. It’s a lot cheaper to add a few lines of instrumentation up front than to try to reproduce a twenty-step bug from scratch.

If you want more breakdowns like this on the tools actually holding up agent pipelines together, come find us at AI Tool Gazette.