The final answer being wrong tells you nothing about which step broke
A single request from Agents and Tool Use might trigger several model calls, a retrieval step, and two or three tool calls before producing one answer. Logging just the initial question and the final response tells you the outcome was wrong. It tells you nothing about which of those intermediate steps is where it actually went wrong.
A trace as a tree of spans, not a single log line
1trace: request_abc1232├── span: retrieve_context (312ms, 5 chunks returned)3├── span: model_call_1 (890ms, decided to call get_invoice)4├── span: tool_call get_invoice(id=4471) (140ms, returned invoice)5├── span: model_call_2 (760ms, final answer)6└── total: 2.1s
With this, a wrong answer is a debugging session, not a guess: check whether retrieval returned the right chunks, whether the model's tool call had the right arguments, whether the tool actually returned what it should have. Each of those is checkable independently instead of re-running the whole request and hoping the same thing happens twice.
A trace is a second copy of everything the model ever saw
The Security Logging and Monitoring article in the Web Security series covers this exact problem for ordinary application logs: logs get shipped to a third-party aggregator, retained for years, and read by more people than the production database ever is. A full prompt and completion trace has the same property, and a new source of leakage that ordinary request logging doesn't: the content isn't just what a user typed, it's whatever a RAG retrieval step pulled in from your document store, which can carry sensitive content into a log purely because it was part of the context, not because anyone chose to log it.
The same discipline applies: decide explicitly what gets logged in full, what gets redacted, and what gets logged only as a hash or a reference rather than raw content. A full trace is genuinely valuable for debugging, that's the entire point of the last section, and valuable enough that it shouldn't be treated as free to keep around forever without the same access controls as anything else holding user data.
A retention window matched to how long debugging realistically takes, not the default the logging platform ships with, is worth setting deliberately here the same way it's worth setting for any other log stream.
"I can't reproduce it" doesn't mean it didn't happen
The same request, sent twice, can come back with two different answers. For a traditional service, that would be the bug report itself. For an LLM call, it's the default behavior, which makes a bug report of "it did the wrong thing once" genuinely harder to chase than the traditional version of the same complaint.
Setting temperature to zero while debugging narrows the variance without eliminating it entirely, and it's worth doing specifically so a fix can be checked against something closer to a fixed target. What actually makes a report investigable, though, is the trace from the first section existing at all: the exact request, exact model version, exact parameters, and exact retrieved context that produced the one bad answer, captured at the time it happened, because getting the identical output again on demand often isn't possible even with temperature at zero.
Tracking model and prompt version alongside every trace turns a vague "it got worse recently" into an answerable question: compare traces from before and after a specific deploy, against the same golden set this series has already built a habit of running, and see whether the scores actually moved.
A generic error rate misses most of what actually goes wrong here
A 500 error rate alert catches a model API being down. It catches almost nothing else that actually goes wrong in an LLM application, because most of the failure modes covered across this series don't throw an error, they return a confident, well-formed, wrong answer.
A few signals worth alerting on specifically. The cascade escalation rate from Latency, Cost, and Streaming, a sudden jump means either traffic got harder or the cheap model regressed, both worth knowing immediately rather than noticing in next month's cost report. Eval scores against a sample of real production traffic, scored the same way as the golden set, to catch drift that a static test suite run only in CI will never see, since it only ever tests the same fixed cases. Tool-call failure rate tracked separately from model-call failure rate, since a tool failing and a model failing point at completely different parts of the system and need different people paged.
None of these replace the tracing from the first section, they're what tells you to go look at a trace in the first place, instead of finding out from a user complaint three months from now.