How to log and monitor an AI feature in production

Why this matters

An AI feature can fail without throwing an error. The model returns a confident answer, retrieval picks a stale document, or a tool call times out and the agent moves on. Your uptime page stays green while users get wrong results. Logging and monitoring turn that silence into evidence.

Classic logs show a request, a stack trace and a status code. An AI call needs more: the input, prompt version, model and settings, retrieved chunks, tool calls, final output, latency, token counts and any human approval. One trace id must tie those pieces together so a complaint becomes a search.

Monitoring does two jobs. It catches live failures fast, and it builds a record for evaluation. Sample traffic, review it on a schedule, and feed real failures back into your test set. This guide covers how to log and monitor an AI feature in production without turning the log store into a privacy problem or a cost sink.

Before you start

Write down what good looks like before you add a log line. For a support answer, that may be a resolved question with no human edit. For a routing agent, it may be the right queue on the first try. Without a success measure, every alert becomes a debate.

Inventory the data that crosses the feature. List what users send, what tools return, and what the model writes. Mark personal, financial and confidential fields. Decide what can be stored, where, for how long, and who can read it. Redact or hash at write time, not in a downstream job that can fail.

Set budgets and limits you can monitor: tokens per request, time per step, retries and a monthly spend cap. Decide which events count as failures: schema error, tool error after retries, empty answer, guardrail block, user thumbs down. Pick a log schema and trace standard early. If you already run an observability stack, send AI spans there instead of building a second silo. OpenTelemetry works if you model each model call and tool call as a span with attributes.

How to do it

Treat one user request as one trace. The API edge creates the trace id, and every hop passes it along: queue, worker, model call, retrieval, tool call, approval step and response. Log events as JSON with a schema version. That single choice makes replay, support and alert routing much easier.

Do not log full prompts and outputs on every request by default. Log metadata on all requests and full text on a sample plus every failure. This keeps cost and privacy exposure down while still giving you real examples. If you need full text for audit, put it in a restricted store with short retention and separate access.

Watch three layers: service health, AI behavior and cost. Service health covers errors, latency and saturation. AI behavior covers schema validation, empty outputs, guardrail trips, retrieval misses, tool failures and user feedback. Cost covers tokens, model mix and spend per feature. Put all three on one dashboard per feature, with filters for prompt and model version. Review a sample weekly and add bad answers to your evaluation set.

1. Define the trace schema and event names. List the fields every AI event must carry: trace id, feature, environment, prompt version, model name, input hash, token counts, latency, status, error code, tool name and approval state. Keep names stable and version the schema. Put the list in the repo so code review can check it.
2. Create a trace id at the edge. Generate a trace id when the request enters your API or job runner. Pass it in headers and job payloads to every worker and tool call. If your stack supports W3C traceparent, use it so AI spans join your existing traces.
3. Redact before you write. Use an allowlist of fields that can be logged in clear text. Replace emails, phone numbers, card numbers, names and account ids with hashes or tokens. Run redaction in the logging library, not in a batch job, so a missed batch cannot leak a payload.
4. Log each model call as a span. Record prompt version, model name, temperature or other settings, input token count, output token count, finish reason, latency and estimated cost. Store the full prompt only on sampled or failed calls. Link the span to the trace id.
5. Log retrieval, tools and approvals. For retrieval, record document ids, scores and index version, not full document text. For tools, record the tool name, redacted arguments, status, retry count and response size. For approvals, record who approved, when and what they saw.
6. Build dashboards that show user pain. Chart errors, p95 latency, empty answers, schema failures, guardrail trips, thumbs down and token spend. Split each chart by feature, model version and prompt version. Add a table of recent failed traces with links to the full trace.
7. Alert on symptoms, not every log line. Page on sustained errors, latency above your time budget, spend above the monthly cap, a guardrail spike or zero traffic when you expect some. Attach a runbook with the first three checks. Send warnings to a channel, not to a phone.
8. Add a feedback and review queue. Capture thumbs down, human edits, retries and overrides with the trace id. Put them in a queue that a person reviews on a schedule. Tag each case with a reason so patterns show up and become eval cases.

How to check it worked

Break things on purpose in staging. Force a model timeout, return malformed JSON from a tool, make retrieval return nothing, and trip a guardrail. Confirm each case creates a trace, a log event and the alert you expect. An alert you have never seen fire is a guess.

Take a known bad answer from a review and reproduce it from logs alone. You should find the trace id, prompt version, model name, retrieved document ids and tool responses. If you cannot, your logs are missing context that matters.

Test redaction with fake sensitive values. Send a test email, a test card number and a test account id through the feature, then search the log store for them. They should not appear in clear text. Check that the retention job removes old records on schedule, and that the dashboard loads fast for the time ranges your team uses.

Mistakes to avoid

Logging raw prompts and outputs everywhere is the most common mistake. It creates a copy of customer data in a second system, often with weaker access control and longer retention. Sample instead, redact at write time, and keep full text only where you have a clear reason.

Skipping prompt and model version makes every investigation a dead end. When quality drops, you need to know which change shipped and when. Put the version in the log schema, not in a wiki page that goes stale.

Alerting on every error or every guardrail trip trains the team to ignore pages. Alert on sustained symptoms and keep the rest for daily review. Avoid dashboards that only show averages: one broken model version can hide behind a healthy overall number.

A monitor is not an evaluation set. Logs tell you what happened; evals tell you whether a change made things better. Keep both. If you need help putting the pieces in place, Cloudverg's AI agent build at $19,500/month covers one production workflow with full logging, budgets and a monitoring dashboard, among other options.

Frequently asked questions