Logs, metrics and traces explained

What it is

A log is a timestamped record of one event. A payment service might write a JSON line with level, service name, order id and the message card_declined. Structured logs keep those fields separate, so you can search by order id instead of grepping a sentence. Logs carry the most detail, and they cost the most to store and search at volume.

A metric is a number tracked over time, usually with a small set of labels. http_requests_total with labels route and status is a counter. Queue depth is a gauge. Request latency is a histogram. Metrics are cheap to keep and fast to query. They show that something changed, such as a rising error rate, but not the cause.

A trace follows one request across services. Each step is a span with a start time, end time, status and a parent span. A checkout trace can show 800 ms in the payment service, 120 ms in the database and a retry to a card processor. Traces are usually sampled, so you see a representative subset rather than every request. Logs, metrics and traces overlap, but each answers a different question.

How it works

You add instrumentation to the app, often with the OpenTelemetry SDK or a language agent. The app writes structured logs to stdout, exposes metrics on a local endpoint or pushes them, and sends spans to a collector. The collector batches, filters and routes data to backends. For traces to work across services, context must travel in request headers using W3C Trace Context, normally the traceparent header. Without that propagation, one request becomes a set of unrelated spans.

Metrics need labels you can control. Keep label values bounded: route templates instead of raw URLs, status codes instead of error strings, environment and version. Use counters for requests and errors, histograms for latency, gauges for queue depth and memory. Scrape or push at a fixed interval, and record the interval in your dashboard.

Logs need a shared schema: timestamp, level, service, environment, trace_id, span_id, message, error type. Put the trace_id and span_id in every line. Then an alert on a high error rate sends you to a trace, and the trace sends you to the exact log lines. Some systems also link metric buckets to traces with exemplars. Set retention by value: short hot storage for search, longer cold storage for audit.

Why it matters for a business

During an incident, the team needs to know what broke, when it started and who is affected. Metrics catch the change, traces show which service or dependency is slow or failing, and logs give the exact error. That turns a guessing session into a short path. Support can ask for a request id and hand engineering a trace instead of a vague complaint.

Outside incidents, the same data supports planning and release checks. A version label on metrics shows whether the last deploy raised errors. A trend in queue depth tells you when to add workers. A trace on a slow report shows whether the bottleneck is your code or a third party. Finance and operations get a factual view of usage instead of anecdotes.

Security reviews and audits often ask how you detect failures and who did what. Admin action logs and access logs answer the second question. Traces help show the path of a request through services during a review. Keep the data long enough to answer those questions, and restrict who can read it.

Common mistakes

The most common metric mistake is high cardinality. Putting user_id, request_id or raw URLs in metric labels creates far too many time series and makes the metric store slow and expensive. Keep those fields in logs or traces. Metrics should use labels with a small, known set of values.

Logs leak secrets. Auth headers, session tokens, passwords and full card numbers end up in debug lines. Redact at the source, not in the dashboard. Also watch log levels: debug logging in production can flood storage and hide the lines that matter.

Teams often run logs, metrics and traces in separate silos with no shared id. That makes incident work slow. Put trace_id in logs and use one query flow. Sampling is another trap: uniform low sampling can miss rare errors. Tail sampling can keep all errors and slow traces while dropping routine traffic.

Alert fatigue comes from paging on causes such as CPU spikes instead of symptoms users feel. Alert on error rate, latency and saturation for a critical path, and attach a runbook with the first three checks. Give each service an owner. Do not keep every log forever; set retention and delete what you do not need.

How to get started

Pick one revenue path: sign in, search or checkout. Write down what good looks like: error rate, p95 latency, queue depth. Instrument that path end to end with OpenTelemetry. Start with metrics and traces, then add structured logs with the trace id. Do not instrument everything at once.

Choose a backend. Hosted observability services are quick to start. Self-managed Prometheus, Grafana, Loki and Jaeger give you control but need care. Cloud provider native tools reduce setup but can tie you to that provider. Compare ingestion cost, query language and retention before you commit. Then build one dashboard and three alerts: high error rate, high latency, saturation. Each alert needs a runbook.

Test in staging. Break a dependency, confirm the alert fires, follow the trace to the failing service, and check that the logs show the cause. Run the same drill after your first real incident. Expand coverage one service at a time, starting with the ones that touch money or customer access.

If you want the plumbing built with the rest of your delivery setup, Cloudverg's Delivery platform setup is a fixed price package at $14,500 that includes monitoring, alerting and dashboards for the services in scope, plus pipelines, environments and training. It is one option among managed services, open source tools and building it in house. Pick based on the team you have and the time you can spare.

Frequently asked questions