Teams buy an observability platform, wire up the default dashboards and consider the problem addressed. Then an incident happens and everyone stares at graphs showing that latency is elevated, which they already knew from the alert, and the actual diagnosis happens by reading logs on a laptop.
01Start from the questions, not the tooling
Write down the questions you actually need to answer during an incident. Which customers are affected? Is it one endpoint or all of them? Did it start with a deploy? Is it us or a dependency? Which tenant is generating the load?
Then check whether your current setup can answer each in under a minute. Most cannot answer the customer question at all, because customer identity is not attached to telemetry. That gap is worth more than another dashboard.
| Question | Needs | Commonly missing |
|---|---|---|
| Which customers are affected? | Tenant id on traces and logs | Very often |
| Did a deploy cause it? | Deploy markers on dashboards | Often |
| Is it us or a dependency? | Per-dependency latency and errors | Often |
| Which request path is slow? | Distributed tracing | Sometimes |
| What changed? | Config and feature flag audit log | Almost always |
02Structured logs, with the fields that matter
Unstructured text is greppable and not much else. Structured logs with consistent fields - request id, trace id, tenant id, user id, endpoint, duration, status - are queryable, and querying is what turns logs from a last resort into a first stop.
The high-value field is tenant id, because the most common urgent question is whether a problem affects one customer or everyone. Without it, answering that takes a support ticket and a manual investigation; with it, it is a filter.
03Tracing is what makes distributed systems debuggable
In a system with several services, a slow request is a puzzle: which hop consumed the time? Traces answer that directly, showing the full path and where the latency accumulated. Without them, teams guess and add logging until the picture emerges, usually over several incidents.
Propagate context across every boundary including queues and background jobs, which is the part usually skipped. A trace that ends when work goes asynchronous is missing exactly the part that tends to be slow.
| Signal | Answers | Cost profile |
|---|---|---|
| Metrics | Is something wrong, and how much | Cheap, low cardinality |
| Traces | Where in the path | Moderate, sample intelligently |
| Logs | Exactly what happened | Expensive at volume |
04Alert on symptoms, not on causes
Alerting on high CPU produces pages for conditions that harm nobody. Alerting on error rate and latency against an objective pages when users are actually affected, which is the only thing that justifies waking someone.
Every alert should be actionable and have a runbook. Alerts that fire regularly and get acknowledged without action are training the team to ignore the paging channel, which is precisely the habit you cannot afford during the incident that matters.
05Control the cost before it controls you
Observability bills grow quickly and quietly, and the usual response - reduce retention, sample aggressively, drop fields - often removes exactly what you need during an incident.
Be deliberate instead. Keep high fidelity for errors and slow requests, sample successful fast requests heavily, keep metrics longer than logs, and review the highest-volume log lines quarterly. Debug logging left on in production is a remarkably common and entirely avoidable line item.
Topics
Marcus Hale
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer