Cloud

Observability and monitoring are not the same thing

Monitoring tells you a known thing has broken. Observability lets you investigate something you did not anticipate. Production needs both.

Key takeaways

  • Monitoring answers anticipated questions; observability answers new ones.
  • Alert on user-visible symptoms and give every alert a response.
  • Correlation identifiers across logs and traces make investigation possible.
  • SLOs and error budgets turn reliability into a decision.

The practical difference

Monitoring watches predefined signals and alerts when they cross a threshold: disk usage, error rate, queue depth. It answers questions you thought of in advance.

Observability is the ability to ask new questions of a running system without shipping new code. It comes from having sufficiently rich telemetry — traces, structured logs and metrics with useful dimensions.

Alert on symptoms, not causes

Alerting on every cause produces noise: high CPU may be entirely fine. Alerting on user-visible symptoms — requests failing, latency beyond an agreed threshold, work not being processed — produces alerts that matter.

Every alert should have a documented response. An alert nobody acts on trains people to ignore the channel, which eventually includes the alert that mattered.

Structure your telemetry

  • Structured logs with consistent fields, including a correlation identifier.
  • Traces across service boundaries so a slow request can be attributed.
  • Metrics with dimensions you will actually want to slice by.
  • A common standard such as OpenTelemetry so instrumentation is not tied to one vendor.

Service level objectives make it actionable

An SLO expresses what good means for users, and the error budget derived from it turns reliability into a decision rather than an argument. When the budget is intact, ship features; when it is spent, fix reliability.

Defining objectives with product owners rather than only within engineering is what makes them stick, because the trade-off is a business one.

Reducing alert fatigue

Most teams have too many alerts, and the cost is invisible until an important one is missed among the noise. A periodic review is one of the highest-value operational habits available.

For each alert, ask when it last fired, what action was taken, and whether it would fire for a problem users would notice. Alerts that have never been actioned should be deleted or downgraded to a dashboard.

Route by severity: things needing immediate action wake someone, things needing attention within a day go to a queue, and everything else is a dashboard. Conflating these is what produces fatigue.


Frequently asked questions

When storage cost or noise starts to outweigh the value. Sample high-volume traces, keep detailed data for a shorter window and aggregate for the long term.

Open-source stacks are capable and cost engineering time to run. Commercial platforms cost money and save that time. Either is defensible; running neither properly is not.

Structured logging with correlation identifiers, then tracing across the two or three services involved in your most important user journey. That combination answers most production questions.

Working on this?

A conversation with our team, no charge and no obligation.

Book a conversation