Observability vs Monitoring #

In managing large-scale computer systems, especially in distributed and dynamic cloud computing environments, we often hear the terms Monitoring and Observability used interchangeably. Many parties consider them the same thing with different names. However, in reality, they represent two very contrasting philosophies and approaches to understanding system health and behavior. Monitoring focuses on overseeing predefined metrics to detect anticipated failures (known unknowns). Conversely, observability gives us the ability to ask new questions never thought of before, to solve complex unexpected problems (unknown unknowns). Understanding this fundamental difference isn’t just a terminology debate — it’s a crucial step toward building resilient systems, minimizing downtime duration, and improving our engineering teams’ operational efficiency.

Basic Concepts and Analogies #

To intuitively understand the difference between monitoring and observability, we can use several real-life analogies.

1. Car Speedometer Analogy vs. Diagnostic Scanner (OBD Scanner) #

• Monitoring (Speedometer & Fuel Indicator): Inside a car dashboard, there are speed, engine temperature, and remaining fuel indicators. These indicators monitor static parameters predetermined by the factory. If engine temperature exceeds safe limits, the red indicator light turns on. This is monitoring. We know exactly what’s being watched (engine temperature) and its limits (heat thresholds). However, when the engine indicator light turns on, the dashboard doesn’t tell us why the engine is having problems.
• Observability (OBD Scanner & Full Sensor Analysis): To find the exact cause of an engine failure, mechanics plug an OBD Scanner into the car’s computer. This tool reads hundreds of sensor parameters in real-time — combustion timing, air-fuel ratios, injector pressure, and even historical cylinder failure logs. With this abundant, interconnected data, mechanics can conclude the failure was caused by a blockage in cylinder number three’s injector due to poor fuel quality. This ability to infer the car’s internal condition from external raw data is what’s called observability.

2. The Nature of Questions Asked #

The difference between the two concepts is reflected in the types of questions we want to answer:

• Monitoring Questions:
  • Is the database server still alive?
  • Is CPU usage exceeding 80%?
  • What’s the current HTTP request error rate?
  • Will the SSL certificate expire within the next 30 days?
• Observability Questions:
  • Why are user checkout transactions failing after we released the new promotion feature, even though CPU and database memory usage metrics look normal?
  • Why does API latency increase only for users accessing from specific regions with older Android devices using e-wallet payment methods?
  • Which microservice triggers the cascading delay domino effect in the API call chain when the shopping cart service (cart-service) experiences slowness?

3. Characteristic Comparison Matrix #

Traditional Monitoring Limitations in the Cloud-Native Era #

Traditional monitoring approaches were designed in the era when our applications ran on single physical servers or static Virtual Machines (VMs) with monolithic databases. Such system characteristics are very predictable: failures are usually caused by full disks, exhausted memory, or dead web server services.

However, in the modern cloud-native era, application architecture has changed into hundreds of micro containers spread across various Kubernetes clusters, running dynamically on autoscaling infrastructure that constantly spawns and dies. Traditional monitoring fails against this new reality due to several key limitations:

1. Loss of Context from Data Aggregation #

Monitoring heavily depends on aggregate metrics (like average CPU utilization per minute). When we combine hundreds of container instances’ performance into one average number, we lose visibility into local failures. For example, our average API latency might show a normal number (200ms), yet behind it, 2% of users experience total failure with latencies above 10 seconds because one container instance has a memory leak.

2. Ephemeral Infrastructure Nature #

In cloud environments, IP addresses and hostnames are temporary. If a Kubernetes container crashes or is stopped by scale-in processes, traditional monitoring machines that only watch host health lose all historical data related to that container. Without telemetry data uploaded to centralized storage before the container is destroyed, we can never investigate that container’s crash cause.

3. False Alarm Problems (Alert Fatigue) #

In complex distributed systems, component dependencies are very high. When the primary database suddenly degrades, traditional monitoring systems send dozens of alarms at once: full database connection usage alarms, rising API Gateway latency alarms, payment service transaction failure alarms, and message queue service timeout alarms. Our engineers get flooded with alarms (alert fatigue), making it hard to find the actual root cause.

Problem Spectrum: Known-Knowns vs Unknown-Unknowns #

To build a good defense system, we must map problem categories based on our knowledge level of the problem:

1. Known-Knowns (Things We Know) #

Basic conditions that are certainly true and easy to periodically check. Example: “We know the payment service runs on port 8080”. We just monitor that port to ensure the service is active.

2. Known-Unknowns (Things We Know Can Fail) #

Problems whose failure types we’ve anticipated, but we don’t know when they’ll occur. Example: “We know server RAM usage can fill up and cause crashes”. The solution is creating memory metric graph visualizations and setting alerts if RAM usage hits the 90% limit.

3. Unknown-Knowns (Things We Know But Don’t Realize) #

Information senior team members hold intuitively but isn’t documented or integrated into automated alarm systems. Example: “We informally know service A always slows down every Monday at 09:00 because of a cron job sync process, but there’s no official runbook for handling it”.

4. Unknown-Unknowns (Mysteries We Never Imagined) #

Complex problems arising from dynamic interactions between new application code, user traffic spikes, third-party dependencies, and cloud network latency. We never predicted these problems would occur, so we can’t possibly prepare static dashboard visualizations or threshold alarms in advance.

This is where observability plays a vital role. Observability assumes failures in distributed systems are certain to happen and their manifestation forms are unpredictable. By collecting rich, contextual telemetry data, we can conduct detective-style investigations to formulate new hypotheses when unexpected incidents strike.

The Three Observability Pillars and Their Correlation #

Observability activities are supported by three main telemetry data pillars: Metrics, Traces, and Logs. They must not stand alone separately; they must be connected through context propagation so investigation processes run smoothly without context-switching obstacles.

flowchart TD
    subgraph Pengguna ["Users & Traffic"]
        User["User (Gets Error 500)"]
    end

    subgraph Gerbang ["API Gateway & Routing"]
        Gateway["API Gateway (Context Injection)"]
    end

    subgraph Layanan ["Microservices Layer"]
        OrderService["Order Service (Context Propagation)"]
        PaymentService["Payment Service (Error Occurs)"]
    end

    subgraph Telemetri ["Observability Diagnosis Cycle"]
        direction LR
        Metrics["1. METRICS (Detection)\n- http_requests_total (5xx)\n- http_request_duration_seconds"]
        
        Traces["2. TRACES (Isolation)\n- trace_id: 8a9b1c...\n- Span: payment-service (Failed)"]
        
        Logs["3. LOGS (Identification)\n- Timeout query details\n- Database error stack trace"]
    end

    User -->|"1. Checkout request"| Gateway
    Gateway -->|"2. Forward request"| OrderService
    OrderService -->|"3. Call payment API"| PaymentService
    PaymentService -->|"4. Connection dropped/failed"| DB[("Database Engine")]

    PaymentService -. "Send error metric" .-> Metrics
    PaymentService -. "Send failure span" .-> Traces
    PaymentService -. "Write JSON error log" .-> Logs

    Metrics -. "Trigger Alert" .-> Alerting["Alert Manager (PagerDuty/Slack)"]
    Alerting -. "SRE engineer analyzes" .-> Traces
    Traces -. "Correlate via trace_id" .-> Logs

1. Three-Pillar Correlation Cycle During Incidents #

When a customer complains about transaction failures on our web application, the efficient troubleshooting workflow using the three observability pillars is as follows:

1. METRICS (Detect): A Slack alarm sounds because the error rate metric on the API Gateway spiked past the SLO tolerance limit. Metrics tell us WHEN the incident occurred and HOW BIG its impact on users is.
2. TRACES (Isolate): We open the distributed tracing visualization for the incident time period. We look for failed transactions (marked in red). From the sequential trace diagram, we see requests flow smoothly through api-gateway and order-service, but experience 500 failures or long timeouts inside payment-service. Traces help us track WHERE the bottleneck or failure is in the dependency chain.
3. LOGS (Identify): Through the trace editor, we click the trace ID (trace_id) of the failed payment-service span. The system automatically displays all structured application log lines with the same trace_id. There, we find the ERROR-level log line: {"trace_id": "8a9b1c...", "message": "Failed to decrypt response from external payment gateway: Key mismatch", "stack_trace": "..."}. Logs provide deep details on WHY the error occurred.

The Important Role of High Cardinality and High Dimensionality #

The main strength of modern observability platforms is their ability to process data with High Cardinality and High Dimensionality characteristics.

1. What is Cardinality? #

In database and data analysis terms, cardinality refers to the number of unique values a data column or dimension has.

• Low Cardinality: Dimensions with only a few unique value variations. Examples: status_code (200, 400, 500), environment (production, staging, development), or http_method (GET, POST, DELETE).
• High Cardinality: Dimensions with thousands to millions of unique value variations. Examples: user_id, transaction_id, ip_address, container_id, or email_address.

2. The Danger of Cardinality Explosion in Traditional Metrics #

Traditional metric systems (like Prometheus) store data as aggregate time series. Every unique label or dimension combination produces a new time-series in the metric database.

For example, if we want to monitor our API latency by adding dimensions:

• http_method (5 values)
• status_code (5 values)
• user_id (1,000,000 unique active user values)

Then the total time series the metric database must store and process is: $$5 \times 5 \times 1,000,000 = 25,000,000 \text{ time series}$$

This suddenly exploding time-series count is called Cardinality Explosion. It exhausts metric database RAM memory, crashes monitoring systems, and triggers fantastic cloud bill spikes. As a result, in traditional monitoring, we’re strictly forbidden from recording high-cardinality data like user_id into metrics.

3. Modern Observability Solutions #

Modern observability systems are designed not to pre-aggregate data upfront. They store telemetry data as Structured Raw Events.

When applications write logs or traces containing user_id and transaction_id data, that data is directly sent to columnar storage systems or fast-indexing databases. When incidents occur, our engineers can run on-demand correlation queries: “Show all failed transactions experienced by user ID 999818 over the last 10 minutes”. We get micro-level search capabilities without breaking monitoring database stability.

Practical Implementation: Instrumentation with OpenTelemetry #

To realize observability, our application code must be instrumented to produce telemetry data. Today, the most recommended open industry standard is OpenTelemetry (OTel), a sandbox project under the CNCF supported by all major cloud and monitoring vendors worldwide. With OTel, our code stays vendor-agnostic; we can switch analysis backends (e.g., from Grafana Loki to Datadog or AWS X-Ray) without changing a single line of our application instrumentation code.

Here’s a practical implementation example of using OpenTelemetry in the Go programming language demonstrating distributed tracing span initialization and context injection when making HTTP calls to other services.

package main

import (
	"context"
	"fmt"
	"net/http"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/trace"
)

// Initialize the global tracer
var tracer = otel.Tracer("order-service")

// ANTI-PATTERN: Making outbound HTTP calls without context propagation
// Engineers can't connect the trace chain between this service and the receiving service.
func callPaymentServiceBad(r *http.Request) {
	req, _ := http.NewRequest("POST", "http://payment-service/pay", nil)
	
	// DON'T: Make a direct call without inserting the traceparent header
	client := &http.Client{}
	client.Do(req)
}

// CORRECT: Using OpenTelemetry for explicit context propagation
func callPaymentServiceGood(ctx context.Context, orderID string) error {
	// 1. Create a new child span from the existing context
	ctx, span := tracer.Start(ctx, "CallPaymentService", trace.WithSpanKind(trace.SpanKindClient))
	defer span.End()

	// Set additional attributes to help investigation (High Cardinality data)
	span.SetAttributes(
		trace.StringAttribute("order.id", orderID),
		trace.StringAttribute("payment.provider", "stripe"),
	)

	req, err := http.NewRequestWithContext(ctx, "POST", "http://payment-service/pay", nil)
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "Failed to create HTTP request")
		return err
	}

	// 2. Inject the tracing context (traceparent header) into the HTTP request header
	// This ensures the same trace_id is forwarded and used by payment-service
	otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

	// 3. Execute the request
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		// Record the error into the telemetry span
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		return err
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		statusErr := fmt.Errorf("payment service returned status: %d", resp.StatusCode)
		span.RecordError(statusErr)
		span.SetStatus(codes.Error, statusErr.Error())
		return statusErr
	}

	span.SetStatus(codes.Ok, "Payment transaction successful")
	return nil
}

func main() {
	// Make sure we've configured the trace provider and global propagator here
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, // W3C Trace Context standard format
		propagation.Baggage{},
	))
}

Building an Observability Culture in SRE Teams #

Adopting observability isn’t just buying the most expensive software or installing telemetry collector agents on all servers. Observability is a culture and a way teams work in designing, building, and maintaining their software systems.

1. Early Instrumentation Habits (Observability-Driven Development) #

Software engineers often only add logs or metrics after major production problems occur. In a healthy culture, instrumentation is considered an important part of writing new code, equal to writing unit tests.

• Before launching new features, teams must ask: “What business metrics will prove this feature works?”, “How can we track this new data write latency in the distributed tracing system?”, and “Do our logs have valid trace_ids?”.

2. Error Budget-Based SLO Evaluation #

Observability culture encourages teams to agree on realistic Service Level Objectives (SLOs) based on real user experiences, not impossible 100% performance targets. We use the Error Budget concept as a failure tolerance measurement tool. If error budgets are still abundant, developer teams have freedom to release new features quickly. However, if error budgets run thin from consecutive incidents, team focus must shift to fixing system stability using insights from observability data.

3. Blameless Post-Mortems #

When system outages occur, our main goal isn’t finding which engineer made the git commit mistake, but finding out why our system allowed that mistake to impact users broadly. We use tracing and log data chronologically to analyze architectural design weaknesses. The post-mortem results must produce concrete system improvement plans, including adding new instrumentation to system areas previously dark from monitoring (telemetry gaps).

Comprehensive Comparison: When to Use What #

We don’t need to throw away our old monitoring systems when adopting observability. Both complement each other and have their own roles in the system operational lifecycle.

• Use Monitoring For:
  • NOC Monitoring Stations: Displaying server health status, global network availability, and external SLA compliance.
  • Resource Capacity Alarms: Sending notifications when database disk capacity drops to 15% remaining or when SSL licenses are about to expire.
  • Rough Cost Monitoring: Overseeing monthly cloud bill consumption against project budgets.
• Use Observability For:
  • Complex Incident Debugging: Tracing unpredictable micro-interaction problems in microservices systems.
  • New Feature Impact Analysis: Measuring whether architectural changes or new code releases cause subtle performance degradation (silent performance degradation).
  • Architecture & Latency Optimization: Using trace flamegraph analysis to identify components hindering overall transaction performance.

Summary #

• Monitoring focuses on anticipated failure conditions (known unknowns), while observability focuses on solving never-before-seen problems (unknown unknowns).
• Traditional monitoring approaches fail in the cloud-native era due to context-removing data aggregation and ephemeral container nature.
• The three observability pillars must interconnect — Metrics to detect anomalies, Tracing to isolate failure locations, and Logs to identify root causes in detail.
• High Cardinality characteristics are the main key to observability — Enabling data query filtering by unique attributes like user_id without breaking monitoring systems.
• Use OpenTelemetry as the open instrumentation standard to avoid vendor lock-in with specific monitoring platform providers.
• Observability culture involves the entire development lifecycle — From writing instrumentation code early, SLO-based performance evaluation, to constructive incident review sessions (blameless post-mortems).

← Previous: Alerting   Next: Cloud Pricing Model →