Metrics #

To maintain the reliability of large-scale cloud systems, we must have quantitative metrics continuously measuring infrastructure and application health status. Metrics are collections of numerical measurements gathered periodically (e.g., every 10 or 15 seconds) representing system conditions over time. Unlike logs, which record details of discrete individual events, metrics present statistical aggregation data — like how many HTTP requests come in per second, what percentage of average CPU utilization is used, or how many milliseconds of response latency our application has. Metrics are the fastest, most efficient indicator for detecting whether a system problem exists (detection), before we finally dive into log files or trace runs to identify what causes the problem (investigation). This article discusses metric vs log differences, metric type classifications, Google SRE monitoring frameworks, and code instrumentation using the OpenTelemetry industry standard.

Fundamental Differences: Metrics vs Logging #

Although metrics and logs are both observability pillars, they’re designed to solve two different problems:

1. Logging (“Finding the Needle in a Haystack”) #

Logs record detailed individual event histories.

• Advantages: Very high information detail (stack traces, request payload details).
• Limitations: Consumes huge storage space and high compute costs to query during heavy traffic loads.

2. Metrics (“Monitoring River Water Levels in Real-time”) #

Metrics ignore individual details and only focus on lightweight numeric aggregate values.

• Advantages: Very small storage consumption, very fast query processing, and very easy to visualize as trend charts on dashboards.
• Synergistic Combination: Metric dashboards trigger Alerts when detecting error rate spikes beyond safe limits, then engineers use the Trace ID from those metrics to find specific log lines for fixing.

Metric vs Log Comparison Table #

Metric Data Types #

To represent various system conditions, we use three main metric data types agreed upon by the Prometheus and OpenTelemetry standards:

1. Counter (Cumulative Metric) #

A Counter is a metric data type whose value can only increase (go up) or return to zero (reset) when the application restarts (reboots). Counters never decrease.

• Usage: Counting total HTTP requests (http_requests_total), total errors (http_errors_total), or successful transaction counts.
• Rate Calculation: The absolute counter value isn’t very useful on dashboards. What we need is calculating the counter’s rate of change within a specific time unit: $$\text{RPS (Requests Per Second)} = \frac{\Delta\text{Counter}}{\Delta t}$$

2. Gauge (Fluctuating Metric) #

A Gauge is a metric data type whose value can go up or down freely, representing the instant condition at one specific time.

• Usage: Measuring CPU temperature (cpu_temperature), memory usage in bytes (memory_used_bytes), active database connection counts, or message queue depth.

Metric Value Characteristic Visualization:

Counter (Only Increases):
   100  │                                    ┌───
    80  │                            ┌───────┘
    60  │                    ┌───────┘
    40  │            ┌───────┘
    20  │    ┌───────┘
        └─────────────────────────────────────────> Time

Gauge (Can Go Up & Down):
   100  │            ┌──┐
    80  │    ┌───────┘  │            ┌──┐
    60  │    │          └────┐       │  └──────┐
    40  │────┘               └───────┘         │
    20  │                                      └──
        └─────────────────────────────────────────> Time

3. Histogram (Value Distribution & Percentiles) #

Histograms measure the frequency distribution of durations or data sizes into predefined bucket groups. Histograms are crucial for measuring application response latency.

flowchart TD
    Request["Request Latency (e.g., 120ms)"] --> BucketCheck{"Evaluate Buckets"}
    BucketCheck -->|"< 50ms"| B1["Bucket: le_50 (Count +0)"]
    BucketCheck -->|"< 100ms"| B2["Bucket: le_100 (Count +0)"]
    BucketCheck -->|"< 250ms"| B3["Bucket: le_250 (Count +1)"]
    BucketCheck -->|"< 500ms"| B4["Bucket: le_500 (Count +1)"]

The Average Latency Trap #

Using the average/mean value to measure application latency is a fatal mistake that often hides problems in the distribution tail.

• Scenario: We receive 100 requests:
  • 95 requests finish very fast in 50ms.
  • 4 requests finish in 200ms.
  • 1 request hits a bottleneck and finishes in 5,000ms (5 seconds).
• Average Calculation: $$\text{Average} = \frac{(95 \times 50) + (4 \times 200) + 5000}{100} = 105.5\text{ ms}$$ The 105.5ms average chart looks very safe and healthy on our dashboard. However, in reality, 1% of our users experience extreme 5-second slowness!
• Solution (Percentiles):
  • P50 (Median): 50% of requests finish under 50ms.
  • P95: 95% of requests finish under 50ms.
  • P99: 99% of requests finish under 5,000ms. By monitoring the P99 metric, the extreme 5-second latency spike in the distribution tail is immediately detected on the dashboard.

Monitoring Frameworks: Four Golden Signals, RED, and USE #

To simplify designing meaningful dashboards without information overload, we can adopt the following three industry-standard frameworks:

1. Four Golden Signals (Google SRE Book) #

• Latency: The time needed to complete a request. We must separate successful request latency from failed request latency.
• Traffic: Measures how much request load is on the system (e.g., RPS, transactions per minute).
• Errors: The failure ratio of incoming requests (e.g., HTTP 5xx count divided by total requests).
• Saturation: Measures how full our resource capacity is (e.g., CPU utilization, remaining memory space, thread pool limits).

2. RED Method (Service-Oriented / Microservices) #

Very suitable for monitoring stateless API application performance:

• Rate: The number of requests per second received.
• Errors: The number of requests that fail triggering errors.
• Duration: The response latency duration time.

3. USE Method (Infrastructure-Oriented / Hardware) #

Very suitable for monitoring the health of server hardware, databases, or disk volumes:

• Utilization: The percentage of resource capacity actively in use (e.g., 75% CPU used).
• Saturation: The number of queued jobs waiting for that resource to free up (e.g., disk queue length or I/O wait).
• Errors: The number of failures or errors reported by that hardware.

Custom Business Metrics #

Technical metrics (like CPU and RAM) can sometimes deceive us. There are times when a web server reports very low CPU utilization (10%) and 0% error rate (healthy), yet our application is actually experiencing a logic bug where the e-commerce checkout button can’t be clicked, so no transactions enter the database at all.

Implementing Custom Business Metrics is the only way to detect these logic anomalies in real-time:

• E-commerce: orders_placed_per_minute (If this number suddenly drops from an average of 50 orders per minute to 0, the system alarm must sound immediately even though web server utilization is healthy).
• Fintech: payment_gateway_success_ratio (Detects bank API failures before the bank makes an official announcement).
• SaaS: active_user_connections_per_hour (Detects global login problems).

Application Instrumentation Using OpenTelemetry #

OpenTelemetry (OTel) is an open-source standard supported by the CNCF for collecting metrics, logs, and traces in a vendor-neutral way. By writing instrumentation code with the OpenTelemetry SDK, we’re free to send our application metrics to any monitoring backend (like Prometheus, Datadog, Grafana Loki, or Dynatrace) without changing application code again.

Here’s a complete example of metric instrumentation in Go recording request counters and latency histograms:

// Example HTTP server metric instrumentation in Go using the OpenTelemetry SDK (CORRECT)
package main

import (
	"context"
	"net/http"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/metric"
)

var (
	meter          = otel.Meter("http-server-meter")
	requestCounter metric.Int64Counter
	requestLatency metric.Float64Histogram
)

func initMetrics() {
	var err error
	// Initialize the Counter
	requestCounter, err = meter.Int64Counter("http_requests_total",
		metric.WithDescription("Total number of HTTP requests entering the server"),
	)
	if err != nil {
		panic(err)
	}

	// Initialize the Histogram for measuring latency durations
	requestLatency, err = meter.Float64Histogram("http_request_duration_seconds",
		metric.WithDescription("Distribution of HTTP request durations in seconds"),
		metric.WithExplicitBucketBoundaries(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
	)
	if err != nil {
		panic(err)
	}
}

func httpHandler(w http.ResponseWriter, r *http.Request) {
	startTime := time.Now()
	ctx := context.Background()

	// Define label attributes for query correlation
	attrs := []attribute.KeyValue{
		attribute.String("method", r.Method),
		attribute.String("path", r.URL.Path),
	}

	// Simulate request processing
	w.WriteHeader(http.StatusOK)
	w.Write([]byte("Metrics Recorded!"))

	// 1. Increment the counter value (+1)
	requestCounter.Add(ctx, 1, metric.WithAttributes(attrs...))

	// 2. Record latency duration into the histogram
	duration := time.Since(startTime).Seconds()
	requestLatency.Record(ctx, duration, metric.WithAttributes(attrs...))
}

func main() {
	initMetrics()
	http.HandleFunc("/api/data", httpHandler)
	http.ListenAndServe(":8080", nil)
}

Summary #

• Metrics measure aggregate time-series numeric data — Very lightweight to consume, cheap to store for long retention, and very fast to query.
• Three basic metric types must be understood — Counters for cumulative counts that only increase, Gauges for instant fluctuating values, and Histograms for grouping data frequencies.
• Don’t use averages (means) for latency — Averages hide extreme slowness in distribution tails; monitor P95 or P99 percentiles instead.
• Apply the Google SRE Four Golden Signals — Latency, Traffic, Errors, and Saturation to universally map application health.
• Use the RED framework for microservices (Rate, Errors, Duration) and the USE framework for infrastructure (Utilization, Saturation, Errors).
• Design Custom Business Metrics — To detect business process logic bugs that don’t trigger system error alarms at the server/infrastructure level.
• Use the OpenTelemetry SDK for agnostic code instrumentation to avoid monitoring vendor lock-in.

← Previous: Logging   Next: Tracing →