Tracing #

In traditional monolithic software development, tracking transaction failures or performance degradation is relatively easy because all code execution runs in the same memory process. On errors, we can easily read the stack trace to find the broken code line. However, when we move to microservices architecture, one simple user request (like pressing the “Checkout” button) can cross 5, 10, or even dozens of independent microservices running in separate containers, written in different programming languages, and managed by separate development teams. A stack trace from one service will never be enough to diagnose end-to-end problems. We need Distributed Tracing technology to map and visualize the complete request journey across all inter-service network boundaries. This article discusses microservices diagnosis challenges, trace and span concepts, context propagation mechanisms, and utilizing the OpenTelemetry industry standard.

Diagnosis Challenges in Microservices Architecture #

Imagine a classic e-commerce platform scenario: A user reports their checkout transaction periodically fails with HTTP 500 errors.

1. Scenario Without Distributed Tracing #

Without distributed tracing, engineering teams must open centralized log systems and manually piece together the puzzle:

• Check API Gateway logs: Request came in, 500 response sent to client.
• Check Order Service logs: Successfully created an order draft.
• Check Payment Service logs: Returns a bank API connection error status.
• Problem: Without a unique identifier explicitly binding all these logs, engineering teams must guess transaction matches based on timestamps that are often not precisely synchronized between different servers. We can’t answer with certainty: How long was the wait at each service? Which service is the main bottleneck? And are there inefficiently retried operations?

2. Scenario With Distributed Tracing #

Distributed tracing unites the entire history of cross-network API call journeys into a single visual flow diagram (Gantt Chart) precisely mapping hierarchical relationships between services end-to-end.

flowchart TD
    subgraph TraceTree ["Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736 (Total: 2.1s)"]
        GW["API Gateway (Span 1: 0 - 2.1s)"]
        Order["Order Service (Span 2: 10 - 250ms)"]
        Inventory["Inventory Check (Span 3: 20 - 110ms)"]
        Payment["Payment Service (Span 4: 250ms - 2.1s)"]
        BankAPI["Bank External API (Span 5: 300ms - 2.0s - BOTTLENECK)"]
        
        GW --> Order
        Order --> Inventory
        GW --> Payment
        Payment --> BankAPI
    end
    
    style BankAPI stroke:#d32f2f,stroke-width:2px

Fundamental Concepts: Traces and Spans #

Distributed tracing is based on two main logical data concepts:

1. Trace #

A Trace is the complete workflow representation of a single request running through our entire distributed system. A trace acts as a container with a globally unique identifier called the Trace ID (a 16-byte hexadecimal string). All services participating in serving that request mark their data with the same Trace ID.

2. Span #

A Span is an individual work unit (building block) inside a trace. One span represents a specific work operation with a start_time and specific execution duration.

• Span Examples: One SELECT query to a database, one HTTP client call to a third-party API, or a JWT token verification process.

Attribute Anatomy Inside a Span #

Context Propagation: The Red Thread Across Services #

For downstream services to know they’re part of a trace started by an upstream service, we must send tracing metadata across network boundaries. This process is called Context Propagation.

Context propagation is done by inserting Trace ID and Span ID details into communication protocol headers (like HTTP Headers or gRPC message metadata).

sequenceDiagram
    autonumber
    participant GW as API Gateway
    participant Ord as Order Service
    
    Note over GW: 1. Receive HTTP Request from Client<br>Generate Trace ID: 4bf92f35...<br>Generate Span ID: 00f067aa...
    
    GW->>Ord: 2. Send HTTP POST /orders<br>Header: traceparent: 00-4bf92f35...-00f067aa...-01
    
    Note over Ord: 3. Extract the traceparent header<br>Inherit Trace ID: 4bf92f35...<br>Set Parent Span ID: 00f067aa...<br>Create New Span: 8b2a1a8c...

Header Format Standard: W3C Trace Context #

In the past, every monitoring vendor had its own proprietary header format (like B3 from Zipkin or the X-Amzn-Trace-Id header from AWS). Today, the industry has agreed on the W3C Trace Context standard with a header format named traceparent:

traceparent format:
00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│  │                                │                └─ Trace Flags (e.g., 01 = Sample Active)
│  │                                └─ Parent Span ID (8-byte hex)
│  └─ Trace ID (16-byte hex)
└─ Version (currently always 00)

When the Order Service receives an HTTP request with that traceparent header, the tracing library inside Order Service extracts the value, sets it as the parent span, and launches a new span under the same trace coordinates.

OpenTelemetry as the Agnostic Observability Standard #

OpenTelemetry (OTel) provides one standard API and SDK for writing distributed tracing instrumentation without dependence on any specific monitoring vendor (vendor-agnostic).

OpenTelemetry Tracing Components: #

1. Tracer Provider: The factory managing the lifecycle of Tracer object creation.
2. Sampler: Determines whether a trace should be fully recorded or discarded to save storage bandwidth.
3. Span Processor: Determines how completed span data is shipped (e.g., sent asynchronously in batch groups).
4. Exporter: Ships collected span data to a central database (like Jaeger, Zipkin, Datadog, or Grafana Tempo).

Here’s a complete example of manual custom span instrumentation in Go using the OpenTelemetry SDK:

// Example manual custom span instrumentation in Go using the OpenTelemetry SDK (CORRECT)
package main

import (
	"context"
	"errors"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
)

var tracer = otel.Tracer("payment-processor-tracer")

func ProcessPayment(ctx context.Context, orderID string, amount float64) error {
	// 1. Start a new span under the running context (automatically inherits parent if any)
	ctx, span := tracer.Start(ctx, "ProcessPaymentTransaction")
	defer span.End() // ✓ CORRECT: Ensure the span ends when the function finishes

	// 2. Set metadata attributes for dashboard search needs
	span.SetAttributes(
		attribute.String("payment.order_id", orderID),
		attribute.Float64("payment.amount", amount),
		attribute.String("payment.gateway", "Stripe"),
	)

	// Simulate payment processing business logic
	err := executeBankTransaction(amount)
	if err != nil {
		// 3. Record the error into the span if the operation fails
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error()) // Set span status to ERROR
		return err
	}

	span.SetStatus(codes.Ok, "Payment Successful")
	return nil
}

func executeBankTransaction(amount float64) error {
	time.Sleep(100 * time.Millisecond) // Simulate a network call
	if amount > 1000000 {
		return errors.New("limit_exceeded: Transaction exceeds bank limit")
	}
	return nil
}

Analyzing Tracing Gantt Charts to Diagnose Bottlenecks #

After trace data is collected to the backend, it’s visualized as a Gantt Chart. When investigating slow performance, we must look for the following critical patterns:

Gantt Chart Visualization (Bottleneck Problem):
API-Gateway   [============================================================] (2.1s)
  Order-Service   [====] (120ms)
    DB-Insert         [==] (60ms)
  Payment-Service        [=================================================] (1.98s)
    Card-Validate        [==] (120ms)
    Bank-API-Call          [=============================================] (1.8s)  <-- BOTTLENECK!

1. Finding the Critical Path & Bottleneck #

Look for the longest span dominating most of the root span’s total duration. In the example visualization above, the total transaction time is 2.1 seconds. We can see the external bank API call (Bank-API-Call) takes 1.8 seconds (85% of total time). This definitively identifies that our performance bottleneck is on the third-party integration side, not our internal database.

2. Sequential vs Parallel Calls #

If we see two child spans (e.g., Card-Validate and Check-Account-Balance) running sequentially, ask the architecture team: “Do these two operations depend on each other? If not, run them in parallel using goroutines/async await to cut the total latency duration.”

3. Inter-Span Gaps (Overhead Gaps) #

If Span A finishes at millisecond 100, but Span B only starts at millisecond 150 with no activity in between, this indicates a hidden gap overhead. This is usually triggered by oversized JSON payload deserialization overhead, or queue wait time on servers before requests are processed.

Tracing Sampling Strategies #

Sending 100% of tracing data for applications with millions of requests per second drastically multiplies our cloud storage billing costs. We must apply Sampling methods:

1. Head-based Sampling #

The decision whether a request will be traced or discarded is made at the start of the workflow (e.g., at the API Gateway level) before the request is processed further.

• Method: Using probability modules (e.g., only record 5% of total traffic randomly).
• Advantages: Very cost-efficient in compute and network bandwidth.
• Disadvantages: If a rare error occurs on a request not selected for tracing, we lose that diagnostic data.

2. Tail-based Sampling #

All spans are first collected into temporary memory (buffers) at the collector level (OpenTelemetry Collector). The decision to keep or discard trace data is made after the entire request finishes executing completely.

• Policy Rules:
  • If a trace ends with ERROR status (HTTP 5xx) ────> Keep 100%.
  • If trace duration exceeds 1.5 seconds (slow) ────> Keep 100%.
  • If a trace runs successfully and fast (normal) ────> Only keep 1% (as a baseline comparison).
• Advantages: Guarantees we always have complete diagnostic data for every error and slowness incident, without wasting budget storing uninteresting normal transaction data.

Summary #

• Distributed tracing maps end-to-end request flows across network boundaries in distributed microservices architectures.
• Traces act as trees stringing together a set of Spans (the smallest work units) using globally unique Trace ID coordination.
• Context propagation uses the traceparent HTTP header (W3C standard) connecting trace information between upstream and downstream services.
• OpenTelemetry is the agnostic observability industry standard freeing us from specific monitoring vendor lock-in.
• Visually analyze trace Gantt Charts to detect critical path bottlenecks, network overhead gaps, and inefficient sequential processing.
• Apply Tail-based Sampling at the OTel Collector level to ensure 100% of error logs are safely stored without wasting budget storing success logs.

← Previous: Metrics   Next: Alerting →