Logging #

In distributed IT systems, especially in cloud computing environments, Logging is our eyes and ears. Every incoming HTTP request, every slow database query, every failed payment transaction, and every OS crash — all these events are recorded as log lines. In the traditional single-server era (single virtual private server), we could easily SSH into the server and read log files directly using the tail -f command. However, in modern cloud architecture involving hundreds of containers dynamically born and dying, those traditional logging systems are no longer sufficient. We need a centralized logging architecture capable of collecting, filtering, indexing, and searching millions of log lines in real-time. Well-structured logs are our main helper during system outages, while messy logs just become junk data that lengthens recovery time (Mean Time to Resolution / MTTR).

Structured Logging vs Plain Text #

The biggest difference separating amateur logging architecture from enterprise-class is the log writing format.

1. Plain Text Logs (Unstructured) #

The traditional log format is a single line of plain text manually combined using string manipulation.

• ANTI-PATTERN: Writing log lines as ordinary sentences concatenated with data variables.
• Problem: These logs are very hard to automatically parse by log collection engines. We’re forced to write very complex, error-prone regular expressions (Regex) to extract data like User IDs or processing durations. If a developer changes the spacing or sentence grammar in the latest application version, all our regex filters break instantly.

2. Structured Logs (Structured JSON) #

The modern log format where every log line is written as a single-line JSON object (JSON Lines or JSONL).

• CORRECT: Packaging all important metadata into structured key-value pairs.
• Advantages: Any log search engine (like Elasticsearch, OpenSearch, or Grafana Loki) can automatically index each key. We can easily perform complex filter searches like: “Show all logs from the ‘payment-service’ with the ‘payment_failed’ event where the amount is above Rp 1,000,000 over the last hour”.

// ANTI-PATTERN: Plain Text Log (Unstructured sentence)
2026-06-21 12:30:45 [ERROR] Failed to process transaction order_id 98765 for user 12345. Error: Insufficient balance. Duration: 245ms.

// --- Separator ---

// CORRECT: Structured JSON Log (Structured & Easily Indexed)
{
  "timestamp": "2026-06-21T12:30:45.123Z",
  "level": "ERROR",
  "service": "payment-service",
  "event": "payment_failed",
  "user_id": "12345",
  "order_id": "98765",
  "duration_ms": 245,
  "error_code": "INSUFFICIENT_FUNDS",
  "error_message": "Insufficient balance",
  "trace_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "environment": "production"
}

Anatomy and Standard Log Field Schema #

To ensure search consistency across all microservices, all development teams must agree on a standard field schema that must exist in every JSON log line:

1. Core Fields #

• timestamp: The precise time the event occurred at the application level. Format must use the ISO 8601 UTC standard (e.g., 2026-06-21T12:30:45.123Z), not local server time, to simplify chronological correlation during cross-region incidents.
• level: Log severity category (DEBUG, INFO, WARN, ERROR, FATAL).
• service: The name of the microservice triggering the log (e.g., auth-service, cart-service).
• message: A short, human-readable description of the event.

2. Context & Correlation Fields #

• trace_id: A unique ID inherited from the user’s initial request as it passes through the API Gateway down to the farthest microservice. This trace ID links all logs scattered across dozens of different servers for the same request transaction.
• user_id / tenant_id: Identifies the user or corporate client affected by the event.
• environment: Marks the origin environment (development, staging, production).

3. Performance & Error Fields (Telemetry Fields) #

• duration_ms: Operation processing duration in milliseconds.
• error_code & stack_trace: Machine error codes and code memory stack dumps on crashes (only at ERROR and FATAL levels).

// Example structured logger initialization in Node.js using Winston (CORRECT)
const winston = require('winston');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json() // Automatically converts logs to JSON format
  ),
  defaultMeta: { service: 'order-service', environment: 'production' },
  transports: [
    new winston.transports.Console() // ✓ CORRECT: Write to Console (stdout)
  ]
});

// Usage in code
logger.error('Order checkout process failed', {
  order_id: '98765',
  user_id: '12345',
  error_code: 'STOCK_OUT',
  duration_ms: 120,
  trace_id: '9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d'
});

Log Level Discipline #

Indisciplined severity level (log level) grouping triggers alert fatigue for operations teams. We must separate log levels according to the following guidelines:

Centralized Log Delivery Architecture #

In cloud environments, our application containers are ephemeral. If a Kubernetes container crashes or is destroyed by auto-scaling processes, all log files stored on the container’s local disk vanish forever.

Therefore, our applications must obey the Twelve-Factor App rule: “Applications must not manage their own log files. Applications may only write logs to standard output (stdout) and standard error (stderr) as event streams” without thinking about where the logs will be sent.

flowchart TD
    subgraph PodContainer ["Application Pod Container"]
        App["Application Code (Write JSON to stdout)"]
    end
    
    subgraph K8sNode ["Kubernetes Worker Node VM"]
        Engine["Container Engine (Docker/Containerd)"]
        LogFile["Node Local Disk (/var/log/containers/*.log)"]
        Agent["Log Shipper Agent (Fluent Bit / Vector / Filebeat)"]
        
        App -->|"stdout stream"| Engine
        Engine -->|"Temporary storage"| LogFile
        Agent -->|"Tail & read files non-blocking"| LogFile
    end
    
    subgraph Aggregator ["Centralized Log Analytics System"]
        SearchDB["Log Database (OpenSearch / Grafana Loki / Datadog)"]
        UI["Visualization Dashboard (Kibana / Grafana)"]
        
        Agent -->|"Batch send via HTTPS (Batch & Compress)"| SearchDB
        SearchDB --> UI
    end

Log Delivery Pipeline Mechanism: #

1. stdout Capture: The container writes JSON logs to the console (stdout). The container engine system (like containerd) captures that output and stores it as temporary text files on the VM host server’s local disk (/var/log/containers/).
2. Log Agent Tail: A lightweight log agent (Log Shipper like Fluent Bit, Vector, or Filebeat) runs as a daemon on the VM host server. This agent reads local log files non-blocking, adds extra metadata (like the Kubernetes Pod name, container ID, VM host name, and cloud region name), then ships them to the central database.
3. Aggregation & Indexing: The managed central database (like OpenSearch, Datadog, or Grafana Loki) receives the log stream, compresses data, indexes JSON fields, and displays them on visualization dashboards (like Kibana or Grafana) for our engineering teams to query.

Sensitive Data Classification: What Must Not Go into Logs #

Logs are plain text files. Storing confidential information or sensitive customer data in logs is a fatal data leakage hole. Security compliance audit teams (like PCI-DSS or HIPAA) can immediately revoke our company’s security certifications if they find sensitive data stored in logs.

Data Strictly Forbidden from Logs: #

• Credentials: Plain-text passwords, access keys, secret tokens, PINs, and session tokens.
• Financial Data: Full credit card numbers (PAN), card expiration dates, and CVV/CVC numbers.
• PII (Personally Identifiable Information): National ID numbers, full home addresses, and mobile phone numbers. Just use user_id or do one-way hashing (SHA-256) if data correlation is needed.
• Medical Data: Patient medical records, disease diagnoses, and prescriptions (HIPAA law violations).

// Example JavaScript middleware automatically masking passwords before writing logs (CORRECT)
const maskSensitiveData = (data) => {
  const jsonString = JSON.stringify(data);
  
  // Regex searching for password, cvv, or token keys and masking their values
  const maskedString = jsonString.replace(
    /("(password|cvv|access_token)":\s*")([^"]+)(")/gi, 
    '$1[REDACTED]$4'
  );
  
  return JSON.parse(maskedString);
};

// Usage in HTTP Request Logging
app.use((req, res, next) => {
  // Mask the request body before writing logs
  const safeBody = maskSensitiveData(req.body);
  logger.info('HTTP Request received', {
    path: req.path,
    method: req.method,
    payload: safeBody // ✓ CORRECT: Payload is clean of sensitive data
  });
  next();
});

Log Retention and Cost Control Strategies #

Storing all log data for years in a fast-search database triggers cloud bill explosions. We must apply multi-tiered retention policies:

Log Storage Tier Division #

Logging Cost Reduction Strategies: #

1. Log Sampling: For very high-capacity successful requests (like load balancer health-check endpoints returning HTTP 200 every 5 seconds), we don’t need to record 100% of the logs. Just record a 1% sample of successful requests, but still record 100% of logs if requests fail (HTTP 5xx).
2. Severity Filtering: Make sure DEBUG-level logs never reach production environments. Configure log agents to discard log lines below INFO level before shipping over the network.

Summary #

• Structured JSON Logging is mandatory so log search engines can automatically index and precisely filter queries.
• Applications may only write logs to stdout/stderr per the Twelve-Factor App principle, letting Log Shipper Agents manage log delivery.
• Timestamps must use the ISO 8601 UTC format to align chronological event synchronization during cross-server/region incidents.
• Use Log Levels with discipline — Disable DEBUG in production, limit ERROR to business transaction failures, and use FATAL when applications crash.
• Apply automatic masking sensors at the middleware level to prevent secret, financial, and customer PII data leaks.
• Apply multi-tiered retention policies (Hot, Warm, Cold) to balance operational debugging needs with cloud storage cost efficiency.

← Previous: Zero Trust   Next: Metrics →