Load Balancing #

In large-scale cloud system engineering, the Load Balancer acts as the front-line gateway and a vitally important network traffic police officer. Generally, a load balancer receives all data packets and incoming connections from the public internet and distributes them fairly and evenly across several backend server instances (target group) behind it. Without a load balancer, we could never run more than one server instance for a service — all traffic would pile up in one place. The load balancer is the absolute foundation enabling horizontal scaling, high availability, auto-healing, and zero-downtime deployments to materialize effectively.

The Core Problems Solved by a Load Balancer #

To understand the urgency of a load balancer, let’s compare a system architecture without one against the modern architecture that uses it:

1. System Without a Load Balancer (Single Node) #

In a system without load balancing, our application’s domain name in the DNS server maps directly to the single application server’s public IP address.

flowchart LR
    User["User (Internet)"] -->|"DNS resolve: 54.123.45.1"| ServerA["Server A (Only VM)"]

Operational Problems:

• Compute Bottleneck: Once user traffic spikes, Server A runs out of CPU and RAM, causing slow system responses until crash.
• Single Point of Failure (SPOF): If Server A dies from physical hardware failure in the cloud, our entire application dies instantly.
• Scaling Hurdle: We can’t split compute load to new servers because DNS only records Server A’s single IP address.

2. System with a Load Balancer (Multi Node) #

By placing a Load Balancer in front, our application’s DNS domain points to the Load Balancer’s IP address (or DNS name).

flowchart TD
    User["User (Internet)"] -->|"DNS resolve: Load Balancer IP"| LB["Load Balancer (Managed Gateway)"]
    LB --> ServerA["Server A (Private IP)"]
    LB --> ServerB["Server B (Private IP)"]
    LB --> ServerC["Server C (Private IP)"]

Operational Advantages:

• Even Load Distribution: Traffic load is fairly divided across Servers A, B, and C.
• High Resilience: If Server B crashes, the Load Balancer detects the failure and automatically redirects data traffic to Servers A and C, unnoticed by users.
• Scaling Flexibility: We’re free to add or remove backend server counts at any time (auto-scaling) behind the Load Balancer.

Layer 4 vs Layer 7 Load Balancers: Speed vs Intelligence #

Based on the OSI Model (Open Systems Interconnection), load balancers are divided into two main types based on how deeply they inspect data packets:

1. Layer 4 Load Balancing (Transport Layer) #

Layer 4 load balancers (L4) work at the transport protocol level (TCP/UDP). L4 doesn’t read the payload of HTTP/HTTPS packets users send; it only checks the sender/receiver IP addresses and network ports in the TCP/UDP packet headers.

• Mechanism: Once a TCP connection arrives, L4 immediately picks one backend server and forwards the data packet there (packet forwarding).
• Advantages: Very fast with minimal CPU usage (low latency), because no SSL/TLS decryption or HTTP parsing is needed.
• Disadvantages: Can’t route traffic based on URL paths, HTTP headers, or cookies.
• Technology Examples: AWS Network Load Balancer (NLB), HAProxy (TCP mode).

2. Layer 7 Load Balancing (Application Layer) #

Layer 7 load balancers (L7) work at the application level (HTTP/HTTPS/WebSockets). L7 reads and understands HTTP traffic content deeply.

• Mechanism: L7 accepts TCP connections from clients, decrypts SSL/TLS, analyzes HTTP request structure (URL paths, headers, cookies), then creates new TCP connections to forward requests to the right backend.
• Advantages: Has very smart routing features (content-based routing) and supports Web Application Firewall (WAF) integration.
• Disadvantages: Requires higher CPU processing to handle encryption decryption processes.
• Technology Examples: AWS Application Load Balancer (ALB), Nginx (HTTP mode), Traefik.

Content-Based Routing at Layer 7 #

The main power of a Layer 7 Load Balancer is its ability to route data traffic based on application-level information. This lets us consolidate hundreds of microservices under a single domain address.

flowchart TD
    LB["L7 Load Balancer<br>(api.example.com)"]
    
    LB -->|"Rule 1: Path /api/v1/users/*"| UserSvc["User Service"]
    LB -->|"Rule 2: Path /api/v1/orders/*"| OrderSvc["Order Service"]
    LB -->|"Rule 3: Header 'X-Canary: true'"| CanaryApp["Canary App Version"]
    LB -->|"Rule 4: Default"| FrontendApp["Frontend Web App"]

L7 Routing Types: #

• Path-Based Routing: Routing traffic by URL path (e.g., /api/v1/users goes to the User Service, while /api/v1/orders goes to the Order Service).
• Host-Based Routing: Using one load balancer to serve multiple domains at once (e.g., requests to api.example.com go to the API backend, while admin.example.com goes to the admin panel).
• Header-Based & Cookie Routing: Routing traffic by special headers (e.g., if a request has an X-Developer: true HTTP Header, route it to the staging environment, or use cookies to route users to a Canary application version).

Load Balancer Traffic Distribution Algorithms #

To decide which server a user request goes to, the load balancer uses the following distribution algorithms:

1. Round Robin: Distributes requests one by one to each backend server in turn. Suitable when backend server specifications are equal.
2. Weighted Round Robin: A Round Robin modification where we assign weights to servers with larger hardware capacity so they receive a bigger share of requests.
3. Least Connections: Routes new requests to the server currently handling the fewest active connections. Ideal for requests with varying processing durations.
4. IP Hash: Hashes the client IP address to lock users to one backend server (Session Affinity/Sticky Sessions). However, this pattern is an anti-pattern for cloud-native applications because it breaks even load distribution.
5. Consistent Hashing: An advanced algorithm minimizing data redistribution when backend servers are added or removed, very important in distributed caching design.

Health Checks: The Backend Health Detection Mechanism #

Health Checks are the heart of load balancer reliability. The load balancer periodically sends HTTP requests or TCP pings to every backend server to ensure it’s still functioning normally.

Health Check Threshold Configuration:
  - Interval: 15 seconds (Wait time between pings).
  - Timeout: 3 seconds (Server response tolerance limit).
  - Unhealthy Threshold: 3 times (3 consecutive failures = Server marked Unhealthy).
  - Healthy Threshold: 2 times (2 consecutive successes = Server marked Healthy again).

If Server A fails to respond 3 consecutive times, the Load Balancer immediately removes Server A from the traffic rotation. Once Server A is healthy again (e.g., after completing an automatic restart), the Load Balancer automatically returns it to rotation.

SSL/TLS Management Patterns: Termination vs Passthrough #

The load balancer also acts as a security decryption gateway using two main patterns:

1. SSL/TLS Termination (SSL Offloading) #

Encrypted HTTPS connections from clients are decrypted at the Load Balancer level. Traffic from the Load Balancer to backend server instances is forwarded using plain HTTP over the trusted private VPC network.

• Advantages: Saves backend server CPU usage because they don’t need to perform TLS decryption cryptographic computation. SSL/TLS certificate management becomes centralized in one place (the Load Balancer), simplifying automatic certificate renewal.

2. SSL/TLS Passthrough #

The Load Balancer passes encrypted data packets directly to backend servers without decryption at the Load Balancer level. Decryption is done independently on each backend server.

• Advantages: End-to-end security. Useful for industries with extremely strict compliance laws (like finance) prohibiting data decryption at intermediary levels.
• Disadvantages: The Load Balancer can’t do Layer 7 routing because the HTTP content is encrypted.

Auto-Scaling Integration: Connection Draining & Target Tracking #

To support dynamic auto-scaling agility, the Load Balancer integrates tightly with Auto-Scaling Groups (ASGs) through two critical features:

1. Connection Draining (Deregistration Delay) #

When the auto-scaling system detects declining traffic and decides to terminate a VM instance (scale-in), or when an instance is marked unhealthy and will be destroyed, the Load Balancer must not cut connections unilaterally.

• Mechanism: The Load Balancer places the instance in Draining status. It immediately stops sending new requests to that instance, but gives a grace period (default 300 second timeout) for the instance to finish active in-flight transactions. After the time expires or all connections finish processing, the instance may be physically terminated without corrupting user transactions.

2. Target Tracking Scaling Policy #

The Load Balancer provides special real-time metrics that can trigger auto-scaling.

• Primary Metric: ALBRequestCountPerTarget (Requests per server). We can set a rule: “If the average request count per instance exceeds 1000 requests per minute, immediately deploy 2 new VM instances horizontally.”

Code Example: Application Load Balancer Configuration with Terraform #

Here’s an example Terraform declaration for deploying an Application Load Balancer (ALB), a Target Group with Health Check parameters, an HTTPS Listener (Port 443), and Listener Rules for path-based routing:

# ✓ CORRECT: Use Terraform to design an L7 Load Balancer with path-based routing

# 1. Create the Application Load Balancer (ALB)
resource "aws_lb" "main_alb" {
  name               = "production-alb"
  internal           = false # Internet-facing
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb_sg.id]
  subnets            = [aws_subnet.public_az1.id, aws_subnet.public_az2.id]
}

# 2. Target Group for the API Service
resource "aws_lb_target_group" "api_tg" {
  name     = "api-target-group"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  # Periodic Health Check configuration
  health_check {
    path                = "/api/v1/healthz"
    protocol            = "HTTP"
    matcher             = "200"
    interval            = 15
    timeout             = 3
    healthy_threshold   = 2
    unhealthy_threshold = 3
  }
}

# 3. Target Group for the Web Frontend
resource "aws_lb_target_group" "web_tg" {
  name     = "web-target-group"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  health_check {
    path                = "/"
    protocol            = "HTTP"
    matcher             = "200"
    interval            = 30
    timeout             = 5
    healthy_threshold   = 2
    unhealthy_threshold = 3
  }
}

# 4. HTTPS Listener (Port 443) with SSL Certificate
resource "aws_lb_listener" "https_listener" {
  load_balancer_arn = aws_lb.main_alb.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-2016-08"
  certificate_arn   = "arn:aws:acm:ap-southeast-1:123456789012:certificate/abc-123-xyz" # Dummy ARN

  # Default Action: Send traffic to the Web Frontend
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.web_tg.arn
  }
}

# 5. Path-Based Routing Rule
resource "aws_lb_listener_rule" "api_routing" {
  listener_arn = aws_lb_listener.https_listener.arn
  priority     = 10 # Smallest priority rules are evaluated first

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.api_tg.arn
  }

  # URL Evaluation: If the request starts with /api/, route to the API Target Group
  condition {
    path_pattern {
      values = ["/api/*"]
    }
  }
}

Summary #

• A load balancer is a prerequisite for horizontal scaling — traffic can’t be distributed across multiple instances without a centralized load balancer.
• Layer 4 NLBs are very fast and suit game servers/databases, while Layer 7 ALBs are smart and ideal for web app routing.
• Content-based routing at Layer 7 allows separating microservice traffic by URL path, host domain, or HTTP headers.
• Health checks monitor server health in real-time to automatically remove crashed servers from data traffic rotation.
• SSL Termination at the Load Balancer centralizes certificates and lightens backend server CPU loads from TLS decryption.
• Connection Draining (Deregistration Delay) prevents cutting active user requests during scale-in or server replacement.
• Avoid sticky sessions when possible so traffic distributes evenly across all backends.

← Previous: NAT, Firewall, Security Group   Next: Object Storage →