Autoscaling #

Elasticity is the key differentiating characteristic that makes cloud computing far superior to traditional data centers (on-premise). The main service realizing this elasticity in practice is Autoscaling. Autoscaling is an automatic mechanism tasked with dynamically increasing or decreasing compute capacity (like Virtual Machine instances or containers) based on real-world workload metrics, without requiring manual intervention from network administrators. Without autoscaling, we’re forced to choose between two bad decisions: over-provisioning that wastes financial budget, or under-provisioning that risks crashing our application when traffic spikes arrive. This article deeply discusses the autoscaling reconciliation cycle mechanism, scaling policy types, application code architectural prerequisites, and system stability optimization using cooldown periods.

How Autoscaling Works: Control Loop Reconciliation #

Behind the scenes, autoscaling systems (like Auto Scaling Groups on AWS or Virtual Machine Scale Sets on Azure) run as a continuous reconciliation control loop working repeatedly to match running capacity with the desired ideal state.

flowchart TD
    subgraph ControlLoop ["Autoscaling Reconciliation Cycle"]
        direction TB
        Monitor["1. MONITOR METRICS<br>(Collect metrics from CloudWatch/Monitor: CPU, Request Count, Queue Depth)"]
        Evaluate["2. EVALUATE THRESHOLD<br>(Compare metric averages against scaling policy rules)"]
        Decide["3. DECIDE ACTION<br>(Calculate the number of new instances needed or to be released)"]
        Act["4. ACTIVATE CHANGE<br>(Run APIs to launch new VMs or remove old VMs)"]
        
        Monitor --> Evaluate
        Evaluate --> Decide
        Decide --> Act
        Act -->|"Wait for Cooldown to finish"| Monitor
    end

Step-by-Step Cycle: #

1. Monitor: The cloud monitoring system collects telemetry data (e.g., average CPU utilization from all VMs in the group) every 1 minute.
2. Evaluate: The scaling policy rules evaluate metrics. Suppose we have a rule: “Maintain CPU utilization at 60%”. If the system records an average CPU of 85%, this triggers an alarm signal for capacity addition (Scale-out).
3. Decide: The autoscaling service performs mathematical calculations to determine how many additional VM instances must be launched to bring CPU utilization back to the 60% target.
4. Act: The system triggers the cloud compute API to launch new instances, registers them with the Load Balancer, and once those instances are declared healthy, traffic starts being routed to them.

Scaling Policy Types #

We can configure several scaling policy types based on our application’s traffic pattern characteristics:

1. Target Tracking Scaling #

This model works like a room thermostat. We set one specific metric target, and the cloud system independently calculates and adjusts instance counts to maintain that metric’s stability.

• Example: We set the target: CPU Utilization = 50%.
• Calculation: If we currently have 3 VMs with 80% average utilization, the system calculates: $$\text{New Instance Count} = \text{Current Count} \times \left( \frac{\text{Current Metric}}{\text{Target Metric}} \right)$$ $$\text{New Instance Count} = 3 \times \left( \frac{80%}{50%} \right) = 4.8 \approx 5 \text{ Instances}$$ The system automatically adds 2 new VMs to bring the average CPU back near 50%.

2. Step Scaling #

This mechanism lets us create more aggressive gradual responses (steps) when detecting extreme traffic spikes.

• Example:
  • If CPU is between 50% - 70%: add 1 VM.
  • If CPU is between 70% - 85%: add 2 VMs.
  • If CPU > 85%: aggressively add 4 VMs.
• Advantage: Provides very fast reaction time when the application is hit by massive sudden traffic surges.

3. Scheduled Scaling #

This policy is used when we have workload patterns highly predictable by the work calendar.

• Example: An internal company HR application is heavily accessed every workday at 08:00 AM when employees clock in, and quiet on weekends.
• Configuration:
  • Every Monday - Friday at 07:30: raise the group’s minimum capacity to 10 VMs (anticipating load before it happens).
  • Every Monday - Friday at 18:00: lower the minimum capacity to 1 VM (reducing night costs).

4. Predictive Scaling #

Uses the cloud provider’s built-in machine learning algorithms to scan traffic history from the last several weeks, detect recurring daily or weekly patterns, and automatically schedule capacity additions 15-30 minutes before the predicted traffic surge actually arrives.

Scaling Policy Comparison #

Architectural Prerequisites for Effective Autoscaling #

Autoscaling won’t work correctly if the application inside our instances isn’t specifically designed to support elastic architecture. Here are the mandatory architectural prerequisites:

1. Applications Must Be Stateless #

This is the most critical rule. New VM instances launched by autoscaling must serve user requests randomly without depending on sessions stored in the instance’s local memory.

• ANTI-PATTERN: Storing user login session files in VM local RAM. When autoscaling shuts that VM down to reduce capacity, user sessions disconnect and they’re forced to log in again.
• CORRECT: Store user sessions in a centralized shared database or distributed memory caching (like Redis) outside the autoscaling VM cluster.

2. Very Fast Startup Time (Fast Bootstrapping) #

If our VM instances take 10 minutes to boot (because they must download code from the internet, install NPM dependencies, and configure databases at startup), then when traffic spikes arrive, our application dies before new VMs are ready to help serve traffic.

• Mitigation: Use ready-made VM images (Golden Images / AMIs baked with Packer) or container architecture. All code and dependencies must already exist in the image, so VMs/containers are ready to serve traffic in under 30 seconds from power-on.

3. Graceful Shutdown (Connection Draining) #

When traffic load decreases, autoscaling triggers VM removal (Scale-in). We must ensure instances aren’t suddenly killed while still processing user transactions.

• Connection Draining: The Load Balancer must be configured to immediately stop sending new requests to the targeted VM, but give a tolerance window (e.g., 30 seconds) for that VM to finish in-flight transaction requests.

// Example OS termination signal (SIGTERM) handling for Graceful Shutdown in Go
package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	server := &http.Server{Addr: ":8080"}

	go func() {
		if err := server.ListenAndServe(); err != http.ErrServerClosed {
			log.Fatalf("HTTP server ListenAndServe error: %v", err)
		}
	}()

	// Wait for termination signals from the OS Autoscaler (SIGTERM or SIGINT)
	stop := make(chan os.Signal, 1)
	signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
	<-stop

	log.Println("Termination signal received, starting graceful shutdown...")

	// Give a 30-second tolerance window (Connection Draining window)
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	if err := server.Shutdown(ctx); err != nil {
		log.Fatalf("Server Graceful Shutdown failed: %v", err)
	}
	log.Println("Server successfully shut down safely without cutting active connections.")
}

Cooldown Period and Stabilization Mechanisms (Anti-Flapping) #

The biggest operational challenge in autoscaling is controlling stability to prevent an unstable condition called Flapping (Oscillation).

Flapping happens when instance capacity continuously scales out and scales in repeatedly within a short time due to momentary metric fluctuations, wasting VM startup costs and harming application performance.

Flapping Event Chronology (Without Cooldown):
Minute 01: Average CPU 85% ────> Scale-out triggered: add 2 new VMs (total 4 VMs).
Minute 02: New VMs booting, average CPU suddenly drops to 25%.
Minute 02: CPU 25% detected ────> Scale-in triggered: remove 2 new VMs (total 2 VMs).
Minute 03: VMs shut down, CPU rises again to 85% ────> Triggers Scale-out again.
(Endless oscillation triggers overhead and system instability)

The Cooldown Period’s Role #

To prevent oscillation, we must set a Cooldown Period. Cooldown is a pause after a scaling action occurs, during which autoscaling locks the system and refuses to perform new scaling actions even if metrics cross thresholds.

• Scale-out Cooldown (Shorter, e.g., 60-120 seconds): When load rises, we want new servers launched responsively. We give a short cooldown so the system monitors whether the first addition is enough to lower CPU before adding more instances.
• Scale-in Cooldown (Longer, e.g., 300-600 seconds): When load is detected dropping, we want to be careful. We wait at least 5 to 10 minutes to ensure the load drop is genuinely stable and not just a momentary dip, before starting to remove VMs from the cluster.

Choosing the Right Metrics #

Using the wrong metric to trigger autoscaling can lead to system failure. We must match metrics with the main bottleneck type of our application.

1. CPU Utilization #

• Suitable for: Applications doing heavy mathematical calculations or compilation (e.g., video processing servers, database indexing).
• Problem: Not suitable for I/O-bound applications (like Node.js API servers spending most of their time waiting for database queries, where CPU stays low even under heavy traffic).

2. Request Count per Target (RPS) #

• Suitable for: General web servers and RESTful APIs. We set a rule: “Scale-out if traffic exceeds 1000 requests per second per instance”. This metric is very accurate because it directly measures real traffic load.

3. Queue Depth #

• Suitable for: Worker servers processing asynchronous tasks from message queues (like RabbitMQ, AWS SQS, or Kafka).
• Scaling Formula: $$\text{Ideal Instance Count} = \frac{\text{Number of Messages in Queue}}{\text{Target Messages per Worker}}$$ If there are 5,000 messages in the queue and we target each worker processing a maximum of 50 messages at once, the autoscaler automatically sets running capacity to 100 worker instances.

Summary #

• Autoscaling works on a closed control loop constantly monitoring metrics and adjusting instance capacity to match our policy rules.
• Target Tracking policies are the best default option because they automatically calculate the required instance count proportionally, like a thermostat.
• Stateless applications are an absolute prerequisite so instance replacement by autoscaling doesn’t accidentally cut user login sessions.
• Apply Golden Images (pre-baked images) to ensure fast VM startup times (under 30 seconds) when processing traffic surges.
• Set Cooldown Periods to prevent flapping — use short cooldowns for scale-out for quick responses, and long cooldowns for scale-in to maintain capacity stability.
• Choose scaling trigger metrics reflecting real bottlenecks — use Queue Depth for asynchronous processing and Request Count for API servers.

← Previous: Managed Compute   Next: Spot/Preemptible Instance →