Spot/Preemptible Instance #

Inside public cloud providers’ physical data centers, millions of physical servers operate at all times. Because users’ compute resource usage patterns fluctuate (e.g., quiet at night and busy during the day), cloud providers always have large amounts of spare capacity sitting idle. Rather than letting those physical servers run without generating revenue, cloud providers offer them at fantastically discounted prices — ranging 70% to 90% cheaper than normal (On-Demand) rental rates — with one absolute condition: the provider may forcibly reclaim those instances at any time if that physical capacity is needed by full-price-paying On-Demand users. This service is called the Spot Instance (on AWS), Preemptible VM (on Google Cloud), or Spot VM (on Azure). When paired with the right workload types, Spot Instances are the most powerful tool for drastically cutting our compute spending.

How Spare Capacity Works and Its Economics #

Spot Instance pricing is based on the supply and demand dynamics of spare capacity at each cloud provider Availability Zone location.

flowchart TD
    Request["New Compute Request"] --> CheckType{"Request Type?"}
    CheckType -- "1. On-Demand / Reserved (Full Price)" --> Priority["Top Priority:<br>Allocate Capacity Immediately"]
    CheckType -- "2. Spot (Discounted Price)" --> CheckSpare{"Spare Capacity Available?"}
    
    CheckSpare -- "Yes" --> RunSpot["Run Spot Instance<br>(70-90% Discount)"]
    CheckSpare -- "No" --> Reject["Reject Request (Out of Capacity)"]
    
    Priority -->|"If physical capacity runs short, force-reclaim Spot"| Reclaim["Send Interruption Signal & Terminate Spot"]
    Reclaim --> RunSpot

Capacity Allocation Logic #

1. On-Demand Priority: On-Demand users pay full rental rates and get instance availability guarantees (SLA availability guarantee).
2. Remainder Utilization: Spot users utilize idle remaining capacity. Spot rental rates are dynamic, changing periodically based on how much spare capacity remains in that AZ for a specific instance type.
3. Reclaim Process (Interruption): When On-Demand user demand surges (or when spare capacity allocation in that AZ runs thin), the cloud provider’s hypervisor immediately sends a reclaim signal to our Spot instances to terminate them within seconds to minutes.

Understanding Interruption Risks and Termination Signals #

The only trade-off of cheap Spot Instance pricing is lifecycle uncertainty (lifetime volatility). Our instance could run smoothly for weeks without disruption if we use a less popular server type in a quiet AZ, but could also be interrupted within 1 hour of launch if we choose a highly demanded server type in a main AZ.

Interruption Signal Tolerance Times #

When the provider decides to reclaim our Spot physical capacity, the hypervisor doesn’t abruptly kill the VM without notice. We’re given a very short tolerance window for graceful cleanup:

• AWS Spot: Gives 2 minutes notice before termination.
• Google Cloud Preemptible: Gives 30 seconds notice before termination.
• Azure Spot VM: Gives 30 seconds notice before termination.

Coding an Interruption Signal Handler #

This interruption notice is sent by the cloud controller to the instance’s internal metadata server. Applications inside our VM must constantly monitor that metadata server to promptly stop transaction processes, save temporary data (checkpointing), and shut down safely before the server dies.

# Example Bash script to monitor AWS Spot interruption signals from inside the instance
# Run this script as a daemon (background process) inside the VM
#!/bin/bash

METADATA_URL="http://169.254.169.254/latest/meta-data/spot/instance-action"

while true; do
  # Using IMDSv2 Security Token
  TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
  
  # Check whether the spot instance-action endpoint returns HTTP 200 OK
  HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -H "X-aws-ec2-metadata-token: $TOKEN" $METADATA_URL)
  
  if [ "$HTTP_CODE" -eq 200 ]; then
    # Interruption Signal Detected!
    ACTION_DETAILS=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" $METADATA_URL)
    log "WARNING: Spot interruption signal detected! Details: $ACTION_DETAILS"
    
    # 1. Trigger the data rescue script (Save Checkpoint)
    /usr/local/bin/save-app-state.sh
    
    # 2. Notify the Load Balancer to stop traffic (Deregister target)
    /usr/local/bin/deregister-from-lb.sh
    
    # 3. Exit the loop and let the OS shut down
    exit 0
  fi
  
  # Perform periodic checks every 5 seconds
  sleep 5
done

Suitable vs Unsuitable Workloads for Spot Instances #

The key to successfully leveraging Spot Instances is our ability to identify application workload types tolerant of sudden interruption.

Workload Classification Table #

Mixed Strategy: Mixed Instance Groups (Spot & On-Demand) #

To achieve high reliability with minimal cost in production environments, we must not use 100% Spot instances. The best strategy is using Mixed Instance Groups within our Auto Scaling Group.

Safe Production Cluster Composition Visualization:
┌────────────────────────────────────────────────────────┐
│  BURST CAPACITY (70% SPOT INSTANCES)                    │  <-- Saves costs during heavy traffic
│  [Spot VM 1] [Spot VM 2] [Spot VM 3] [Spot VM 4]        │
├────────────────────────────────────────────────────────┤
│  BASELINE CAPACITY (30% ON-DEMAND INSTANCES)           │  <-- Guarantees services never fully die
│  [On-Demand VM 1] [On-Demand VM 2]                     │
└────────────────────────────────────────────────────────┘

Mathematical Cost Savings Simulation #

Let’s calculate the monthly cost efficiency of a mixed strategy managing a cluster of 10 VM instances running 730 hours a month.

• On-Demand Rate: $0.10 / hour
• Spot Rate (80% discount): $0.02 / hour

Option A: 100% On-Demand #

$$\text{Option A Cost} = 10\text{ VMs} \times 730\text{ hours} \times $0.10 = $7,300 / \text{month}$$

Option B: Mixed (30% On-Demand Baseline + 70% Spot Burst) #

• On-Demand cost (3 VMs): $3\text{ VMs} \times 730\text{ hours} \times $0.10 = $2,190$
• Spot cost (7 VMs): $7\text{ VMs} \times 730\text{ hours} \times $0.02 = $1,022$
• Option B Total Cost: $$$2,190 + $1,022 = $3,212 / \text{month}$$

Financial Evaluation Results: #

Using the mixed strategy, we cut the monthly bill by 56% while still ensuring that if the cloud provider mass-reclaims Spot instances, we still have 3 On-Demand VMs standing by serving baseline traffic.

Key to Success: Instance Type Diversification #

When building a Spot cluster, rule number one is: Never rely on a single instance type. If we configure an Auto Scaling cluster requesting only c5.xlarge, and at some point the spare capacity of c5.xlarge in that AZ runs out, our cluster can’t scale out.

• Solution: Enable the instance diversification feature. Configure the group to accept equivalent alternative types, e.g., c5.xlarge, c4.xlarge, c5a.xlarge, and m5.xlarge. The cloud autoscaling system automatically picks the alternative type with the lowest interruption rate at that moment.

Spot Instances in Kubernetes Environments (Spot Worker Nodes) #

Kubernetes is ideal when paired with Spot Instances because Kubernetes was designed from the start to handle dynamic, node-failure-tolerant workloads (fault-tolerant orchestrator).

Inside Kubernetes, we apply Node Pools separation to maintain cluster stability:

# Example pod deployment configuration restricting deployment only to the Spot node pool
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stateless-web-app
spec:
  replicas: 5
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "intent"
                operator: In
                values:
                - "spot-workers" # ✓ CORRECT: Direct pods to the Spot Node Pool
      tolerations:
      - key: "sku"
        operator: "Equal"
        value: "spot"
        effect: "NoSchedule" # Tolerates the NoSchedule taint on spot nodes

Important Kubernetes Configurations for Spot: #

1. Taints and Tolerations: We mark the Spot node pool with a taint (e.g., sku=spot:NoSchedule). This prevents critical system pods (like kube-system databases, CoreDNS, Ingress Controllers) from being accidentally scheduled on unstable Spot nodes. Only stateless application pods with the toleration are allowed to run there.
2. Pod Disruption Budget (PDB): A policy limiting how many pods may die simultaneously during node eviction. This guarantees Kubernetes won’t allow eviction if the remaining active pod count would violate our application’s minimum quorum.
3. K8s Node Termination Handler: A small open-source controller installed in our cluster. This controller monitors cloud provider metadata termination signals. The moment the 2-minute interruption signal is detected, the controller immediately executes automatic node draining commands:

   # Step 1: Mark the node to stop accepting new pods (Cordon)
   kubectl cordon <node-name-spot>
   
   # Step 2: Safely move running pods to other healthy nodes (Drain)
   kubectl drain <node-name-spot> --ignore-daemonsets --delete-emptydir-data
   

   This automatic draining process ensures our application pods have successfully moved to other healthy On-Demand VMs before the Spot VM is force-terminated by the cloud hypervisor.

Summary #

• Spot / Preemptible VMs offer rental discounts up to 90% with the compromise that instances can be force-terminated by the provider at any time.
• Cloud providers give a short tolerance window before interruption — ranging from 30 seconds (GCP/Azure) to 2 minutes (AWS), monitorable through internal metadata servers.
• Ideal for batch processing, CI/CD runners, and ML training — workloads safe to pause briefly and resume through checkpoint systems.
• Don’t use Spot for primary databases — sudden power loss on database servers is highly prone to binary file integrity corruption.
• Use mixed combinations (Mixed Instance Groups) — keep an On-Demand baseline pillar (20-30%) for minimum traffic and use Spot (70-80%) for dynamic traffic.
• Implement Node Termination Handlers in Kubernetes — so clusters automatically execute cordon and drain commands moving pods to other nodes before Spot VMs are removed.

← Previous: Autoscaling   Next: IAM →