Block Storage #

In cloud compute infrastructure, Block Storage is the managed storage type most similar to the physical hard disk drive (HDD) or solid state drive (SSD) in our local computers. Unlike object storage, which presents whole files through HTTP APIs, block storage cuts data into raw binary blocks (raw blocks) of fixed size (for example, 4 KB or 8 KB). These blocks are transferred using special high-speed network protocols (like iSCSI or NVMe-oF) and attached to Virtual Machines as raw disk drive devices (raw block devices). Inside the VM’s OS, we can format this disk with our filesystem of choice (like ext4 on Linux or NTFS on Windows), create partitions, and use it like local storage media with very low latency characteristics and consistent performance.

How Block Storage Works: Network Block Mapping #

Inside the cloud provider’s physical data center, our Virtual Machine instances (Compute Nodes) and physical storage media (Storage Nodes) are usually on different server racks. They’re connected by a very high-speed internal fiber optic network (storage area network).

flowchart TD
    App["Application (Database / Filesystem)"] -->|"1. Read/Write Data (File)"| FS["Filesystem (ext4 / NTFS)"]
    FS -->|"2. Translate to Block Sectors (4 KB)"| OS["OS Block Device Driver (/dev/sdb)"]
    OS -->|"3. Transport Protocol (iSCSI / NVMe-oF)"| Network["Storage SAN Network (Cloud Private Network)"]
    Network -->|"4. Physical Sector Distribution"| StorageBackend["Storage Node (Managed Physical Disks)"]

Read/Write Operation Flow: #

  1. Application Layer: The application (e.g., a PostgreSQL database) performs file write operations.
  2. Filesystem Layer: The OS translates the file into fixed-size binary block chunks on specific sectors.
  3. Device Driver: The OS driver sends the data blocks out through the network card using a data transport protocol (like iSCSI or NVMe-over-Fabrics).
  4. Storage Backend: The cloud provider’s storage controller receives the blocks and redundantly distributes them across several physical disks to guarantee data safety.

This entire network encapsulation process happens below the Virtual Machine’s operating system level. To the OS, this external block storage volume looks and behaves exactly like a physical hard drive plugged directly into the motherboard.

Persistent vs Ephemeral Block Storage #

When launching a Virtual Machine instance in the cloud, we must understand the critical difference between these two storage media types. Choosing the wrong storage type is a leading cause of permanent data loss in the cloud.

1. Ephemeral Storage (Instance Store) #

Ephemeral storage is physical SSD media installed directly inside the physical host chassis where our Virtual Machine runs.

  • Advantages: Extraordinary data transfer speeds (can reach millions of IOPS) with sub-millisecond latency because data packets don’t need to cross the data center’s fiber network.
  • Biggest Danger (Volatility): Data in ephemeral storage is temporary and will be lost forever if our VM instance is stopped, terminated, or automatically moved to another physical server by the cloud provider’s auto-healing system.

Ideal Ephemeral Scenarios: Only use for temporary (scratch data) like caches, buffer memory, temporary files, swap space, or log data already forwarded to centralized log servers.

2. Persistent Block Storage (EBS, Managed Disk, Persistent Disk) #

Persistent block storage is a managed volume whose data lifecycle is completely separate from the Virtual Machine.

  • Advantages: Our data is guaranteed safe even if the VM instance is stopped, rebooted, or destroyed. We can detach the volume from an old VM and attach it to a new VM without losing a single bit of data. The cloud provider also automatically replicates data internally within one Availability Zone to prevent data corruption from physical hard drive failures.

Key Performance Metrics: IOPS, Throughput, and Latency #

To choose the right block storage volume type, we must understand these three performance metrics:

  1. IOPS (Input/Output Operations Per Second): The number of read/write operations a disk can handle in one second. Higher IOPS means a more responsive disk for small random transactions (random read/write), like OLTP transactional databases.
  2. Throughput (MB/s): The volume’s speed in continuously reading or writing large volumes of data (sequential), like during giant database backups or reading long log files. The mathematical relationship is: $$\text{Throughput} = \text{IOPS} \times \text{Block Size}$$
  3. Latency: The time a disk needs to complete one I/O operation (usually measured in milliseconds).

Host Protocol Evolution: SCSI vs NVMe #

In modern infrastructure, cloud providers are moving from the traditional SCSI (Small Computer System Interface) protocol to NVMe (Non-Volatile Memory Express) to connect block storage volumes to hypervisors.

  • SCSI Protocol: A legacy protocol designed for mechanical disks. Has a single queue holding at most 32 commands at once, causing bottlenecks when many CPU cores try to write simultaneously.
  • NVMe Protocol: Designed specifically for high-speed flash SSDs. Supports up to 64,000 parallel queues, each capable of holding 64,000 commands simultaneously. This minimizes lock contention on our multi-core database servers. With NVMe, data transfer latency between the Hypervisor and VM is significantly reduced.

Network Hurdles & EBS-Optimized Instances #

One often-overlooked aspect is the throughput limitation of the Virtual Machine instance itself. If we rent a super-fast SSD volume with 10,000 IOPS but attach it to a very small VM instance (like t3.micro), disk performance gets throttled.

This happens because the network link between the VM and storage backend doesn’t have enough capacity.

  • Solution: Make sure we use VM instance types labeled EBS-Optimized (on AWS). These instances have a dedicated network circuit path for storage I/O traffic, separate from regular internet traffic.

Transparent Encryption at the Hypervisor Level #

Modern persistent block storage supports full Encryption at Rest using KMS encryption keys (AES-256). This encryption and decryption process is handled directly at the physical hypervisor level (host CPU) before data is sent across the SAN network.

  • Performance Impact: Because it’s handled by dedicated hardware at the hypervisor level, this encryption feature doesn’t burden the operating system CPU inside our VM, resulting in nearly 0% latency overhead.

Dynamic Volume Scaling Without Downtime (Elastic Volumes) #

One of the extraordinary advantages of persistent block storage in the cloud is the Elastic Volumes feature. In traditional data centers, enlarging hard disk capacity required shutting down the server, physically plugging in new drives, and reconfiguring RAID groups.

In the cloud, we can enlarge disk capacity (e.g., from 100 GB to 200 GB) directly on an active volume being used by a production VM instance without rebooting or experiencing downtime.

OS-Level Steps to Enlarge Disk Capacity (Linux ext4):

  1. Change the volume capacity in the Cloud Console/API (e.g., from 100GB to 200GB).
  
  2. Use the growpart command to enlarge the local disk partition:
     $ sudo growpart /dev/nvme0n1 1
     
  3. Run resize2fs to expand the filesystem so it recognizes the new space:
     $ sudo resize2fs /dev/nvme0n1p1
     
  4. The new storage space is now ready to use instantly without stopping the application.

Limitation Rules: Cloud providers usually enforce a cooling-off period (e.g., 6 hours on AWS) before we’re allowed to modify size or performance on the same volume again.

Multi-Attach Volumes & the Split-Brain Danger #

Several major cloud providers offer the Multi-Attach feature, which lets one persistent block storage volume be attached to multiple VM instances simultaneously within the same Availability Zone.

flowchart LR
    VM1["Virtual Machine A"] <-->|"Concurrent Mount"| SharedVol["Shared Persistent Volume"]
    VM2["Virtual Machine B"] <-->|"Concurrent Mount"| SharedVol

Critical Data Security Warning: We’re strictly forbidden from attaching a regular block storage volume (like ext4 or XFS) to multiple VMs simultaneously. Standard filesystems are written assuming only one OS coordinator modifies data.

If VM A and VM B write to the same disk sectors without coordination, the Split-Brain phenomenon occurs, causing instant filesystem metadata structure corruption (immediate filesystem corruption).

To use Multi-Attach safely, we must use a Cluster-Aware Filesystem (like GFS2 or OCFS2) with a distributed lock manager mechanism, or use it exclusively for special clustered applications managing disk I/O at the software application level (like Oracle RAC). For most file-sharing needs, File Storage is far safer and recommended.

Snapshots: A Robust Incremental Backup Mechanism #

Snapshots are the primary method for backing up block storage volumes. A snapshot takes a point-in-time copy of the disk state.

How Incremental Snapshots Work: #

  • First Snapshot: The system copies all used data blocks (e.g., 100 GB) and stores them on an encrypted object storage platform behind the scenes.
  • Second Snapshot: The system only tracks changed data blocks (delta changes, e.g., 5 GB). Only those 5 GB of blocks are copied. This makes backups very fast and saves storage costs.
  • Restore Process: Although snapshots are incremental, when we create a new volume from the Second Snapshot, the system automatically reassembles the complete data (100 GB) transparently.

Crash Consistency Note: Before snapshotting an active production database volume, we’re advised to freeze the database’s I/O write process or use OS utilities like fsfreeze so cached data in OS memory is perfectly flushed to disk. This guarantees our snapshot is crash-consistent.

Code Example: Creating and Automating Volume Backups with Terraform #

Here’s an example Terraform configuration for deploying an EC2 Virtual Machine, creating an additional Persistent EBS GP3 volume with specific specifications, attaching it, and configuring an automatic daily backup policy (Data Lifecycle Manager):

# ✓ CORRECT: Use Terraform to deploy Persistent Block Storage with automated backup policies

# 1. Main VM Instance
resource "aws_instance" "app_server" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  subnet_id     = aws_subnet.private_az1.id

  tags = {
    Name = "production-app-vm"
  }
}

# 2. Create an Additional Persistent EBS Volume (GP3)
resource "aws_ebs_volume" "app_data_vol" {
  availability_zone = "ap-southeast-1a"
  size              = 100 # 100 GB size
  type              = "gp3"
  iops              = 3000 # gp3 base provisioned IOPS
  throughput        = 125  # gp3 base throughput (MB/s)

  tags = {
    Name = "production-app-data"
  }
}

# 3. Attach the Volume to the VM Instance
resource "aws_volume_attachment" "ebs_att" {
  device_name = "/dev/sdh"
  volume_id   = aws_ebs_volume.app_data_vol.id
  instance_id = aws_instance.app_server.id
}

# 4. Daily Automated Backup Policy (Lifecycle Policy)
resource "aws_dlm_lifecycle_policy" "daily_backup" {
  description        = "Automatic daily backup policy for production volumes"
  execution_role_arn = aws_iam_role.dlm_lifecycle_role.arn
  state              = "ENABLED"

  policy_details {
    resource_types = ["VOLUME"]

    target_tags = {
      Name = "production-app-data"
    }

    schedule {
      name = "Daily Snapshots"

      create_rule {
        interval      = 24
        interval_unit = "HOURS"
        times         = ["20:00"] # Runs at 8 PM
      }

      retain_rule {
        count = 7 # Keep snapshots for the last 7 days
      }

      copy_tags = true
    }
  }
}

Summary #

  • Block storage presents raw block devices attached to VMs through a dedicated low-latency SAN (Storage Area Network).
  • Ephemeral storage is volatile and its data is destroyed instantly when the VM stops; use it only for temporary data like caches.
  • Persistent block storage has a lifecycle independent of the VM, guaranteeing data safety during parent VM failures.
  • Choose the NVMe protocol over SCSI to minimize CPU lock queue contention during parallel I/O operations on multi-core databases.
  • Use EBS-Optimized VMs to ensure I/O bandwidth between the instance and storage backend doesn’t hit network bottlenecks.
  • Enable transparent built-in encryption at the hypervisor level to secure disk data without burdening VM CPU performance.
  • Use the Elastic Volumes feature to resize disk capacity and performance directly in production without reboot downtime.
  • Use Multi-Attach wisely only with cluster filesystems (like GFS2) to anticipate the data-destroying split-brain danger.
  • Perform periodic backups via efficient Incremental Snapshots, and ensure crash-consistency before snapshotting active databases.

← Previous: Object Storage   Next: File Storage →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact