Durability vs Availability #

When designing cloud data storage infrastructure, we often encounter the terms durability (data resilience) and availability (data accessibility). Although these two concepts sound similar and often confuse new developers, they measure very different performance parameters and are calculated independently. Durability answers the question: “Will our data still exist and remain uncorrupted in the long term?”, while availability answers: “Can our data be accessed instantly when we need it right now?” Understanding the fundamental difference, calculation metrics, and trade-off scenarios between them is crucial for designing robust storage systems without wasting cloud operational costs.

Durability: Measuring Data Existence #

Durability is defined as the probability that stored data will not be lost, deleted, or corrupted (like binary data corruption) within a certain period (usually calculated per year). Durability focuses entirely on the physical integrity of data.

The “Eleven Nines” Concept #

Leading cloud service providers generally offer durability up to 99.999999999% (commonly called eleven nines) for their object storage services. To understand how tough this mathematical number is, let’s assume the following scenario:

• If we store 10,000,000 (10 million) objects in cloud storage.
• With 99.999999999% durability, the physical loss probability is only 1 object every 10,000 years.

This number far exceeds the resilience capability of local storage media or traditional self-managed NAS servers in local data centers.

How Cloud Providers Achieve High Durability #

Cloud providers don’t rely on the physical miracle of a single hard drive to reach this resilience level. Behind the scenes, they apply several layers of advanced technology:

1. Distributed Geographic Replication (Multi-AZ Replication): Every time we upload an object, the cloud storage controller automatically duplicates the file to at least three different physical data centers (Availability Zones), separated by tens of kilometers within one region.
2. Erasure Coding: An advanced mathematical technique splitting one file into several data fragments ($N$) and parity fragments ($M$). These fragments are spread across dozens of different physical disks. The system can still reassemble the original file completely even if several physical disks suffer total damage simultaneously. The efficiency formula is: $$\text{Redundancy Overhead} = \frac{N + M}{N}$$ With erasure coding, we get durability equivalent to 3x replication, but with far more economical physical disk space consumption (only about 1.2x to 1.5x the original file size).
3. Active Scrubbing (Background Integrity Check): Cloud storage systems constantly scan files in the background using hashing algorithms (like MD5 or CRC32). If the system detects bit rot (binary data corruption from magnetic media degradation), it automatically removes the damaged fragment and rebuilds healthy new fragments from other replica locations.

flowchart TD
    Upload["1. Client Uploads File"] --> Controller["2. Storage Controller Layer"]
    Controller -->|"Hashing Process & Checksum Validation"| Verify["3. Integrity Validation"]
    Verify -->|"Split with Erasure Coding (Data + Parity)"| Split["4. Data Fragments"]
    Split -->|"Distribute"| AZ1["Storage Node AZ 1"]
    Split -->|"Distribute"| AZ2["Storage Node AZ 2"]
    Split -->|"Distribute"| AZ3["Storage Node AZ 3"]

Real Threats to Durability #

Although cloud infrastructure guarantees data durability from hardware failures, there’s one thing we must realize: Cloud providers only guarantee resilience against their infrastructure failures, not against our mistakes.

• User Errors: If our application code accidentally deletes a file, or our cloud account is hacked by ransomware doing mass encryption, that data is lost from our perspective.
• Mitigation: We must enable Object Versioning, WORM (Write Once Read Many / Object Lock), and MFA Delete multi-factor authentication for object deletion to protect data durability from non-infrastructure factors.

Availability: Measuring Service Accessibility #

Unlike durability, which focuses on data survival, Availability measures the percentage of time in a year when the cloud storage system is online and ready to accept and successfully serve our read and write operations.

Availability SLAs and Downtime Limits #

Availability is calculated based on the Service Level Agreement (SLA) offered by the cloud service provider. To see how significant the decimal difference is, let’s look at the availability percentage conversion table against maximum downtime tolerance per year:

If a cloud storage service promises a 99.9% availability SLA, then within a year, if the service’s total downtime or request refusal exceeds 8.7 hours, the cloud provider usually compensates with a monthly bill discount (service credit).

Why Is Availability Lower Than Durability? #

If we notice, standard object storage services generally offer 99.999999999% (11 nines) durability, but their SLA availability is “only” around 99.99% (4 nines) or 99.9% (3 nines).

This happens because keeping a service always accessible in real-time on the network is far harder than just storing data safely on physical disks. Availability is very vulnerable to internet network disruptions, DNS system failures, software update failures on API Gateways, power outages at data center router switches, and DDoS (Distributed Denial of Service) attacks.

Durability vs Availability Relationships and Scenarios #

To solidify our understanding of how these two parameters work separately, let’s analyze the following scenario matrix:

                  HIGH DURABILITY
                  ┌───────────────────────────────┬───────────────────────────────┐
                  │ Scenario A:                   │ Scenario B:                   │
                  │ High Durability               │ High Durability               │
                  │ Low Availability              │ High Availability             │
                  │ (e.g., Glacier Archive /      │ (e.g., S3 Standard /          │
                  │  Network Partition Quorum)    │  Replicated Multi-AZ SSD)     │
AVAILABILITY ─────┼───────────────────────────────┼───────────────────────────────┼───── AVAILABILITY
   LOW            │ Scenario C:                   │ Scenario D:                   │    HIGH
                  │ Low Durability                │ Low Durability                │
                  │ Low Availability              │ High Availability             │
                  │ (e.g., Single Local Disk      │ (e.g., Redis In-Memory Cache /│
                  │  without backup & unstable net)│  RAID 0 Storage Node)         │
                  └───────────────────────────────┴───────────────────────────────┘
                                  LOW DURABILITY

Scenario A: High Durability, Low Availability (Data Safe, But Hard to Access) #

This is a very common scenario in storage cost optimization.

• Example 1: Cold Storage / Archive Tier (Glacier): We place files in this tier. Data is guaranteed very safe from loss because it’s spread across many AZs (11 Nines durability). However, if we want to read those files, we must send a restore request first and wait 3 to 5 hours before the files can be downloaded. During that wait, the file’s availability is 0% because it can’t be accessed instantly.
• Example 2: Transit Network Failure (Network Partition): Suppose a distributed storage cluster requires a majority vote (quorum of 3 out of 5 nodes) to approve a new file write for data consistency. A fiber optic failure isolates 3 nodes from the external network. Data on all 5 nodes remains safe (durable), but the system is forced to reject all new write requests from clients to prevent data corruption (split-brain). The system becomes unavailable temporarily until the connection recovers.

Scenario D: Low Durability, High Availability (Fast Access, But Easily Lost) #

This scenario is commonly used to speed up application processing performance.

• Example 1: In-Memory Database (Redis without Persistence): All data is stored directly in server RAM memory. Operation latency is very fast (sub-millisecond) and ready to serve millions of requests per second (very high availability). However, if the server suddenly suffers a power failure or is rebooted, all data in memory vanishes instantly because RAM is volatile (zero durability).
• Example 2: RAID 0 (Disk Striping without Redundancy): We combine two hard drives into one volume to double data read/write speed. Data transfer speed is very high for serving applications (high availability). However, if either of the two disks suffers even minor physical damage, all data in the combined volume is instantly and irrecoverably corrupted (very low durability).

Impact on Storage Tier Choices and Costs #

Understanding this difference helps us significantly save monthly cloud spending by matching our data characteristics with the right storage tier.

Cloud Storage Class Comparison (Object Storage Case Study) #

Tier Selection Case Analysis #

1. Case 1: Profile Picture Thumbnail Storage:
   • Characteristics: Profile images must always be quickly accessible by users when they open the app (needs high availability). However, if profile image files are accidentally lost in the cloud, we can regenerate the thumbnails from the original master image files stored in our main system.
   • Best Tier: One Zone-IA or standard storage with minimal replication. We can save up to 20% by lowering durability tolerance for non-critical data.
2. Case 2: Financial Transaction Log Backup Files:
   • Characteristics: These logs must be kept for 7 years to comply with national tax regulations. Not a single byte may be lost (needs extra-high durability). However, these logs are almost never read again except during official government audits (very rare access, no instant availability needed).
   • Best Tier: Archive Tier (Glacier Deep Archive). The monthly per-GB cost is very cheap (almost 90% cheaper than the standard tier), yet data remains guaranteed safe with 11 nines durability.

Consistency and the CAP Theorem: A Complementary Dimension #

Besides durability and availability, there’s a third dimension often forgotten in distributed cloud storage systems: Consistency (data consistency).

In computer science, the CAP Theorem states that in a distributed data storage system connected over a network, when a connection failure occurs (Partition Tolerance / $P$), we can only choose one of two guarantees:

• Consistency ($C$): Every read operation is guaranteed to get the most recent write data, or returns an error if data isn’t consistent across all nodes.
• Availability ($A$): Every request is guaranteed a successful response, without the guarantee that the response contains the most recent data.

flowchart TD
    Partition{"Network Partition occurs?<br>(Inter-node connection cut)"}
    Partition -- Yes --> Choice{"Architecture Choice"}
    Choice -- "Choose Consistency (CP)" --> CP["Reject requests if nodes aren't synced<br>(Availability drops)"]
    Choice -- "Choose Availability (AP)" --> AP["Serve stale data from the nearest node<br>(Consistency drops)"]
    Partition -- No --> Normal["System runs normally with C and A simultaneously"]

Eventual Consistency vs Strong Consistency in the Cloud #

• Eventual Consistency: When we update a config.json object in the cloud, the change takes milliseconds to seconds to replicate to all physical data centers. If our application reads that file moments after the write completes, it might receive old data (stale data). This model prioritizes very high network availability.
• Strong Consistency: Since late 2020, storage services like AWS S3 upgraded their architecture to support strong read-after-write consistency natively at no extra cost. After a new object write (PUT) or old object overwrite succeeds with an HTTP 200 OK response, all subsequent read operations (GET) from anywhere are guaranteed to receive the newest data version. This simplifies our application programming logic because we no longer need to write replication wait-time handling code at the application level.

Summary #

• Durability guarantees data isn’t physically lost from cloud infrastructure hardware failures. Measured as a data resilience percentage (e.g., 99.999999999% / 11 nines).
• Availability guarantees data can be accessed instantly over the network when clicked or downloaded by applications. Measured as a yearly uptime percentage per SLA (e.g., 99.99%).
• Cold storage is an example of high durability but low availability — data is guaranteed very safe from loss, but needs a restore process taking hours before it can be read.
• In-memory caches (Redis) are an example of high availability but low durability — data is accessible super fast with sub-millisecond latency, but is totally lost when the server reboots.
• Cloud providers are only responsible for their infrastructure’s durability — we must protect durability from internal application or user errors by enabling versioning and WORM (Object Lock) features.
• Use the CAP theorem to understand compromises — choose architectures prioritizing data consistency ($C$) or network availability ($A$) during distributed network failures.

← Previous: File Storage   Next: Lifecycle & Data Tiering →