Object Storage #

Object storage is the data storage model most synonymous with cloud architecture — and the most frequently used storage type in modern cloud-native systems. Unlike traditional storage media that present data as raw blocks or directory folder hierarchies (file systems), object storage treats every file as an independent data unit called an Object. Each object is stored in one large centralized container called a Bucket. Objects are accessed and modified exclusively through HTTP RESTful API protocols (like GET, PUT, and DELETE). With nearly unlimited horizontal scaling characteristics, extreme durability, and relatively very cheap costs, object storage is the main foundation for various modern workloads — from AI data lake storage, periodic backup storage, to global-scale static media content distribution.

How It Works: Flat Namespace & Object Structure #

To deeply understand object storage, we must throw away our understanding of traditional folder directory structures. In local filesystems (like NTFS, ext4, or APFS), the operating system maintains a complex tree structure index (directory tree) to track folders and subfolders.

In contrast, object storage operates on a Flat Namespace model. Inside a bucket, there are no physical folders at all. All objects are stored side by side at the same single storage level.

Traditional Filesystem (Hierarchical):
  /data/
    ├── users/
    │   ├── profile-123.jpg
    │   └── profile-456.jpg
    └── reports/
        └── 2024-q4.pdf
  
  Access: Through multi-level directory paths.
  Modification: Can edit specific parts of a file in-place (seek, write).

Object Storage (Flat Namespace):
  Bucket: my-app-storage
    ├── Key: "users/profile-123.jpg"    ← The "/" slash character is only part of
    ├── Key: "users/profile-456.jpg"      the name (string key), not a real folder.
    └── Key: "reports/2024-q4.pdf"
  
  Access: HTTP API Request (GET/PUT/DELETE) to the object endpoint URL.
  Modification: Atomic/immutable. Can't edit part of a file; the file must be fully re-uploaded.

Object Anatomy #

Every object we upload consists of the following structured components:

• Data (Payload): The file’s binary content itself (can be text, images, videos, or binary backups).
• Key (Identifier): A unique string name identifying the object inside the bucket.
• Metadata: Key-value pairs describing object properties. Consists of System Metadata (like Content-Type, Content-Length, Last-Modified, ETag) and Custom User Metadata (like Author: Budi, Project: Alpha).
• Version ID: A unique token used if the Versioning tracking feature is enabled on the bucket.

Data Consistency: Eventual vs Strong Consistency #

One technical aspect that used to confuse object storage users is the data consistency model. In the early cloud days (especially Amazon S3 before December 2020), object storage followed the Eventual Consistency principle for overwrite and delete operations.

Eventual Consistency Scenario (Old Model):
  1. We upload a new version of "logo.png" to overwrite the old version.
  2. When a client sends a GET request for "logo.png" moments after the upload finishes,
     that client may still receive the old "logo.png" version for a few milliseconds
     to seconds, until the new data replicates to all of the cloud provider's physical servers.

In December 2020, Amazon S3 officially updated its architecture to guarantee full Strong Read-After-Write Consistency with no performance or cost degradation. Other major cloud providers followed this standard.

• Modern Impact: Now, immediately after receiving an HTTP 200 OK success response for a PUT or DELETE operation, subsequent GET requests from any client in the world are guaranteed to read the latest data. This greatly simplifies our application programming logic because we no longer need to write polling lock scripts to wait for data consistency.

Cost Optimization with Storage Classes & Lifecycle Policies #

One of the biggest advantages of managed object storage is the availability of different Storage Classes based on data access frequency. This lets companies significantly optimize costs.

For example, AWS S3 provides several main storage classes:

1. Standard: Designed for active data accessed continuously. Most expensive storage cost, but no retrieval fees.
2. Infrequent Access (Standard-IA): Designed for rarely accessed data (e.g., once a month) that needs instant access when required. Cheaper storage cost (~50% of Standard), but charges per GB when data is downloaded (retrieval fee).
3. Glacier Instant Retrieval: Designed for rarely accessed archives (a few times a year) needing millisecond response times.
4. Glacier Flexible Retrieval (Archive): Very low-cost archives. Data retrieval takes from minutes to hours.
5. Glacier Deep Archive: The cheapest storage class in the cloud. Designed for compliance data storage accessed once every few years. Retrieval takes up to 12 hours.

Lifecycle Policies #

To avoid manually managing data class transitions, we can configure Automated Lifecycle Policies based on object age:

flowchart LR
    Upload["1. Object Uploaded (v1.0)"] -->|"After 30 Days"| IA["2. Move to Infrequent Access (Standard-IA)"]
    IA -->|"After 90 Days"| Glacier["3. Move to Glacier Archive (Cheapest Cost)"]
    Glacier -->|"After 365 Days"| Delete["4. Permanently Delete Object"]

Compliance & Large-Scale Transaction Features: Object Lock & Multipart Upload #

For enterprise data governance needs and massive-scale data transfer efficiency, object storage provides the following two important features:

1. Object Lock (WORM Model - Write Once, Read Many) #

Object Lock lets us prevent an object from being deleted or overwritten for a certain retention period.

• Compliance Guarantee: Once enabled on an object in Compliance mode, no one (including our cloud root administrator account) can delete that object until the expiration period ends. This feature is vital for complying with financial industry regulations (like SEC rules) and acts as the strongest shield against ransomware attacks trying to delete our backup files.

2. Multipart Upload #

When we need to upload very large files (usually above 100 MB up to the 5 TB maximum capacity), uploading in a single HTTP request is very risky. If the network connection drops mid-way at 99%, we must restart the upload from zero.

• Mechanism: The Multipart Upload feature automatically splits the large file into dozens of small parts, uploads them in parallel, and reassembles them into one complete object on the cloud side after all parts are successfully received. If one part fails mid-way, we only need to re-upload the broken part, saving time and bandwidth.

flowchart TD
    File["Large File (1 GB Video)"] --> Split["Split into 10 Parts (100 MB/part)"]
    subgraph ParallelUpload["Parallel HTTP Upload (Part 1 - 10)"]
        Part1["Part 1"]
        Part2["Part 2"]
        Part3["Part 3"]
    end
    Split --> ParallelUpload
    ParallelUpload --> Assemble["Reassemble in the S3 Bucket"]
    Assemble --> Object["Complete Standalone Object"]

Security Features: Bucket Policies and Pre-signed URLs #

Data security in object storage is controlled through a very flexible, layered authorization system:

1. Bucket Policies vs IAM Policies #

• IAM Policies: Identity-based policies. Used to define what specific users or servers (roles) within our cloud account may access.
• Bucket Policies: Resource-based policies. Attached directly to buckets and used for macro-level access control — like allowing public download access for website assets, or restricting the bucket to only be accessible from a specific VPC CIDR.

2. Pre-signed URLs (Signed URLs) #

In modern application architectures, we often face scenarios where users need to download private files (like PDF payslips) or upload large files (like recorded videos).

If all those files must pass through our application backend server, the backend will hit bandwidth and CPU bottlenecks from acting as a data transfer intermediary. The best solution is using Pre-signed URLs:

sequenceDiagram
    participant User as Browser Client
    participant App as Backend App (Stateless)
    participant S3 as Object Storage (Private Bucket)
    
    User->>App: Request file upload access for "video.mp4"
    Note over App: Validate IAM Authorization &<br/>Create Pre-signed URL with cryptographic signature
    App-->>User: Send Pre-signed URL (Valid 15 Minutes)
    
    User->>S3: PUT /mybucket/video.mp4 (Direct to S3 with signed URL)
    Note over S3: Verify cryptographic signature<br/>& Accept the video file
    S3-->>User: HTTP 200 OK (Upload Successful)

With this pattern, our application backend server only handles access checking (authorization), cryptographically signs the URL using its private credentials, and returns the temporary URL to the client. The client can directly upload the giant file straight to the private bucket securely, freeing the backend from data transfer load.

High Durability & Network Replication #

Managed object storage in the cloud offers extremely high Durability (data resilience against physical damage), generally reaching 99.999999999% (11 Nines).

What does 11 Nines Durability mathematically mean? If we store 10,000,000 (10 million) files in object storage, on average we only risk losing 1 file every 10,000 years.

This extraordinary resilience is achieved through the following technologies:

• Erasure Coding: Uploaded data files are split into several data fragments and parity fragments, then spread across dozens of separate physical disks.
• Multi-AZ Replication: Automatically duplicates our data to at least 3 physically separate Availability Zones (AZs) within the same geographic region. If one data center is destroyed by disaster, our data remains intact in the other data centers.

Code Example: Creating Pre-signed URLs for Secure Upload & Download #

Here’s a backend implementation example using Node.js with the official AWS SDK (v3) to generate Pre-signed URLs for clients to perform PUT (upload) and GET (download) operations on private files securely:

// presigned-helper.js
const { S3Client, PutObjectCommand, GetObjectCommand } = require("@aws-sdk/client-s3");
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");

// ✓ CORRECT: Initialize the S3 client with configuration from Environment Variables
const s3Client = new S3Client({
  region: process.env.AWS_REGION || "ap-southeast-1"
});

const BUCKET_NAME = process.env.MY_SECURE_BUCKET || "production-private-data";

/**
 * Creates a temporary URL for uploading files directly to S3
 * @param {string} objectKey The destination file name in S3 (e.g., 'invoices/user-123.pdf')
 * @param {string} contentType The file's MIME type (e.g., 'application/pdf')
 * @returns {Promise<string>} Pre-signed URL for upload
 */
async function generateUploadUrl(objectKey, contentType) {
  try {
    const command = new PutObjectCommand({
      Bucket: BUCKET_NAME,
      Key: objectKey,
      ContentType: contentType
    });

    // Create a pre-signed URL expiring in 15 minutes (900 seconds)
    const url = await getSignedUrl(s3Client, command, { expiresIn: 900 });
    return url;
  } catch (error) {
    console.error("Failed to create upload pre-signed URL:", error);
    throw new Error("Failed to initiate file upload.");
  }
}

/**
 * Creates a temporary URL for reading/downloading private files from S3
 * @param {string} objectKey The file name in S3
 * @returns {Promise<string>} Pre-signed URL for download
 */
async function generateDownloadUrl(objectKey) {
  try {
    const command = new GetObjectCommand({
      Bucket: BUCKET_NAME,
      Key: objectKey
    });

    // Create a pre-signed URL valid for 30 minutes (1800 seconds)
    const url = await getSignedUrl(s3Client, command, { expiresIn: 1800 });
    return url;
  } catch (error) {
    console.error("Failed to create download pre-signed URL:", error);
    throw new Error("Failed to fetch the file download link.");
  }
}

module.exports = {
  generateUploadUrl,
  generateDownloadUrl
};

Summary #

• Object storage stores data in a flat namespace without physical folder hierarchies, instead using unique string identifiers (Keys).
• Strong consistency guarantees data is immediately readable right after a successful write operation, simplifying modern application architectures.
• Optimize costs by moving rarely accessed data to cheaper storage classes (Standard-IA/Archive) automatically via Lifecycle Policies.
• Use Object Lock (WORM) to protect critical backup data from modification and destruction by ransomware viruses.
• Multipart Upload splits giant files into small parts to optimize parallel speed and guarantee network disconnection tolerance.
• Pre-signed URLs offload data transfer burdens by letting clients upload files directly to buckets without overloading the application backend.
• Extreme 11 Nines durability in the cloud is achieved by automatically replicating data across Availability Zones and applying erasure coding techniques.
• Always avoid putting dynamic media files on VM local disks; use object storage so our compute stays stateless.

← Previous: Load Balancing   Next: Block Storage →