Back to Notes

Blob & Object Storage

What It Is

Storage for large, unstructured binary data (images, videos, file uploads, backups) — deliberately not a database. Flat namespace of key → blob (bucket + object key), not rows/tables. S3, GCS, Azure Blob are the canonical examples.

Why Not Just Store It in the Database

  • Databases are optimized for structured, queryable, transactional data — a 2GB video in a SQL row destroys buffer-cache efficiency and bloats backups.
  • Blob stores are optimized for exactly the opposite: massive files, high durability (S3: 11 nines), cheap at scale, served directly (often via CDN) without touching application servers at all.
  • The standard pattern: database stores metadata (filename, owner, size, blob-store key, upload timestamp); the actual bytes live in the blob store. The DB row points at the blob, never contains it.
flowchart LR
    Client -->|1. request upload URL| API[App Server]
    API -->|2. generate signed URL| Client
    Client -->|3. upload directly| Blob[(Blob Store<br/>S3-style)]
    API -->|4. save metadata<br/>key, owner, size| DB[(Metadata DB)]

Pre-signed URLs (the key mechanism)

Rather than routing large file uploads through your application server (wasting its bandwidth/CPU on pass-through bytes), the app server generates a time-limited, permission-scoped URL that lets the client upload/download directly to/from the blob store. The app server's job becomes issuing and validating these URLs, not moving bytes.

import boto3
from datetime import timedelta

s3 = boto3.client("s3")

def generate_upload_url(bucket: str, key: str, expires_in: int = 300) -> str:
    return s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": bucket, "Key": key},
        ExpiresIn=expires_in,   # seconds — short-lived, scoped to one object
    )

Chunking

Large files (video, big documents) are split into chunks before upload/storage:

  • Enables resumable uploads — a dropped connection only needs to retry the failed chunk, not the whole file.
  • Enables deduplication — identical chunks across different files (or across versions of the same file, as in Dropbox-style sync) can be stored once, referenced multiple times, hashed by content (content-addressable storage).
  • Enables progressive/streaming delivery — video playback starts from chunk 1 while later chunks are still being fetched/transcoded.

Replication & Durability

Object stores replicate each blob across multiple availability zones/disks automatically — this is why they quote extreme durability numbers (S3: "11 nines" = designed for 99.999999999% durability) versus a typical single-region database setup. Trade-off: write latency is higher than local disk, acceptable because blob writes are rarely on a user's hot request-response path (uploads go through the pre-signed URL flow above, async from the app server's perspective).

Interview Drill

Q: Why does the metadata DB store a reference to the blob instead of the blob itself, and what problem would you hit if you didn't? A: Storing large binaries in the DB bloats every backup/restore cycle, destroys the DB's buffer cache efficiency (large blobs evict the working set of actual queryable data), and makes horizontal read scaling (replicas) expensive to sync. Splitting metadata (small, structured, queryable) from bytes (large, opaque, served directly from a purpose-built store) lets each half scale on its own terms.

Cross-links

[[CDN]] · [[SQL vs NoSQL]] · [[Database Replication & Sharding]]