Instagram and Dropbox

Both products store large files for many users and must make them available anywhere. Both split their data into two kinds: blobs (the photo or file bytes) and metadata (who owns it, its name, its version). Blobs go in cheap, durable object storage, metadata in a database. After that, they diverge. Instagram's hard problem is the feed. Dropbox's hard problem is sync.

Users upload photos, follow other users, see a feed of recent photos from people they follow, like and comment, and search. The service must be highly available, the feed must load fast, and photos must never be lost. Slightly stale feeds are fine, but data durability is not negotiable.

Assume 10 million new photos per day at about 200 KB each.

• Writes: 10M / 86,400 is about 120 uploads per second.
• Storage: 10M x 200 KB = 2 TB per day, so about 730 TB per year, or around 3.6 PB over five years.
• Reads: people view far more than they upload. At 100 views per photo, that is billions of views a day, roughly 12,000 photo reads per second on average, much more at peak.

Petabytes of mostly cold photos means object storage plus a CDN, not a database.

Separate the upload path from the read path. Uploads are slow and hold connections open. If the same servers handle both, a burst of uploads can starve reads. With separate services, each scales on its own.

A photo upload stores the bytes in object storage, then writes a metadata row (photo id, owner, location, time) to a database. Photos are served through a CDN, which keeps popular images near viewers.

For ids, we need unique, roughly time-ordered ids across many shards without a central counter. A common scheme combines a timestamp, a shard number and a sequence number into one 64-bit id, so ids sort by time and no two shards collide. Sorting by id then gives newest first.

The feed shows recent posts from everyone you follow. There are two classic ways to build it.

• Fan-out on read (pull): when you open the app, fetch the latest posts from each person you follow and merge them. Writes are cheap, but each read does a lot of work, which is slow for someone following hundreds of accounts.
• Fan-out on write (push): when someone posts, write the post id into a precomputed feed list for each follower. Reading a feed is then one fast lookup. But an account with 50 million followers triggers 50 million writes per post.

The standard answer is a hybrid. Push for ordinary accounts, since it makes the common read cheap. For celebrity accounts, do not push. Instead, merge their recent posts into your feed at read time. Keep each feed as a short list of post ids in an in-memory store, and look up the metadata when displaying. Ranking by relevance rather than strictly by time adds a scoring step before display.

Shard metadata by a user or photo id, replicate each shard, and cache hot feeds and profiles. Likes and counts can be approximate and updated asynchronously. Search uses a separate index built from captions and tags.

Users store files in folders and see the same files on every device. Changes made on one device appear on the others automatically, large files upload reliably even over bad connections, old versions can be restored, and files can be shared. Data loss is unacceptable, and sync should use as little bandwidth as possible.

Chunking. Split each file into fixed size pieces, say 4 MB. A 1 GB file is 256 chunks. A failed upload resumes from the last good chunk instead of starting over, and chunks can upload in parallel.

Hashing and deduplication. Compute a hash of each chunk. If the system already holds a chunk with that hash, from any user, do not store or upload it again. Many files share content, so this saves large amounts of space and bandwidth.

Delta sync. When you edit a file, only the chunks that changed need to be uploaded. Changing one paragraph in a large document uploads one chunk, not the file.

Separate metadata from blocks. A metadata service records files, folders, versions and which chunk hashes make up each version. A block store holds the chunks themselves, addressed by hash.

A client watcher on the desktop notices a local change. It chunks the file and hashes the chunks, then asks the metadata service which chunks the server lacks. It uploads only those to the block store, then commits a new version in the metadata service. The service notifies other devices of the new version, using a long poll or a persistent connection, and each device downloads just the chunks it is missing.

Two devices may edit the same file while offline. When both reconnect, the server sees two edits from the same base version. Choices: the later write wins (simple, but loses work), or the system keeps both and creates a conflicted copy for a person to merge. Most file sync products choose the second, because losing a user's work is worse than a cluttered folder. Metadata updates need strong consistency, so a version is never half committed. The block store can be eventually consistent, since chunks never change once written.

Shard metadata by user or folder, replicate everything, and keep the block store in durable object storage with copies across regions. Cache hot metadata. A message queue buffers change notifications so a burst of edits does not overwhelm the notifiers. Compression and encryption of chunks, in transit and at rest, are routine.

• Both separate blobs from metadata and use object storage.
• Instagram optimizes for reads at huge scale: CDNs, feed caches, and the push versus pull tradeoff.
• Dropbox optimizes for bandwidth and correctness of syncing: chunking, deduplication, delta sync and conflict handling.
• Both depend on unique ids, sharding by a good key, and replication for durability.