Caching

A cache keeps a copy of a result that was expensive to produce, so the next request for it is cheap. It works because real traffic is lopsided: a small set of items is asked for again and again. Reading from memory takes about 100 nanoseconds, while a trip to a database can take milliseconds, so a cache can be thousands of times faster.

The measure that matters is the hit ratio, the share of requests answered from the cache. Suppose our link service gets 20,000 reads per second. At a 90 percent hit ratio the database sees 2,000 reads per second. At 95 percent it sees 1,000. Small gains in hit ratio remove a lot of database load.

Caches can sit at several layers, and each one saves a different trip.

• The browser keeps files the user already downloaded.
• A CDN (content delivery network) stores static files such as images and scripts on servers near users, so they travel a short distance.
• An application cache is a fast in-memory store in front of the database. A local one lives inside each server and is quick but not shared. A shared one is a separate cache tier that all servers use, so a value cached by one is available to all.
• The database also caches its own hot pages in memory.

Most designs for a read heavy service add a shared application cache first, because it removes the most database load for the least effort.

The most common pattern is called cache aside. The server owns the logic:

• Look in the cache. If the item is there, that is a hit, so return it.
• If it is not there, that is a miss. Read the database, store the result in the cache with an expiry, and return it.

Notice what the cache is not: it is not the source of truth. If it is wiped, the system is slower, not wrong. That makes it safe to lose, which is why it is easy to scale.

When data changes, you must decide how the cache and database learn about it.

• Write-through: write to the cache and the database together, and finish only when both are done. Reads are always fresh, but every write is slower.
• Write-around: write only to the database. The cache fills on the next read. Writes stay fast and rarely read data does not crowd the cache, but the first read after a write is a miss.
• Write-back: write to the cache and return immediately, then save to the database later. This is the fastest for writes, but if the cache dies before saving, those writes are lost. Use it only where small losses are acceptable.

The hardest part of caching is that copies go stale. There are two main tools.

• Time to live (TTL): every entry expires after a fixed time. Simple and safe, but data can be stale until it expires.
• Explicit invalidation: when the data changes, delete or update the cached entry. Fresher, but you must remember to do it everywhere, and a missed path leaves stale data forever.

Most systems use both: invalidate on change, and keep a TTL as a safety net. Decide how stale each kind of data may be. A like count can be a minute old. An account balance cannot.

Memory is finite, so when the cache is full it must remove something. The policy decides what.

• LRU (least recently used): remove what has gone unused the longest. The usual default.
• LFU (least frequently used): remove what is requested least often.
• FIFO: remove the oldest entry, regardless of use. Simple, rarely best.
• Random: cheap and surprisingly decent.

To size the cache, use the 80/20 rule: about 20 percent of items cause about 80 percent of reads. For our service that hot set is about 170 GB, which fits a few machines.

Watch for a stampede: when a popular entry expires, many requests miss at once and all hit the database together. You can prevent it by letting only one request refill the entry while others wait, or by refreshing popular entries slightly before they expire.

A cache costs money, adds a layer that can be wrong, and is wasted when traffic is small or every request is unique. Add it when the numbers show a read heavy load, not by default.