Caching, rate limiting and persistence

System design talked about caches, limiters and databases as boxes. Here we look inside them as classes. The skill is to give each a small interface so the rest of the code does not care how it works.

A cache needs a size limit and an eviction rule. LRU (least recently used) drops whatever has gone longest without use. A Map remembers insertion order, which makes it easy.

class LRU<K, V> {
  private m = new Map<K, V>();
  constructor(private cap: number) {}
  get(k: K): V | undefined {
    const v = this.m.get(k);
    if (v === undefined) return undefined;
    this.m.delete(k); this.m.set(k, v);     // mark as most recent
    return v;
  }
  set(k: K, v: V) {
    this.m.delete(k); this.m.set(k, v);
    if (this.m.size > this.cap) this.m.delete(this.m.keys().next().value as K);
  }
}

Add a time to live if staleness matters. Production caches also need protection against concurrent access, a topic from the previous lesson.

A rate limiter protects a service from too many requests. The token bucket is flexible: a bucket holds up to cap tokens, refills at a steady rate, and each request spends one. A burst is allowed until the bucket empties, then the rate is enforced.

class TokenBucket {
  private tokens: number;
  private last: number;
  constructor(private cap: number, private perSec: number, now: number) {
    this.tokens = cap; this.last = now;
  }
  allow(now: number): boolean {
    this.tokens = Math.min(this.cap, this.tokens + ((now - this.last) / 1000) * this.perSec);
    this.last = now;
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
}

Note that time is passed in, which keeps the class pure enough to test without waiting. A fixed window counter is simpler but lets a double burst through at the boundary of two windows. A sliding window fixes that at the cost of storing more. Keep one bucket per user or key.

Business code should not know SQL. A repository hides storage behind a collection-like interface.

interface TicketRepository {
  find(id: string): Promise<Ticket | undefined>;
  save(t: Ticket): Promise<void>;
}

The domain talks to this interface. One implementation uses SQL, another keeps a map for tests. This is dependency inversion at work. Related ideas: a transaction groups several saves so they all happen or none do, and a data transfer object is a plain shape for crossing a boundary, kept separate from the domain object so the database layout can change freely.

Because the cache and the repository share an interface, caching can be added by wrapping, with no change to either.

class CachedTicketRepository implements TicketRepository {
  constructor(private inner: TicketRepository, private cache: LRU<string, Ticket>) {}
  async find(id: string) {
    const hit = this.cache.get(id);
    if (hit) return hit;
    const t = await this.inner.find(id);
    if (t) this.cache.set(id, t);
    return t;
  }
  async save(t: Ticket) { await this.inner.save(t); this.cache.set(t.id, t); }
}

• An in-process cache is fast but private to one server, and copies can differ across servers.
• A limiter held in memory forgets on restart and does not span servers. Share the counts in a store when you need exact limits.
• A repository adds a layer. It pays off when storage may change or when you want fast tests, and is wasteful for a tiny script.