Leases, fencing and split brain

Many tasks need exactly one owner at a time: the leader of a database, the holder of a lock, the node allowed to write a file. The obvious tool is a lock. But in a distributed system a lock holder can crash, pause or be cut off, and without a way to take the lock back, everyone waits forever. And if the lock is taken back too eagerly, two nodes may both believe they own it. This lesson covers a safe answer: leases, backed by fencing tokens, and the failure they guard against, split brain.

A lease is a lock with an expiry time. A node is granted ownership for a fixed period, say 10 seconds, and must renew it before it runs out. If the holder crashes or is cut off, it stops renewing, the lease expires on its own, and another node can take over. No one needs to know for certain that the holder died. Waiting out the lease is enough.

Leases rely on time, so they inherit time's problems.

• Clock drift: the grantor and the holder measure time with different clocks. Design so that the holder treats its lease as expired a little early, leaving a safety margin, and uses a duration measured locally instead of comparing absolute clocks.
• Renewal before expiry: renew well before the end, for example after one third of the lease, so one lost message does not lose ownership.

Chubby sessions, ZooKeeper sessions, leader leases in databases and distributed cache locks are all leases.

Split brain is when two nodes both believe they are the leader and both act. A network partition is the classic cause: the old leader is cut off from the rest, the others elect a new leader, and now both are running. Both accept writes, the data diverges, and the damage can be permanent.

Prevention uses the earlier patterns.

• Majority quorum: only the side holding a majority can elect a leader or commit writes. The minority side cannot proceed, so there cannot be two.
• Leases: the old leader's authority expires on a schedule, so it should stop acting before the new one starts.
• Fencing the old node by force, for example cutting its power or revoking its access to storage. Operators call it STONITH, "shoot the other node in the head".

Even with a perfect lease, a subtle failure remains. A node acquires a lease and starts to write. Then it pauses, because of a long garbage collection, a stalled virtual machine, or a slow disk, for longer than the lease. The lease expires, a new owner is chosen and begins work. The old node wakes up, still believing it holds the lease, and finishes its write. The check "do I still own it?" happened before the pause, and the world changed during it. Checking the clock before acting does not fix it, because the pause can strike right after the check.

The reliable fix moves the check to the resource being protected. Every time the lock service grants a lease, it issues a fencing token: a number that always increases.

1. Node A acquires the lease and receives token 33.
2. A pauses. Its lease expires.
3. Node B acquires the lease and receives token 34, and writes to storage with token 34. Storage records 34 as the highest seen.
4. A wakes up and writes with token 33. Storage sees 33 is lower than 34 and rejects the write.

The storage service needs only a simple rule: remember the highest token and refuse anything lower. Even if an old owner keeps running, it can do no harm.

The same idea has many names. Raft calls it the term. Kafka uses a leader epoch to reject requests from an old controller and to truncate logs safely. HDFS fences the previous active NameNode before letting the standby take over. Chubby's sequencer carries a generation number for the same purpose. Whenever authority can change hands, attach a generation to it and make everything that matters check it.

• Lease: a time-limited lock that cleans itself up after failures.
• Split brain: two owners at once, usually after a partition, prevented by majority quorums.
• Fencing token: an ever-increasing number checked by the resource, which protects against paused or delayed owners.
• Never trust "I checked a moment ago that I own it". Let the protected resource verify it.