Logs: write-ahead, segmented and high-water mark

A system holds data in memory for speed, but memory vanishes in a crash. Writing the whole state to disk on every change is far too slow. And once you have a stream of changes, you must store it without letting one file grow forever, and decide at what point a change counts as safe. Three connected patterns handle this: the write-ahead log, the segmented log and the high-water mark.

A write-ahead log (WAL) is an append-only file where every change is recorded before it is applied to the main data. The order is the whole point.

1. Append the change to the log and flush it to disk.
2. Apply it in memory or to the data files.
3. Acknowledge the client.

After a crash, the system reads the log from the last known good point and replays the changes, rebuilding exactly the state it had. Appending is a sequential disk write, which is fast, whereas updating data structures in place would be random writes.

Practical details:

• fsync cost: forcing data to disk is the slow step. Systems use group commit, flushing the log for many waiting writes at once.
• Checkpoints: periodically save a full snapshot of state, then discard the log before it, so recovery replays only a short tail.
• Each entry carries an id and a checksum, so recovery stops cleanly at a half-written record.

The same idea appears in relational databases, in Cassandra's commit log, in Raft's replicated log, and in the file system journal.

One endless log file is awkward. It cannot be deleted from the front, it is hard to search, and recovery must scan it all. A segmented log splits the log into a series of segment files, each up to a size or age limit, such as 1 GB. Only the newest segment is written to. Older segments are immutable.

• Naming: each segment is named after the offset of its first entry, such as 0000000000.log, 0000500000.log. To find offset 612,345, binary search the segment names to find the file with the largest starting offset not above it.
• Index: each segment has a sparse index from offset to byte position. Look up the nearest earlier entry, then scan forward a short way.
• Cleaning: deleting old data means deleting whole old files, which is instant, with no rewriting. Retention by time or size becomes trivial.
• Compaction: a background job can rewrite old segments keeping only the latest value for each key.

Kafka stores every partition this way.

In a replicated log, the leader appends an entry before followers have copied it. If it showed that entry to readers right away and then crashed, a new leader might not have it, and readers would have seen something that vanishes.

The high-water mark is the highest offset known to be stored on enough replicas, typically all in-sync ones or a majority. The leader tracks how far each follower has copied, and advances the mark as they catch up. Entries at or below the mark are committed: safe, since they survive a leader failure. Readers and consumers may see only up to it.

After a leader change, a follower may hold entries beyond the new leader's log, which were never committed. The follower truncates back to the common point and then copies the new leader's entries, so all logs become identical.

The opposite end also matters. The low-water mark is the oldest offset still retained. Entries before it have been deleted, so a reader asking for an older offset must be told it is gone, instead of being given the wrong data.

• The WAL gives durability: nothing acknowledged is lost.
• The segmented log gives manageability: bounded files, cheap deletion, fast lookup.
• The high-water mark gives safety in replication: readers see only committed data.

The deeper lesson is that an append-only log is a simple, fast and recoverable foundation, and many sophisticated systems are built as little more than a log plus the machinery to read it well.