Kafka

Large systems are full of data in motion: page views, orders, sensor readings, database changes. Many producers create it and many consumers want it, each at its own pace. Connecting every producer directly to every consumer produces a tangle that breaks whenever one side is slow or down. A messaging system sits in the middle. Kafka is a messaging system with an unusual core idea: it is a distributed, replicated, append-only log.

Instead of deleting a message once it has been delivered, Kafka keeps messages for a configured time. Consumers read from the log at their own position, and several consumers can read the same data independently, or replay it from the start. That single choice makes Kafka suited to event streaming, data pipelines and rebuilding state.

• A message (or record) has an optional key, a value, a timestamp and optional headers.
• A topic is a named stream of messages, such as orders.
• A topic is divided into partitions. Each partition is an ordered, immutable sequence of messages, and each message in it has a sequential number called its offset.
• A broker is one Kafka server. A cluster is several brokers working together.
• A producer writes messages to a topic, and a consumer reads them.
• A consumer group is a set of consumers that share the work of reading a topic.

A single log on a single machine limits throughput and size. Splitting a topic into partitions spreads the load. Different partitions can live on different brokers, be written by different producers at the same time, and be read by different consumers in parallel.

The price is that ordering is guaranteed only within a partition, not across the topic. To keep related messages in order, give them the same key. A producer picks the partition by hashing the key, so all events for one customer land in the same partition and stay in sequence. Messages without a key are spread across partitions in rotation.

A producer batches messages for the same partition and can compress each batch, which raises throughput considerably at the cost of a small delay. It chooses how much confirmation to wait for through the acknowledgement setting.

• acks=0: do not wait. Fastest, may lose data.
• acks=1: wait for the partition leader only. A leader crash before replication loses the message.
• acks=all: wait for all in-sync replicas. Slowest, safest.

Each partition has a replication factor, commonly 3. One replica is the leader, and the others are followers. All reads and writes for the partition go to the leader. Followers continuously copy the leader's log.

The set of replicas that are fully caught up is the in-sync replica set (ISR). If a follower falls behind or dies, it is removed from the ISR. The high-water mark is the highest offset that has been copied to every replica in the ISR. Consumers may read only up to the high-water mark, so they never see a message that could be lost if the leader failed.

With acks=all and a setting for the minimum number of in-sync replicas, say 2, a write succeeds only when at least two replicas have it. That is what lets Kafka survive the loss of a broker without losing acknowledged data. If too few replicas are in sync, writes are refused, favoring consistency over availability.

If a leader fails, one broker in the cluster, the controller, picks a new leader from the ISR, so the promoted replica has all committed data. Cluster metadata, such as which brokers exist and who leads each partition, was kept in ZooKeeper in older versions and is now managed by Kafka's own Raft-based quorum.

A consumer pulls messages from brokers, asking for data from a given offset. Pulling lets each consumer control its own speed, so a slow consumer never overwhelms anything, and it simply lags.

Within a consumer group, each partition is assigned to exactly one consumer, so each message is processed once per group. If a topic has 6 partitions, a group can use up to 6 consumers in parallel. A seventh would sit idle, because there is no partition left. That makes the partition count the ceiling on parallelism.

A different group reading the same topic gets its own full copy of the stream and its own position. A billing service and an analytics service can therefore both read orders without affecting each other.

When consumers join or leave, the group rebalances, reassigning partitions. During a rebalance, consumption briefly pauses, so frequent rebalances hurt throughput.

A consumer records how far it has read by committing an offset, stored in an internal Kafka topic. After a crash, a consumer resumes from the last committed offset. When you commit decides the guarantee.

• At most once: commit before processing. A crash after the commit loses the message. No duplicates.
• At least once: process, then commit. A crash between the two causes the message to be processed again. No loss, so processing must tolerate duplicates, ideally by being idempotent.
• Exactly once: no loss and no duplicate effects. Kafka supports this through an idempotent producer, which attaches sequence numbers so the broker discards retried duplicates, and transactions, which let a program write results and commit offsets atomically. It applies within Kafka's read, process, write loops. When the final effect is outside Kafka, you still need an idempotent receiver.

Kafka reaches very high throughput with ordinary hardware by following the grain of the machine.

• Sequential disk access: it only appends to the end of a log and reads sequentially. Sequential access on disk is far faster than random access, and comparable to memory for large transfers.
• The operating system page cache: Kafka does not keep its own cache. Recently written data stays in the OS cache, so most reads never touch the disk.
• Zero-copy transfer: data moves from the page cache to the network card without being copied through application memory.
• Batching and compression: many small messages are handled as one larger unit.

Each partition is stored as a series of segment files. When a segment reaches a size or age, a new one starts, and old segments are deleted whole, which is cheap. Retention is configured by time, such as seven days, or by size. A second mode, log compaction, keeps only the latest message for each key, which turns a topic into a changelog of current state, useful for rebuilding tables.

Suppose producers send 100 MB per second.

• Retention of 7 days is 100 MB x 604,800 seconds, about 60 TB. With replication factor 3, that is about 180 TB across the cluster.
• If one partition handles about 10 MB per second comfortably, we need at least 10 partitions for the write rate. Choose more, for example 30, to leave room for consumer parallelism and growth, remembering that partitions cannot be reduced easily later.

Good fit: event streaming, log and metric collection, decoupling services, feeding data to analytics and search, and change data capture from databases. Its ability to replay is the unique strength.

Poor fit: simple task queues with per-message acknowledgement and priorities, tiny workloads where a lighter queue is simpler, and request-response patterns that need an immediate answer. Remember the costs: ordering only per partition, operational complexity, and the need to design consumers that tolerate duplicates.