Kafka Concepts

A partition is an ordered, immutable sequence of records within a topic. Partitions are the unit of parallelism in Kafka – a topic with N partitions can be consumed by up to N consumers in a consumer group simultaneously. Each partition is stored on exactly one broker (the leader) with copies on follower brokers. Partition count can be increased but never decreased.

An offset is a monotonically increasing integer (uint64) that uniquely identifies a record's position within a partition. Offsets start at 0 and increase by 1 for each record. Consumers track their committed offset to know where to resume after restart. Kafka never reuses offsets – even after record deletion, the offset sequence continues forward.

Kafka retains records by time (log.retention.hours, default 7 days) or size (log.retention.bytes). Records are stored in segment files; only complete segments are deleted. Log compaction is an alternative policy that keeps only the latest value per key, indefinitely – enabling Kafka to serve as a durable key-value store for event-sourced systems.

Kafka replicates each partition across multiple brokers for fault tolerance. One broker is the leader (handles all reads and writes). The rest are followers (replicate from the leader). The ISR (In-Sync Replicas) is the subset of followers fully caught up. min.insync.replicas controls how many ISR members must acknowledge a write before it is considered successful.

A consumer group is a set of consumers identified by a shared group.id that collectively consume a topic. Kafka assigns each partition to exactly one consumer in the group at a time. When a consumer joins or leaves the group, Kafka triggers a rebalance to redistribute partitions. Multiple independent consumer groups can consume the same topic simultaneously, each maintaining its own offset.

A rebalance is the process by which Kafka redistributes partition assignments among consumers in a group. It is triggered when consumers join, leave, crash, or when topic partitions change. The classic eager rebalance stops all consumers during reassignment. The cooperative incremental rebalance (recommended, default in Kafka 3.1+) only revokes affected partitions, allowing unaffected consumers to continue processing.