Partition
storageA partition is an ordered, immutable sequence of records within a topic. Partitions are the unit of parallelism in Kafka – a topic with N partitions can be consumed by up to N consumers in a consumer group simultaneously. Each partition is stored on exactly one broker (the leader) with copies on follower brokers. Partition count can be increased but never decreased.
Details
Partitions are how Kafka achieves horizontal scalability. When a producer writes to a topic, Kafka decides which partition using: 1. Explicit partition in the ProduceRequest 2. key hash: murmur2(key) % partition_count (consistent for same key) 3. Sticky partitioner (default since Kafka 2.4): fills one partition until a batch is sent, then switches
Partition count decisions: - More partitions = more parallelism for consumers - Each partition is a file on disk – too many partitions increases memory and file handle usage - Recommended: 10–100 partitions per broker, total partitions per cluster ≤ 200,000 (Kafka 2.x) or more with KRaft - Rule of thumb: max(throughput / single_partition_throughput, consumer_parallelism_needed)
Partitions can be increased: run bin/kafka-topics.sh --alter --partitions N. New messages are distributed across all partitions. Existing messages stay in their original partitions. Key-based routing changes after adding partitions – this can break ordered processing per key.
Partitions cannot be decreased: once created, you cannot reduce partition count. Plan ahead.
Key facts
- →
Each partition is an independent ordered log – records within a partition are strictly ordered
- →
A partition has exactly one leader broker and 0+ follower brokers (depending on replication factor)
- →
All reads and writes to a partition go through the leader
- →
Partition count determines maximum consumer parallelism per consumer group
- →
Partition count can be increased, never decreased
Common gotchas
Adding partitions breaks key-based ordering – records with the same key may end up on different partitions after the increase
Too many partitions increases end-to-end latency – each partition adds leader election complexity and ZooKeeper/KRaft metadata overhead
A consumer group with fewer consumers than partitions leaves some partitions unread (idle consumers don't help)