Consumer Rebalance
consumerA rebalance is the process by which Kafka redistributes partition assignments among consumers in a group. It is triggered when consumers join, leave, crash, or when topic partitions change. The classic eager rebalance stops all consumers during reassignment. The cooperative incremental rebalance (recommended, default in Kafka 3.1+) only revokes affected partitions, allowing unaffected consumers to continue processing.
Details
Rebalances are the most common source of consumer downtime and processing gaps in Kafka applications.
Triggers: - Consumer joins group (new consumer started) - Consumer leaves group (consumer stopped or crashed) - Consumer missed heartbeat (session.timeout.ms exceeded) - Topic partition count changed - Topic subscription pattern matched a new topic - Group coordinator broker restarted
Eager rebalance (classic): 1. All consumers send LeaveGroup (or timeout) 2. All consumers revoke ALL their partitions 3. Group coordinator assigns all partitions from scratch 4. All consumers receive new assignments and resume Stop-the-world: zero processing during rebalance. Typical duration: seconds to minutes.
Cooperative incremental rebalance (since Kafka 2.4): 1. Group coordinator calculates minimal changes 2. Only consumers holding partitions that need to move revoke those specific partitions 3. Revoked partitions are assigned to other consumers 4. Unaffected consumers continue processing throughout Requires: partition.assignment.strategy=CooperativeStickyAssignor (default since 3.1)
Tuning to reduce rebalance impact: session.timeout.ms: how long the broker waits before declaring a consumer dead (default 45000ms) heartbeat.interval.ms: how often consumer sends heartbeats (default 3000ms, must be < session.timeout.ms / 3) max.poll.interval.ms: max time between poll() calls before consumer is considered dead (default 300000ms / 5 min) group.instance.id: static membership – same consumer gets same partitions back on reconnect
Commit during rebalance: When a partition is revoked, the consumer should commit its current offset for that partition. Use a ConsumerRebalanceListener.onPartitionsRevoked() callback to commit offsets before partitions are transferred.
Key facts
- →
Rebalances are unavoidable – design consumers to handle them gracefully
- →
CooperativeStickyAssignor minimizes processing interruption – use it in production
- →
max.poll.interval.ms must be > your slowest record processing time, or Kafka declares the consumer dead
- →
Static membership (group.instance.id) eliminates unnecessary rebalances during rolling restarts
Common gotchas
Slow record processing that exceeds max.poll.interval.ms causes unnecessary rebalances. Either increase max.poll.interval.ms or process records faster (offload to thread pool, reduce batch size).
Not committing offsets in onPartitionsRevoked() during rebalance causes records to be reprocessed by the new owner of the partition. Always commit in the revoke callback.
Frequent rebalances cause 'rebalance storm' where the group never stabilizes. Usually caused by consumers crashing repeatedly or GC pauses exceeding session.timeout.ms. Increase heap size or session.timeout.ms.