Consumer Group
consumerA consumer group is a set of consumers identified by a shared group.id that collectively consume a topic. Kafka assigns each partition to exactly one consumer in the group at a time. When a consumer joins or leaves the group, Kafka triggers a rebalance to redistribute partitions. Multiple independent consumer groups can consume the same topic simultaneously, each maintaining its own offset.
Details
Consumer groups are how Kafka enables parallel processing and multiple independent subscribers.
Partition assignment: With 4 partitions and 2 consumers: each consumer gets 2 partitions With 4 partitions and 4 consumers: each consumer gets 1 partition With 4 partitions and 6 consumers: 4 consumers get 1 partition, 2 consumers idle Idle consumers serve as hot standby for failover
Rebalance protocols: Eager rebalance (classic, pre-2.4): all consumers stop and give up partitions during rebalance. High pause time. Triggered by: consumer join, consumer leave, topic partition change. Cooperative/incremental rebalance (since Kafka 2.4, default in 3.1+): only revoked partitions stop processing. Other consumers continue uninterrupted. Requires partition.assignment.strategy=CooperativeStickyAssignor.
Static membership (since Kafka 2.3): Set group.instance.id=<unique-id> on each consumer to assign a stable identity. The broker gives the same partitions back to the same consumer.instance.id when it reconnects within session.timeout.ms. Eliminates unnecessary rebalances during rolling restarts.
Group coordinator: The broker responsible for tracking group membership and triggering rebalances for a given group. Identified via FindCoordinator API: hash(group.id) % __consumer_offsets_partitions.
Consumer lag: The difference between the latest offset (high-watermark) and the committed offset for each partition. High lag means the consumer is falling behind. Monitor with kafka-consumer-groups.sh or Burrow/Kafka Lag Exporter.
Key facts
- →
Each partition is assigned to at most one consumer per group at any time
- →
Adding consumers beyond the partition count does not increase throughput – extra consumers are idle
- →
Multiple groups consume the same topic independently – group A's committed offset does not affect group B
- →
Rebalances pause consumption (eager) or minimally interrupt it (cooperative)
- →
group.id is the only required consumer configuration beyond bootstrap.servers
Common gotchas
Adding more consumers than partitions is wasteful – extra consumers sit idle. Increase partition count to add consumer parallelism.
The default eager rebalance protocol stops all consumers during group membership changes. Switch to CooperativeStickyAssignor for production to minimize downtime.
Long rebalances (minutes) indicate session.timeout.ms is too short relative to processing time. If processing a record takes 30 seconds but session.timeout.ms=10000, the broker thinks the consumer died and triggers rebalance.