Offset

Details

Offsets are the addressing system for Kafka records. They work like a file position pointer for each partition.

Offset semantics: Earliest offset (--from-beginning): seek to offset 0 (or log.retention start) Latest offset: the next offset to be written (current end of log) Committed offset: the last offset a consumer group has processed and acknowledged High-watermark: the last committed offset on the leader (consumers only read up to this)

Committed offset storage: Kafka stores consumer group offsets in the internal __consumer_offsets topic (since Kafka 0.9). This replaced ZooKeeper offset storage. Each consumer group + topic + partition has exactly one committed offset entry.

auto.offset.reset: what happens when a consumer group has no committed offset (new group) or the committed offset is out of range (deleted): earliest: start from the beginning of retained records latest: start from the end (miss all historical records) none: throw exception

Manual offset commits: consumer.commitSync() – synchronous, blocks until broker confirms consumer.commitAsync() – non-blocking, use callback for errors consumer.commitSync(offsets) – commit specific topic-partition offsets

At-least-once pattern: process record, then commit offset At-most-once pattern: commit offset, then process record (if crash after commit, record is lost) Exactly-once: use Kafka transactions to atomically produce + commit offset

Key facts

• →

  Offsets are per-partition – the same offset value exists independently in every partition

• →

  Offsets never repeat, even after log compaction or record expiry

• →

  Consumer groups commit offsets independently – each group has its own offset position

• →

  The __consumer_offsets internal topic stores all group offset commits (replicated, compacted)

• →

  auto.offset.reset=latest is the default – a new consumer group misses all historical records

Common gotchas