Retention and Compaction
storageKafka retains records by time (log.retention.hours, default 7 days) or size (log.retention.bytes). Records are stored in segment files; only complete segments are deleted. Log compaction is an alternative policy that keeps only the latest value per key, indefinitely – enabling Kafka to serve as a durable key-value store for event-sourced systems.
Details
Retention controls when Kafka discards old records. Unlike AMQP where messages are deleted after consumption, Kafka keeps records for all consumers until the retention policy triggers.
Time-based retention: log.retention.hours=168 (7 days, default) log.retention.minutes, log.retention.ms: finer granularity The segment's last-modified time determines when it is eligible for deletion
Size-based retention: log.retention.bytes=-1 (disabled by default) Applies per partition, not per topic. A topic with 8 partitions and log.retention.bytes=1GB retains up to 8 GB total. When both time and size are set, whichever limit is reached first triggers deletion.
Segment files: Partitions are stored as segment files of log.segment.bytes (default 1 GB) each. An active segment (currently being written) is never deleted. Old segments are deleted when all their records are past retention. Smaller segments = more frequent deletion = more predictable disk usage. Larger segments = fewer files = lower overhead.
Log compaction (log.cleanup.policy=compact): Kafka scans the log and keeps only the most recent record for each key. A record with a null value (tombstone) marks the key for deletion. Compaction runs in the background without blocking producers/consumers. Use cases: database changelogs (Debezium CDC), materialized views, current state tables. Compacted topics have no time-based guarantee – a key's latest record is kept indefinitely.
Mixed retention (log.cleanup.policy=delete,compact): Applies both: compact the log to keep latest per key, but also delete records older than retention time. Default for __consumer_offsets topic.
Key facts
- →
Records are retained based on time or size, NOT based on whether consumers have read them
- →
Only complete segments are eligible for deletion – active segment is always preserved
- →
Compaction keeps latest value per key indefinitely – enables event-sourced state reconstruction
- →
Tombstone records (null value) mark a key for deletion during compaction
- →
Default 7-day retention means consumers can fall behind by up to 7 days before losing access to records
Common gotchas
log.retention.bytes applies per partition – a topic with 100 partitions and log.retention.bytes=10GB retains 1 TB total
Compaction is not real-time – there is always a compaction lag. A consumer may see older values until compaction runs. Do not assume the compacted topic has only the latest value at any instant.
Setting retention too short causes consumer lag issues – if a consumer is down for longer than retention, it misses records permanently (auto.offset.reset=earliest will land at the retention boundary, not the beginning)