enable.idempotence
reliabilitydefault: true (since Kafka 3.0)
type: boolean
enable.idempotence=true gives each producer a unique PID (Producer ID) and adds a sequence number to every record. The broker deduplicates retried records by tracking the last sequence number per (PID, partition). This prevents duplicate records from network retries. Idempotence is required for exactly-once delivery and is enabled by default since Kafka 3.0.
Details
Without idempotence, a producer retry after a network failure can create duplicate records: 1. Producer sends record, network times out 2. Broker actually wrote the record but ACK was lost 3. Producer retries → duplicate record
With enable.idempotence=true: - Producer receives a PID (Producer ID) from the broker at startup - Each record in each partition gets a monotonic sequence number - Broker tracks (PID, partition, sequence) → deduplicates retries within one producer session - Idempotence is per-session – a producer restart gets a new PID, so cross-session deduplication requires application-level IDs
Required settings when enable.idempotence=true: acks=all (automatically set) retries > 0 (automatically set to Integer.MAX_VALUE) max.in.flight.requests.per.connection ≤ 5 (automatically enforced)
Exactly-once delivery: enable.idempotence=true is a prerequisite for Kafka transactions. With transactions: 1. Producer begins transaction: beginTransaction() 2. Sends records to multiple topics/partitions 3. Commits consumer offsets atomically: sendOffsetsToTransaction() 4. Commits or aborts: commitTransaction() / abortTransaction() The broker guarantees either all records are visible or none are.
Recommended values
| Scenario | Value |
|---|---|
| All production workloads | enable.idempotence=true (default since 3.0) |
| Exactly-once pipelines | enable.idempotence=true + transactional.id=<unique-id> |
| Maximum raw throughput (loss/dupes OK) | enable.idempotence=false (explicitly disable) |