Kafka Producer Configuration

acks controls how many broker acknowledgements the producer waits for before considering a write successful. acks=0: fire-and-forget (no ack, possible data loss). acks=1: leader acknowledges (data loss if leader fails before replication). acks=all/-1: all in-sync replicas acknowledge (no data loss as long as min.insync.replicas is met). acks=all is the default since Kafka 3.0.

Maximum durability (payments, orders): acks=all with min.insync.replicas=2

enable.idempotence=true gives each producer a unique PID (Producer ID) and adds a sequence number to every record. The broker deduplicates retried records by tracking the last sequence number per (PID, partition). This prevents duplicate records from network retries. Idempotence is required for exactly-once delivery and is enabled by default since Kafka 3.0.

All production workloads: enable.idempotence=true (default since 3.0)

batch.size controls the maximum number of bytes in a single batch of records sent to a partition. The producer batches records destined for the same partition. When the batch fills up, it is sent immediately. A larger batch.size reduces per-request overhead and improves throughput at the cost of higher latency. Combine with linger.ms to control when incomplete batches are sent.

High throughput pipeline: batch.size=131072 with linger.ms=20

linger.ms tells the producer to wait up to N milliseconds for more records before sending an incomplete batch. Default 0 sends batches immediately (lowest latency, lower throughput). Increasing linger.ms allows the producer to accumulate more records into fewer, larger batches – improving throughput and compression ratio at the cost of added latency.

Real-time, low latency required: linger.ms=0 (default)

delivery.timeout.ms is the upper bound on the total time a producer will wait for a record to be successfully sent, including retries and queuing. If the record cannot be delivered within this window, the producer reports failure. It must be >= linger.ms + request.timeout.ms. Combined with retries=Integer.MAX_VALUE, delivery.timeout.ms is the primary way to bound delivery attempts.

Critical data (payments, orders): delivery.timeout.ms=120000 with dead-letter on failure

compression.type enables batch-level compression before sending to brokers. Compression reduces network bandwidth and broker disk usage at the cost of producer and consumer CPU. LZ4 offers the best balance of compression ratio and speed. ZSTD provides the highest compression ratio (better than gzip) with good speed. Compression is applied per batch – larger batches compress better.

General production workloads: compression.type=lz4