batch.size
performancedefault: 16384 (16 KB)
type: int (bytes)
batch.size controls the maximum number of bytes in a single batch of records sent to a partition. The producer batches records destined for the same partition. When the batch fills up, it is sent immediately. A larger batch.size reduces per-request overhead and improves throughput at the cost of higher latency. Combine with linger.ms to control when incomplete batches are sent.
Details
Kafka producers batch records internally to reduce the number of network requests. Records going to the same partition accumulate in a batch until batch.size bytes is reached or linger.ms elapses.
Batch size and throughput: Larger batches reduce overhead: fewer Produce requests, better network utilization, better compression ratio (if enabled) Larger batches increase latency: the producer waits longer to accumulate enough data Default 16 KB is conservative. For high-throughput workloads, 64 KB–1 MB is common.
batch.size vs record size: A record larger than batch.size is always sent in its own request (not dropped) batch.size is a per-partition maximum
Relationship with buffer.memory: buffer.memory (default 32 MB) is the total memory for all inflight batches. If the producer is producing faster than brokers accept, batches queue up in buffer.memory until max.block.ms timeout.
Tuning recipe: High throughput, latency OK: batch.size=131072 (128 KB) + linger.ms=20 Low latency, throughput secondary: batch.size=16384 (default) + linger.ms=0 Compressed pipelines: batch.size=1048576 (1 MB) + linger.ms=100 + compression.type=lz4
Recommended values
| Scenario | Value |
|---|---|
| High throughput pipeline | batch.size=131072 with linger.ms=20 |
| Low latency events | batch.size=16384 (default) with linger.ms=0 |
| Compressed high-volume logs | batch.size=1048576 with linger.ms=100 |