compression.type
compressiondefault: none
type: string (none, gzip, snappy, lz4, zstd)
compression.type enables batch-level compression before sending to brokers. Compression reduces network bandwidth and broker disk usage at the cost of producer and consumer CPU. LZ4 offers the best balance of compression ratio and speed. ZSTD provides the highest compression ratio (better than gzip) with good speed. Compression is applied per batch – larger batches compress better.
Details
Kafka compresses entire batches (not individual records). The compression header is stored in the RecordBatch metadata. Consumers decompress automatically.
Algorithm comparison: none: 0% savings, 0% CPU overhead, 0% latency added gzip: ~60–80% savings, high CPU, high latency, no longer recommended snappy: ~40–50% savings, low CPU, low latency, good balance lz4: ~40–60% savings, very low CPU (hardware-assisted), very low latency. Recommended default. zstd: ~60–80% savings, moderate CPU, moderate latency. Best ratio/speed tradeoff for cold data.
When to use compression: Always recommended for production workloads. Network bandwidth and broker storage are usually the bottlenecks, not CPU. Essential for: log aggregation (text compresses extremely well), JSON events (repeated field names), Avro/Protobuf schemas (schema overhead compresses well)
Broker-side decompression: If the producer uses lz4 and broker.compression.type=gzip, the broker decompresses and recompresses. Avoid mismatched compression by setting broker compression.type to match producer, or use 'producer' to pass through as-is.
Compression and batching: Compression ratio improves dramatically with larger batches. A 16 KB batch of JSON logs at 30% ratio. A 1 MB batch of the same logs at 85% ratio. Always pair high compression with high linger.ms and batch.size.
Recommended values
| Scenario | Value |
|---|---|
| General production workloads | compression.type=lz4 |
| Maximum compression, disk-bound | compression.type=zstd |
| Legacy compatibility | compression.type=snappy |
| Low-volume or latency-sensitive | compression.type=none (default) |