Apache Kafka
ActiveApache Kafka is a distributed event streaming platform built around an immutable, append-only commit log. Producers append records to topic partitions; consumers read records by tracking their offset position. Unlike traditional message queues, Kafka retains records for a configurable period – consumers can replay history, rewind, and build new derived views from existing events. Kafka handles millions of events per second at LinkedIn, Uber, Airbnb, and most large-scale data architectures.
In one line
Apache Kafka is a distributed append-only log. Producers write records to topic partitions; each record has an offset (unique position within the partition). Consumer groups track their own offsets independently – multiple groups can consume the same topic at different positions. Brokers replicate partitions across the cluster (leader + followers). Records are retained by time or size. Kafka is the standard for event streaming, event sourcing, real-time data pipelines, and log aggregation.
Quick Reference
| Field | Size | Description |
|---|---|---|
| Topic | Named log | A named, ordered, immutable sequence of records. Topics are divided into partitions. Topic names are UTF-8 strings. Default: auto-created if auto.create.topics.enable=true (disable in production). |
| Partition | Ordered sub-log | The unit of parallelism. Each partition is an independent ordered log. A topic with N partitions can be consumed by up to N consumers in a group simultaneously. Partition count can be increased but not decreased. |
| Offset | uint64 position | A monotonically increasing integer that uniquely identifies a record's position within a partition. Offsets never repeat. Consumers track their committed offset to resume after restart. |
| Broker | Kafka server node | A Kafka server that stores partitions and handles producer/consumer requests. One broker per partition is the leader (handles all reads and writes); others are followers (replicate from leader). |
| Replication factor | N copies | How many brokers hold a copy of each partition. replication.factor=3 means 1 leader + 2 followers. Recommended minimum: 3 for production. ISR (In-Sync Replicas) is the subset of followers caught up to the leader. |
| Consumer group | Named subscriber set | A set of consumers identified by a group.id string. Each partition is assigned to exactly one consumer in the group. Multiple groups read the same topic independently at their own offsets. One group = one independent view of the topic. |
| Committed offset | Checkpoint position | The offset a consumer commits to __consumer_offsets as 'processed up to here'. On restart, consumption resumes from the committed offset. auto.commit.enable=true (default) commits periodically; manual commit gives exact-position control. |
| Retention | Time or size | log.retention.hours (default 168 = 7 days) and log.retention.bytes control when old segments are deleted. Unlike AMQP, Kafka never deletes based on consumption – records are deleted only when retention expires. |
| Segment | Log file chunk | Partitions are physically stored as segment files (default 1 GB). Kafka only deletes complete segments. A new segment is created when log.segment.bytes or log.roll.hours is exceeded. The active segment is never deleted. |
| Compaction | Log compaction | log.cleanup.policy=compact keeps only the most recent record per key, removing all older values. Used for maintaining current state (like a database changelog). Mixed: log.cleanup.policy=delete,compact applies both. |
Key Characteristics
Append-only immutable log
Records are never modified or deleted during retention. Consumers can seek to any offset, replay from beginning, and build new derived views. This is fundamentally different from queues where messages are deleted after consumption.
Sequential disk I/O
Kafka achieves high throughput by writing records sequentially to disk and using the OS page cache. Sequential writes are nearly as fast as memory on modern SSDs. Zero-copy transfer (sendfile syscall) moves data from disk to network without CPU involvement.
Decoupled producers and consumers
Producers write at their own rate; consumers read at their own rate. A slow consumer does not block producers or other consumer groups. Consumer groups can be added or removed without any producer changes.
Partition ordering, not global ordering
Kafka guarantees message order within a partition, not across partitions. If global ordering is required, use a single partition (sacrifices parallelism) or include a sequence number in the message payload.
Message Format
# Kafka record format (RecordBatch, message format v2 – Kafka 0.11+)
# Each record in a batch:
# offset delta (varint) – position within the batch
# timestamp delta (varint) – ms since batch base timestamp
# key length (varint) – -1 for null key
# key bytes – partition routing key
# value length (varint) – -1 for null value (tombstone)
# value bytes – application payload (Avro/JSON/Protobuf)
# headers count (varint) – number of headers
# headers – key-value metadata (tracing IDs, schema version, etc.)
# ProduceRequest (API Key 0) – simplified
Request Headers: api_key=0, api_version=9
transactional_id: null
acks: -1 # -1 = all, 1 = leader only, 0 = fire-and-forget
timeout_ms: 30000
topic_data:
- topic: "orders"
partition_data:
- partition: 0
records: [RecordBatch]# ProduceResponse
topic: "orders"
partition: 0
error_code: 0 # 0 = success
base_offset: 10423 # first offset of the written batch
log_append_time: -1
log_start_offset: 0
# FetchRequest (API Key 1) – consumer reads
Request Headers: api_key=1, api_version=13
replica_id: -1 # -1 = consumer (not broker replica)
max_wait_ms: 500 min_bytes: 1
max_bytes: 52428800
topics:
- topic: "orders"
partitions:
- partition: 0
fetch_offset: 10424 # where consumer left off
partition_max_bytes: 1048576
# FetchResponse
topic: "orders"
partition: 0
error_code: 0
high_watermark: 10500 # last committed offset on leader
records: [RecordBatch] # records from fetch_offset to now
# Consumer offset commit (API Key 8)
group_id: "order-processor-v2"
topics:
- topic: "orders"
partitions:
- partition: 0
committed_offset: 10430 # "I have processed up to here"