AMQP vs Kafka
AMQP and Kafka solve messaging from fundamentally different architectural assumptions. AMQP (via RabbitMQ) is a message broker. Publishers push messages to exchanges; exchanges route to queues based on binding rules; consumers pull from queues and acknowledge each message. Once acknowledged, the message is deleted. AMQP is designed for reliable transactional delivery: at-least-once with acknowledgements, optional exactly-once with transactions. Complex routing (direct, fanout, topic, headers exchanges) allows sophisticated message dispatch without changes to publishers or consumers. Kafka is a distributed commit log. Producers append events to topic partitions; consumers read events from partitions by offset (position). Events are retained for a configurable period (hours, days, or indefinitely) regardless of whether consumers have read them. Multiple independent consumer groups can each read the same topic from different offsets – one group processes orders in real-time, another reprocesses historical orders for analytics. The fundamental difference: in AMQP, messages are consumed and deleted. In Kafka, events are appended and retained. This makes Kafka ideal for event sourcing, audit trails, and building derived views from event history. It makes AMQP ideal for task distribution, job queues, and request-reply patterns. Kafka is significantly more operationally complex than RabbitMQ. Kafka requires ZooKeeper or KRaft for cluster coordination, careful partition management, and consumer group offset tracking. RabbitMQ is simpler to deploy and operate for standard messaging use cases.
AMQP (RabbitMQ) is a message broker optimized for complex routing, request-reply patterns, and per-message acknowledgements. Kafka is a distributed log optimized for high-throughput event streaming, replay, and long-term event retention. AMQP delivers and deletes messages; Kafka retains events in immutable log segments. Choose AMQP for task queues and flexible message routing. Choose Kafka for event streaming, event sourcing, and pipelines where consumers need to replay or reprocess history.
| Feature | AMQP | Kafka |
|---|---|---|
| Architecture | Message broker (push-based delivery) | Distributed commit log (pull-based consumption) |
| Message retention | Deleted after consumer acknowledgement | Retained for configurable time regardless of consumption |
| Replay | No – consumed messages are gone | Yes – any consumer can seek to any offset and replay |
| Consumer model | Competing consumers share a queue | Consumer groups: each group gets all events independently |
| Throughput | Thousands to low millions msg/sec per broker | Millions to tens of millions msg/sec at scale |
| Routing | Rich: direct, fanout, topic (wildcards), headers | Simple: topics and partitions, no routing logic |
| Ordering | Per-queue FIFO. No global ordering. | Per-partition ordering. No cross-partition ordering. |
| Acknowledgement | Per-message ack/nack with retry | Offset commit per consumer group (batch) |
| Delivery guarantee | At-least-once (ack). At-most-once (no-ack). Exactly-once with transactions. | At-least-once by default. Exactly-once with idempotent producers + transactions. |
| Protocol | AMQP 0-9-1 (RabbitMQ) or AMQP 1.0 (OASIS) | Kafka binary protocol (custom, no RFC). Also supports AMQP 1.0 via Kafka-AMQP bridge. |
| Schema | No built-in schema enforcement | Confluent Schema Registry (Avro/Protobuf/JSON Schema) |
| Operational complexity | Low-moderate – RabbitMQ simple to deploy | High – ZooKeeper/KRaft cluster, partition management |
| Primary use cases | Task queues, job distribution, request-reply, microservice decoupling | Event streaming, event sourcing, log aggregation, analytics pipelines |
When to use AMQP
AMQP (RabbitMQ) is the right choice for: task queues where workers compete to process jobs (email sending, image processing, order fulfillment), complex routing requirements (send messages to different queues based on content), request-reply patterns (RPC over messaging), dead letter queues for failed message handling, and any scenario where messages should be deleted after processing.
When to use Kafka
Kafka is the right choice for: high-throughput event streaming (millions of events per second), event sourcing and CQRS architectures where the event log is the source of truth, building multiple independent derived views from the same event stream, log aggregation and analytics pipelines, real-time data integration between microservices, and any scenario where replay and reprocessing of historical events is needed.
Common Mistakes
- Using Kafka as a task queue – Kafka's partition-based model makes per-message retry, dead-lettering, and priority queuing awkward. These are natural in AMQP. Kafka is not a drop-in replacement for RabbitMQ task queues.
- Using RabbitMQ for event streaming – once messages are consumed and acknowledged, they are gone. RabbitMQ cannot replay historical events to a new analytics service. Kafka's retention model is purpose-built for this.
- Not planning Kafka partition count carefully – too few partitions limits throughput and consumer parallelism. Partitions cannot be reduced once created (only increased). Plan partition count based on expected throughput and consumer group size.
- Treating Kafka consumer group lag as a queue depth – Kafka lag (how far behind a consumer group is) is not the same as message backlog. Multiple consumer groups can have different lags; messages are not removed when one group consumes them.
FAQ
Can I use Kafka and RabbitMQ together?
Yes, and this is common. Kafka handles high-volume event streams and analytics pipelines. RabbitMQ handles task queues and request-reply patterns. The two serve different workloads and complement each other.
Is Kafka always faster than RabbitMQ?
Kafka has higher peak throughput at scale due to sequential disk writes and consumer pull model. RabbitMQ delivers lower latency for individual messages. For typical task queue workloads (thousands of messages per second), RabbitMQ performance is more than sufficient. Kafka's throughput advantage matters at hundreds of thousands to millions of events per second.