Apache Kafka vs Redis Streams
Both Kafka and Redis Streams implement a persistent, append-only log with consumer group semantics. The architecture, durability model, and operational complexity are very different. Kafka is a distributed system purpose-built for durable, high-throughput streaming. A topic is divided into partitions. Each partition is an ordered, immutable sequence of records stored on disk. Consumer groups independently track their position (offset) in each partition. Records are retained for a configured duration (days, weeks, forever with compaction), regardless of whether anyone has consumed them. This makes Kafka the backbone of event sourcing, CDC (change data capture), data pipelines, and audit logs. Kafka's throughput scales by adding partitions. Its replication model (ISR – in-sync replicas) guarantees no data loss on broker failures. Redis Streams (Redis 5.0+) bring the same conceptual model – append-only log, consumer groups, per-consumer ACK tracking – to Redis's in-memory architecture. A stream entry has an auto-generated ID (timestamp-sequence). Consumer groups track delivered-but-unacknowledged messages in a pending entry list (PEL). XACK acknowledges processing; XCLAIM handles stuck consumers. Redis Streams can hold millions of entries in memory efficiently. With AOF persistence, messages survive restarts. With RDB-only or no persistence, a Redis crash loses recent entries. The critical difference is the durability model. Kafka writes to disk (with OS page cache) and replicates across brokers before acknowledging producers. Redis Streams is in-memory first; durability is optional and limited. For compliance data, financial transactions, or any stream where losing events is unacceptable, Kafka's persistence model is the right choice. Operationally, Kafka is significantly more complex: a Kafka cluster requires ZooKeeper (pre-3.x) or KRaft (3.x+), broker configuration, partition assignment, ISR management, and consumer group rebalancing. Redis Streams is just a Redis data type – if you already run Redis, there is no additional infrastructure. Typical pattern in 2026: Redis Streams for application-internal event fan-out (notifications, activity feeds, job queues) where sub-millisecond latency matters and event history beyond a few hours is not needed. Kafka for inter-service event buses, data pipelines, CDC, and any stream that feeds multiple downstream consumers including analytics, ML, and operational databases.
Kafka is a distributed, durable, partitioned commit log designed for high-throughput streaming at scale: millions of events/sec, years of retention, replayable history, multiple independent consumer groups. Redis Streams is an in-memory append-only log with optional persistence: low-latency fan-out, consumer groups, and simple event queues for moderate throughput. Kafka is the right choice when durability and scale are paramount. Redis Streams fits low-latency event fan-out, job queues, and activity feeds that don't need long-term retention.
| Feature | Apache Kafka | Redis Streams |
|---|---|---|
| Storage model | Disk-backed distributed log (OS page cache + fsync) | In-memory (RDB/AOF persistence optional) |
| Durability guarantee | Configurable acks (0/1/all). acks=all = no data loss if ISR >= min.insync.replicas. | In-memory by default. AOF adds crash durability. No replication within Redis Streams itself. |
| Retention | Time-based (hours, days, infinite) or size-based. Log compaction for latest-value-per-key. | Memory-bounded. MAXLEN trims old entries. No long-term retention. |
| Replayability | Full replay from any offset. Multiple consumer groups start from any position. | Replay possible while entries are in memory (within MAXLEN). No long-term replay. |
| Throughput | Millions of events/sec per cluster. Scales horizontally by adding partitions. | Hundreds of thousands of events/sec on a single Redis node. Limited by memory. |
| Latency | Low but not sub-millisecond – network + disk write. Typically 1–10 ms p99. | Sub-millisecond. In-memory write. Typically 0.1–1 ms p99. |
| Consumer groups | Multiple independent consumer groups, each with its own committed offset per partition. | Multiple consumer groups via XREADGROUP. PEL (pending entry list) tracks unacked messages. |
| Message ordering | Total order within a partition. No cross-partition ordering guarantee. | Total order within a stream. Single stream = single Redis key = single thread. |
| Partitioning | Native topic partitioning. Multiple partitions = parallel consumers. | Single stream per key. Parallelism requires multiple streams + application-level routing. |
| Schema registry | Confluent Schema Registry (Avro, Protobuf, JSON Schema). Enforces schema evolution. | No schema registry. Entries are field-value maps – no type enforcement. |
| Exactly-once | Kafka transactions + idempotent producer = exactly-once semantics end-to-end. | At-least-once (XACK-based). No built-in exactly-once. |
| Ops complexity | High: cluster management, ZooKeeper/KRaft, partition rebalancing, ISR tuning. | Low: standard Redis operation. Streams are just data types on a Redis instance. |
| Managed cloud | Confluent Cloud, AWS MSK, Aiven for Kafka, Upstash Kafka | Redis Cloud, AWS ElastiCache for Redis, Upstash Redis |
| Best for | Data pipelines, CDC, event sourcing, audit logs, inter-service event buses | Job queues, activity feeds, notification fan-out, real-time leaderboards with event history |
When to use Apache Kafka
Kafka is the right choice when: you need guaranteed, durable delivery with no message loss, multiple independent consumer groups need to replay the same events from different positions, you are building a data pipeline that feeds analytics, ML, and operational databases, you need CDC (change data capture) from a database, you need schema evolution enforcement, or your event volume exceeds what a single Redis node can hold in memory.
When to use Redis Streams
Redis Streams is the right choice when: you already run Redis and want a simple job queue or activity feed without additional infrastructure, latency requirements are sub-millisecond, event retention beyond a few hours is not needed, a single Redis node's throughput (hundreds of thousands of ops/sec) is sufficient, or you want the simplicity of a Redis data type without managing a Kafka cluster.
Common Mistakes
- Using Redis Streams without persistence for critical events – if Redis restarts without AOF, all stream entries since the last RDB snapshot are lost. Enable AOF with appendfsync everysec for at-least-once durability. For truly critical events, use Kafka.
- Setting MAXLEN too low – MAXLEN trims stream entries older than the limit. If consumers are slow or offline, entries are trimmed before they are consumed. Set MAXLEN based on realistic consumer lag, not memory pressure alone.
- Under-partitioning Kafka topics – a Kafka topic with 1 partition can only be processed by 1 consumer in a group. Under-partitioned topics are a common bottleneck. Partition count = max parallelism for a consumer group. Increase partition count before you need it (repartitioning is possible but disruptive).
- Using Kafka for simple job queues that Redis Streams handles fine – Kafka has a high operational overhead. For a simple background job queue (send email, resize image, process webhook), Redis Streams (or even a Redis List with BRPOP) is simpler, faster to implement, and requires no additional infrastructure.
- Not implementing dead letter handling in Redis Streams – messages in the pending entry list (PEL) that are not acknowledged accumulate. XCLAIM allows claiming stuck messages. Without a dead letter strategy, failed messages pile up in the PEL indefinitely, consuming memory.
FAQ
Can Redis Streams replace Kafka for event sourcing?
No for production event sourcing at scale. Event sourcing requires long-term, durable retention of all events – the full history is the source of truth. Redis Streams is memory-bounded. Once MAXLEN trims old events, they are gone. Kafka retains events indefinitely (or for a configured period). For event sourcing, Kafka, Pulsar, or an event store (EventStoreDB) are appropriate. Redis Streams can be used for short-lived event windows.
What is the throughput difference in practice?
A single Redis node handles ~500K-1M XADD operations/sec. A Kafka cluster with 3 brokers and properly tuned producers handles 5-20 million records/sec for small messages. For most applications, both are far more than sufficient. The choice should be based on durability, retention, and operational complexity, not throughput alone.
Does Kafka Streams have anything to do with Redis Streams?
No. Kafka Streams is a Java stream processing library for transforming Kafka topics (filter, map, join, aggregate). Redis Streams is a Redis data type (append-only log). They share the word 'Streams' but are completely different technologies at different abstraction levels.