Apache Kafka Concepts

TL;DR

Explanation

Kafka core: Topics divided into Partitions. Producers append records. Consumers maintain their own offset — they can replay from any point. Consumer Groups: each group processes each message once; multiple groups get all messages independently. Offsets committed explicitly — enables at-least-once delivery. Retention: configurable (days, size) — Kafka is a durable log, not just a queue. Key concepts: partition key determines which partition a message lands in (same key = same partition = ordering). Replication factor for fault tolerance. Use cases: event sourcing, audit logs, real-time pipelines, stream processing.

Common Misconception

Why It Matters

Common Mistakes

• Using Kafka where a simple job queue suffices — operational complexity isn't always justified.
• Choosing the wrong partition key — uneven distribution creates hot partitions.
• Not handling offset commit correctly — duplicate processing or message skipping.

Code Examples

// Wrong partition key — all messages to one partition:
$producer->produce(0, $message, 'constant-key'); // Hot partition

// Partition by user ID — even distribution:
$producer->produce(RD_KAFKA_PARTITION_UA, $message, $userId);

// Manual offset commit for at-least-once:
$consumer->consume(30000);
processMessage($message);
$consumer->commitAsync(); // Commit only after successful processing

References

• ↗ https://kafka.apache.org/documentation/

Tags

Related Terms