Apache Kafka Concepts
debt(d9/e7/b7/t7)
Closest to 'silent in production until users hit it' (d9). The detection_hints field explicitly states automated: no, and the code_pattern hint (kafka|rdkafka) only identifies Kafka usage, not misuse. Wrong partition keys causing hot partitions, incorrect offset commits causing duplicates or skips — none of these are caught by linters or static analysis; they surface only under production load or through data inconsistency reports.
Closest to 'cross-cutting refactor across the codebase' (e7). The quick_fix describes changes to partition key strategy, offset commit ordering, and replication factor — but partition key changes affect data distribution and consumer topology across the entire system. Fixing a wrong partition key choice means repartitioning topics, migrating consumers, and potentially reprocessing historical data, which is a cross-cutting, multi-service effort. Not quite architectural rework (e9) but well beyond a single component.
Closest to 'strong gravitational pull' (e7). Kafka applies to cli and queue-worker contexts and carries distributed, streaming, and messaging tags. Every downstream service consuming or producing to Kafka must conform to its partitioning, offset management, and replication decisions. The choice of partition key and consumer group strategy shapes how all producers and consumers are written and evolved — a strong gravitational pull that every future change must respect.
Closest to 'serious trap — contradicts how a similar concept works elsewhere' (t7). The misconception explicitly states that developers treat Kafka like a faster RabbitMQ, when it is fundamentally a durable distributed log with replay semantics. This directly contradicts the mental model of traditional message brokers. Common mistakes (wrong partition key, incorrect offset commits) compound this: offset commit behavior in particular inverts expectations (committing before processing feels safe but causes loss; after processing risks duplicates), contradicting intuitions from other queuing systems.
TL;DR
Explanation
Kafka core: Topics divided into Partitions. Producers append records. Consumers maintain their own offset — they can replay from any point. Consumer Groups: each group processes each message once; multiple groups get all messages independently. Offsets committed explicitly — enables at-least-once delivery. Retention: configurable (days, size) — Kafka is a durable log, not just a queue. Key concepts: partition key determines which partition a message lands in (same key = same partition = ordering). Replication factor for fault tolerance. Use cases: event sourcing, audit logs, real-time pipelines, stream processing.
Common Misconception
Why It Matters
Common Mistakes
- Using Kafka where a simple job queue suffices — operational complexity isn't always justified.
- Choosing the wrong partition key — uneven distribution creates hot partitions.
- Not handling offset commit correctly — duplicate processing or message skipping.
Code Examples
// Wrong partition key — all messages to one partition:
$producer->produce(0, $message, 'constant-key'); // Hot partition// Partition by user ID — even distribution:
$producer->produce(RD_KAFKA_PARTITION_UA, $message, $userId);
// Manual offset commit for at-least-once:
$consumer->consume(30000);
processMessage($message);
$consumer->commitAsync(); // Commit only after successful processing