Apache Kafka
debt(d7/e7/b7/t7)
Closest to 'only careful code review or runtime testing' (d7). The detection_hints indicate no automated detection (automated: no), and the listed tools (rdkafka, confluent-platform, kafka-php) are Kafka client libraries, not linting or static analysis tools. Misuse patterns — wrong partition count, missing consumer groups, not handling rebalances, or using Kafka for simple task queues — only surface under load or through architectural review, not through any automated tooling.
Closest to 'cross-cutting refactor across the codebase' (e7). The quick_fix frames this as a foundational architectural choice (Kafka vs. SQS/RabbitMQ). Replacing Kafka with a simpler queue, or correcting a too-few-partitions design at scale, requires touching producers, consumers, deployment configuration, and potentially data pipeline integrations across the codebase. It is not a single-file fix.
Closest to 'strong gravitational pull' (e7). Applies to web, cli, and queue-worker contexts per applies_to. Kafka's partitioning model, consumer group semantics, and operational infrastructure (brokers, ZooKeeper/KRaft, schema registry) shape every downstream service and data flow decision. The tags (distributed-systems, streaming) confirm this is load-bearing infrastructure. Every new consumer or producer must conform to Kafka's model.
Closest to 'serious trap' (t7). The misconception field explicitly states: 'Kafka replaces RabbitMQ — they serve different purposes.' A competent developer familiar with message queues will assume Kafka is a drop-in upgrade to RabbitMQ, when in fact they have fundamentally different delivery and consumption semantics. The common_mistakes reinforce multiple non-obvious behavioral gotchas (consumer groups, partition limits, rebalance handling) that contradict intuitions built from other queue systems.
Also Known As
TL;DR
Explanation
Kafka differs from RabbitMQ fundamentally: messages are retained for a configurable period (not deleted on consumption), ordered within partitions, and replayable from any offset. Consumers track their own offset — multiple consumer groups can independently read the same topic. Partitions enable parallelism: a topic with 6 partitions supports 6 concurrent consumers in a group. Use Kafka for event sourcing, activity feeds, log aggregation, and stream processing. Use RabbitMQ for task queues where messages should not be replayed.
Diagram
flowchart TD
P1[Producer] & P2[Producer] --> T[Topic: orders]
subgraph Partitions
T --> PA[Partition 0]
T --> PB[Partition 1]
T --> PC[Partition 2]
end
subgraph Consumer Group A
PA --> C1[Consumer 1]
PB --> C2[Consumer 2]
PC --> C3[Consumer 3]
end
subgraph Consumer Group B
PA & PB & PC --> C4[Consumer 4<br/>independent offset]
end
style T fill:#6e40c9,color:#fff
style PA fill:#1f6feb,color:#fff
style PB fill:#1f6feb,color:#fff
style PC fill:#1f6feb,color:#fff
Common Misconception
Why It Matters
Common Mistakes
- Too few partitions — you cannot scale consumers beyond the partition count; add partitions before you need them.
- No consumer group — a consumer without a group reads from the beginning on every restart.
- Not handling rebalances — partition reassignment during consumer group scaling interrupts processing.
- Using Kafka for simple task queues — the operational overhead is not justified; use RabbitMQ or SQS.
Code Examples
// Single partition — only one consumer processes, no parallelism:
$admin->createTopics([new NewTopic('orders', 1, 1)]);
// With 1 partition, adding more consumer instances does nothing
// All 1000 orders/sec go through one consumer// Multiple partitions — parallel processing:
$admin->createTopics([new NewTopic(
'orders',
numPartitions: 12, // 12 partitions = 12 parallel consumers max
replicationFactor: 3 // Replicated across 3 brokers — fault tolerant
)]);
// Consumer group — each partition assigned to one consumer:
$consumer->subscribe(['orders']);
// 12 consumer instances each get ~1 partition — parallel processing