Topics & Partitions

TL;DR

Explanation

Topic: a named stream of records. Partition: an ordered, immutable log. Records appended with offset. Multiple partitions per topic: parallel reads/writes. Partition assignment: by key hash (same key → same partition → ordering), round-robin (no key → load balanced → no ordering), custom partitioner. Replication: each partition has one leader and N-1 followers. Leader handles reads/writes; followers replicate. Partition count: set at creation (can increase, never decrease). More partitions → more parallelism → more overhead (file handles, memory). Typical: 3-12 partitions for throughput, 1 for strict ordering.

Common Misconception

Why It Matters

Common Mistakes

• Too many partitions for the workload — overhead without benefit.
• Changing partition count without updating consumers — key routing changes.
• No replication in production — single partition leader = single point of failure.

Code Examples

# 1 partition — no parallelism:
kafka-topics --create --topic orders --partitions 1 --replication-factor 1
# 1000 partitions — unnecessary overhead

# Balanced: 6 partitions, 3 replicas:
kafka-topics --create --topic orders --partitions 6 --replication-factor 3

# PHP producer with partition key:
$conf->set('partitioner', 'consistent_random'); // Same key = same partition
$producer->produce(RD_KAFKA_PARTITION_UA, 0, $payload, $orderId); // Key = orderId

References

• ↗ https://kafka.apache.org/documentation/#intro_topics

Tags

Related Terms