Exactly-Once Delivery

TL;DR

Explanation

True exactly-once requires: idempotent producer (Kafka enable.idempotence=true), transactional producer (write + offset commit in one DB transaction), and transactional consumer. Kafka: exactly-once via transactions — producer writes and commits consumer offsets atomically. Cost: lower throughput, higher latency. Alternatives: at-least-once + idempotent consumer (practically equivalent, more portable). Exactly-once semantics (EOS) in Kafka: enable.idempotence=true + transactional.id + isolation.level=read_committed. RabbitMQ: no native exactly-once — use outbox pattern + deduplication. Most real-world systems use at-least-once + idempotency.

Common Misconception

Why It Matters

Common Mistakes

• Enabling Kafka idempotence without transactional consumer — still at-least-once end-to-end.
• Using exactly-once where at-least-once + idempotency is simpler and sufficient.
• Not understanding that network failures make true exactly-once impossible across systems.

Code Examples

// Claimed exactly-once but not transactional end-to-end:
$producer->send($topic, $message); // Idempotent producer
$consumer->process($message); // Not transactional — can fail after produce

// Kafka transactional exactly-once:
$producer->initTransactions();
$producer->beginTransaction();
try {
    $producer->send($outputTopic, $result);
    $producer->sendOffsetsToTransaction($offsets, $consumerGroupId);
    $producer->commitTransaction();
} catch (Exception $e) {
    $producer->abortTransaction();
}

References

• ↗ https://kafka.apache.org/documentation/#transactions

Tags

Related Terms