Message Serialization (Avro/Protobuf)

TL;DR

Explanation

JSON: human-readable, no schema, large. Avro: binary, schema required, schema registry for evolution. Protobuf: binary, .proto schema, excellent language support, smaller than Avro. MessagePack: binary JSON (no schema). Schema registry (Confluent): stores Avro/Protobuf schemas, enforces compatibility (backward/forward/full). Schema evolution: Avro backward compatible — add optional fields with defaults. Protobuf: add fields with new numbers, never reuse numbers. For PHP: use avro-php, google/protobuf PHP library. JSON is fine for low-volume; use binary for high-throughput Kafka pipelines.

Common Misconception

Why It Matters

Common Mistakes

• No schema validation on JSON messages — producer sends wrong structure, consumer crashes.
• Not using a schema registry — schema changes break consumers silently.
• Reusing Protobuf field numbers — causes decoding errors in old consumers.

Code Examples

// JSON — verbose, no schema enforcement:
$producer->send(json_encode(['userId' => 1, 'amount' => '10.00'])); // Is amount int or string?

// Protobuf — typed, compact:
$msg = new OrderCreated();
$msg->setUserId(1);
$msg->setAmountCents(1000); // Explicit type
$producer->send($msg->serializeToString());

// Consumer:
$event = new OrderCreated();
$event->mergeFromString($rawMessage);

References

• ↗ https://protobuf.dev/
• ↗ https://avro.apache.org/

Tags

Related Terms