Apache Avro
ActiveApache Avro is a schema-based binary serialization format developed as part of the Hadoop project. Schemas are defined in JSON and embedded in or sent alongside data. Avro is the dominant wire format for Apache Kafka and the Confluent Schema Registry, enabling schema evolution without breaking consumers.
In one line
Apache Avro is a schema-first binary serialization format from the Apache Hadoop ecosystem (spec 1.11). Schemas are defined in JSON. Data is serialized without field names – only values, in schema-defined order – making Avro significantly more compact than JSON. Schema evolution rules (BACKWARD, FORWARD, FULL compatibility) let producers and consumers use different schema versions. Avro is the standard wire format for Apache Kafka with Confluent Schema Registry.
Quick Reference
| Field | Size | Description |
|---|---|---|
| Schema format | JSON | Schemas are JSON documents. Primitive types: null, boolean, int, long, float, double, bytes, string. Complex types: record, enum, array, map, union, fixed. |
| Encoding | Binary (no tags) | Values encoded in schema-defined field order with no field names or type tags. Schema required for decoding. Much smaller than JSON or Protobuf for wide rows. |
| Schema evolution | 3 modes | BACKWARD: new schema reads old data. FORWARD: old schema reads new data. FULL: both directions. Default values required for BACKWARD compatibility when adding fields. |
| Null handling | union [null, T] | Fields are non-nullable by default. Nullable fields use union type: ["null", "string"]. The null branch must be first for BACKWARD compatibility. |
| Logical types | 7 defined | date (int days since epoch), time-millis, time-micros, timestamp-millis, timestamp-micros, local-timestamp-*, duration, decimal (bytes/fixed + precision/scale). |
| MIME type | avro/binary | application/vnd.apache.avro+binary (binary) and application/vnd.apache.avro+json (JSON encoding). No IANA registration. |
| Container file | .avro | Object Container File: 4-byte magic OBj\x01, schema JSON, codec (null/deflate/snappy/zstd), blocks of records. Self-describing. |
| Single-object enc. | 10-byte header | Magic 2 bytes (0xC3 0x01) + 8-byte schema fingerprint + binary payload. Used in Kafka with Schema Registry. |
Key Characteristics
Schema-first compact
No field names in wire format – only values in schema order. A row with 100 fields costs zero overhead per field name. Best compression ratio of the major formats for wide schemas.
Schema evolution
BACKWARD, FORWARD, and FULL compatibility modes. Schema Registry enforces compatibility at publish time. Consumers can lag producers by multiple schema versions.
Kafka native
Confluent Schema Registry stores schemas by ID. Kafka messages carry a 5-byte prefix: 0x00 + 4-byte schema ID. Consumer fetches schema by ID to decode.
Schema required
Avro binary data is meaningless without the schema. The writer schema and reader schema must be reconciled for decoding. JSON encoding avoids this at the cost of size.
Message Format
// Avro schema (JSON) – User record
{
"type": "record",
"name": "User",
"namespace": "com.example",
"fields": [
{"name": "id", "type": "long"},
{"name": "name", "type": "string"},
{"name": "email", "type": ["null", "string"], "default": null},
{"name": "score", "type": "double", "default": 0.0}
]
}
// Schema evolution: adding a field with default
// BACKWARD compatible – old readers ignore unknown, new readers use default
{"name": "tier", "type": "string", "default": "free"}// Kafka message with Confluent Schema Registry framing
// [0x00][schema_id 4 bytes BE][avro binary payload]
// Example: schema ID 42
00 00 00 00 2A <avro bytes...>
// Avro binary encoding of User{id:1, name:"Alice", email:null, score:98.5}
// id (long, zigzag varint): 0x02 (1 encoded as 2)
// name (string, len + bytes): 0x0A 41 6C 69 63 65 (5 chars "Alice")
// email (union index 0 = null): 0x00
// score (double, 8 bytes LE): 0x00 00 00 00 00 A0 58 40
// Object Container File header
4F 62 6A 01 // Magic: "OBj" + 0x01
<meta block: schema JSON, codec>
<sync marker 16 bytes>
<data blocks>