KIP-482 — Flexible versions and tagged fields

Status: implemented — back to the KIP index.

What the KIP changes in Apache Kafka

From Kafka 2.4, each API declares a cutover version at which its encoding becomes "flexible": strings, bytes, and arrays switch to compact (unsigned-varint) length prefixes, request headers gain a v2 shape and response headers a v1 shape, and every structure ends in a tagged-field block — an extensible (tag, size, value) section that lets either side attach optional data without bumping the API version. Unknown tags must be skipped, which is what makes the scheme forward-compatible.

How kaas implements it

The whole mechanism lives in crates/kaas-codec:

• crates/kaas-codec/src/tagged.rs — the envelope: uvarint(num_fields) || (uvarint(tag), uvarint(len), bytes)*. read consumes and discards every tag (Apache's documented forward-compat contract); read_into surfaces them for the handlers that inspect specific tags; write_empty emits the single-byte uvarint(0) that is by far the most common case on the wire.
• crates/kaas-codec/src/primitives.rs — compact strings, bytes, and arrays with the KIP's count + 1 length encoding (read_compact_string, read_compact_array_len, and friends).
• crates/kaas-codec/src/headers.rs — the HeaderVersion V0/V1/V2 split: flexible requests use v2 headers (client id + tagged block), flexible responses use v1 (correlation id + tagged block).
• crates/kaas-codec/src/api/registry.rs — each ApiSpec carries min_flexible: Option<i16>; is_flexible(version) and the per-API header-version functions mirror Apache's ApiKeys.requestHeaderVersion(apiKey, apiVersion) table. The "Flexible" column of the API matrix is generated from this table, so the docs cannot disagree with what ApiVersions advertises.

One honest asymmetry: kaas reads tagged fields everywhere but never writes a non-empty block — every response ends in write_empty. That is also what Apache 3.7 does on most paths; kaas simply has no optional server-side tags to attach yet. See Wire protocol & framing for how this sits alongside the byte-opacity contract.

How it's verified

• crates/kaas-codec/src/tagged.rs unit tests: empty_block_roundtrips, multi_field_roundtrip, unknown_tags_discarded_by_read.
• crates/kaas-codec/src/headers.rs: request_v2_flexible_tagged_block, response_v1_flexible_with_tag_block (asserts the exact 5-byte shape).
• crates/kaas-codec/src/api/registry.rs: flex_predicate pins the flexible-from version for ApiVersions itself.
• Every per-API codec module round-trips its flexible versions; the leave_group.rs v5 Java-client fixture is a real regression pin — see KIP-800 for the bug it caught.
• scripts/kafka-broker-api-versions.sh exercises the ApiVersions response (itself a flexible API) against a live broker.