KIP-447 — EOS v2

Status: implemented — see the KIP index.

What the KIP changes in Apache Kafka

KIP-447 (Kafka 2.5) made exactly-once consume-process-produce scale. EOS v1 required one producer per input partition, because zombie fencing hung entirely off the transactional.id. v2 lets one producer serve a whole process: sendOffsetsToTransaction takes the consumer group's metadata, TxnOffsetCommit v3 carries member.id / generation.id / group.instance.id, and the group coordinator fences offset commits from producers whose consumer generation is stale. Offsets committed inside a transaction stay invisible until the transaction commits.

How kaas implements it

The offsets-in-transactions flow rides the KIP-98 machinery; the state walkthrough is in Transactions & idempotence.

• AddOffsetsToTxn (key 25, crates/kaas-broker/src/handlers/add_offsets_to_txn.rs) records the consumer group on the transaction's entry in the txn state store, so EndTxn later knows which groups to settle.
• TxnOffsetCommit (key 28, v0–v3, crates/kaas-broker/src/handlers/txn_offset_commit.rs) runs on the group coordinator (non-coordinators return NOT_COORDINATOR) and stages the offsets in the pending layer of crates/kaas-coordinator/src/offset_store.rs, keyed on (group_id, producer_id). Staged offsets are not visible to OffsetFetch (TxnOffsetCommit).
• EndTxn (key 26) fires the offset hook for every recorded group: commit materialises the pending offsets into the group's committed set (commit_pending); abort discards them (discard_pending). The timeout reaper's abort path fires the same hook, so a reaped transaction can't leak half-committed offsets.

The pending layer is memory-only by design — an unfinished transaction's staged offsets reset on broker restart, which the source frames as Apache's "in-flight offsets aren't recovered" contract. Note the shape of that trade honestly: Apache stages pending commits in the replicated __consumer_offsets log, so its window is narrower; in kaas a group coordinator restart between TxnOffsetCommit and EndTxn(commit) drops the staged offsets, and the application re-processes from the last committed offset (at-least-once across that crash, never offset-without-data).

The KIP's headline fencing is not enforced: TxnOffsetCommit v3's generation_id / member_id / group_instance_id are decoded (crates/kaas-codec/src/api/txn_offset_commit.rs) but the handler never checks them against the group's live generation, so a zombie producer is fenced by its producer epoch (KIP-360) rather than by consumer-group generation. The wire surface a v2 client needs is complete; the extra fencing layer is not.

How it's verified

bins/kaas/tests/eos_v2.rs drives the full v2 round trip (InitProducerId → AddPartitionsToTxn → transactional Produce → AddOffsetsToTxn → TxnOffsetCommit → EndTxn) over real TCP: eos_commit_path_records_visible_to_read_committed and eos_abort_path_populates_aborted_transactions. The staging contract is pinned by pending_invisible_to_fetch_until_commit_pending and discard_pending_drops_unmaterialised_offsets in offset_store.rs, and by happy_path_stages_pending_offsets / no_manager_returns_not_coordinator in the handler. Shell suite: scripts/kafka-txn-coordinator.sh probes the coordinator wire surface; scripts/kafka-transactions.sh covers the admin tool (the Kafka 4.x CLI dropped --transactional-id from kafka-verifiable-producer, so the producer-side flow lives in the integration tests above).