Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

KIP-101 — leader-epoch-based log truncation

Status: partial — see the KIP index.

What's missing

  • The leader-epoch cache and lookup. DiskStorageEngine::offset_for_leader_epoch (crates/kaas-storage/src/disk.rs) returns the (-1, -1) sentinel — the segment epochs are recorded on disk but no epoch→offset cache is materialized.
  • Wire key 23 (OffsetForLeaderEpoch) is not registered — clients see it as unsupported via ApiVersions (it appears in the matrix's gap table).

What the KIP changes in Apache Kafka

KIP-101 replaced high-watermark-based follower truncation with leader-epoch-based truncation: each replica tracks which offset ranges were written under which leader epoch, and a rejoining follower asks the leader "what's the last offset for epoch E?" (OffsetForLeaderEpoch) to truncate divergent history precisely instead of over- or under-truncating.

What kaas has

Half of the KIP's substance — the epoch bookkeeping — exists structurally: every segment filename carries the leader epoch it was written under ({epoch:08x}-{base_offset:020d}.log), so the divergent-history problem the KIP solves is prevented by construction rather than repaired after the fact. A deposed leader's late writes land in files named with a dead epoch and are never part of the new leader's log (storage engine).

Because kaas has no replication, there are no followers to truncate — the KIP's driving scenario doesn't arise inside the broker. What does still want the lookup is the client-side surface: KIP-320-aware consumers send OffsetForLeaderEpoch to detect log truncation after unclean leadership changes. Registering key 23 backed by a real epoch cache is the tracked follow-up.

How the partial state is verified

The segment-epoch construction is exercised by the storage engine's unit tests and the takeover paths in bins/kaas/tests/cluster_smoke.rs; the stubbed lookup is explicit in crates/kaas-storage/src/disk.rs (grep for offset_for_leader_epoch).