Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

KIP-107 — DeleteRecords

Status: implemented — back to the KIP index.

What the KIP changes in Apache Kafka

Kafka 0.11 added AdminClient.deleteRecords() and the DeleteRecords API (key 21): per partition, advance the log start offset to a caller-chosen target (-1 = purge everything up to the high watermark). Records below the new log start become invisible to Fetch and eligible for physical deletion; the response returns the resulting low watermark.

How kaas implements it

  • The handler (crates/kaas-broker/src/handlers/delete_records.rs, key 21, v0–v2, flexible from v2) gates on partition ownership the same way Produce does: with a cluster coordinator wired, partitions this broker doesn't lead answer NOT_LEADER_OR_FOLLOWER (6). A target past the high watermark maps to OFFSET_OUT_OF_RANGE (1).
  • The storage side (Partition::delete_records in crates/kaas-storage/src/partition.rs) runs under the partition mutex: resolve -1 to the HWM, advance log_start, then physically unlink every closed segment (log + index) that falls entirely below the new log start, and publish a fresh read snapshot. Because only the leader holds open file handles (file-handle ownership), the unlink actually frees space on NFS instead of silly-renaming.
  • The advanced log_start persists via manifest.json, which is written on relinquish / segment roll rather than inline — the reopen path recovers it. It also surfaces everywhere Apache surfaces it: ListOffsets EARLIEST, the Produce response's log_start_offset, and Fetch visibility.
  • The same primitive drives size-based retention: the cleaner (crates/kaas-storage/src/cleaner.rs) computes a target via cleanup_target_for_size_bytes and calls delete_records with it, so there is exactly one log-start-advance code path.
  • One asymmetry against upstream: delete_records drops closed segments only, and the cleaner never touches the active segment either. A purge-to-HWM makes the active segment's records invisible immediately, but its bytes are reclaimed only after the segment rolls and a later cleanup pass drops it.

How it's verified

  • crates/kaas-storage/src/partition.rs unit tests: delete_records_advances_log_start, delete_records_purge_to_hwm, delete_records_past_hwm_is_offset_out_of_range.
  • crates/kaas-storage/src/disk.rs: reopen_recovers_state proves the advanced log start survives an engine restart (HWM 6, log start 2 after reopen).
  • scripts/kafka-delete-records.sh runs the real kafka-delete-records.sh tool end-to-end: produce 10 records, delete to offset 7 via --offset-json-file, assert earliest = 7.