KIP-354 — tombstone retention

Status: partial — config surface only; tombstone expiry is not yet enforced. See the KIP index.

What the KIP changes in Apache Kafka

Upstream, KIP-354 (Kafka 2.3) is titled "Add a Maximum Log Compaction Lag" and adds max.compaction.lag.ms: an upper bound on how long a record — including a tombstone — can sit uncompacted, so deletion is guaranteed to happen within a bounded time (the GDPR-shaped use case). The tombstone-lifetime knob itself, delete.retention.ms, predates the KIP by years: it bounds how long a delete marker survives after its segment is cleaned, so lagging consumers replaying the compacted log still observe the deletion.

What kaas actually has

kaas's source (crates/kaas-storage/src/topicconfig.rs) attributes its delete.retention.ms field to KIP-354 — the shared goal is bounded tombstone lifetime — but to be precise: kaas implements the delete.retention.ms config surface, and max.compaction.lag.ms does not exist anywhere in kaas (no config field, no CRD entry, no defaults row). The declared kaas semantics also differ from Apache in two ways:

• Expiry granularity is per batch, keyed on the batch baseTimestamp, where Apache decides per record — a consequence of the storage engine never opening batches (Storage hot path).
• 0 is documented as "tombstones live forever" (crates/kaas-broker/src/topic_config_defaults.rs), where Apache's 0 means tombstones are removable as soon as the segment is cleaned.

Like KIP-58, what ships today is the plumbing, not the enforcement. The field flows from KafkaTopic.spec.config through the operator into /data/<topic>/.config.json (delete_retention_ms, topicconfig.rs), IncrementalAlterConfigs accepts the key (crates/kaas-broker/src/topic_cr_writer.rs), and DescribeConfigs advertises a default of 86400000 (24 h, matching Apache) from topic_config_defaults.rs. But the compactor that would drop expired tombstones does not exist yet: crates/kaas-storage/src/cleaner.rs is size-based retention only and marks the compactor with both gh #116 knobs as follow-up on gh #158. Since no compaction runs at all, no tombstone is ever dropped — currently indistinguishable from delete.retention.ms = 0 under kaas's declared semantics, and safely conservative with respect to the KIP's guaranteed-removal goal (nothing disappears early; nothing is guaranteed to disappear).

How it's verified

Config-plumbing level only: roundtrip_preserves_unset_vs_zero in crates/kaas-storage/src/topicconfig.rs round-trips a set delete_retention_ms next to unset siblings, and scripts/kafka-configs.sh covers the --describe / --alter surface against a live broker. Tombstone-expiry behaviour has no tests because the behaviour is not implemented.