Tombstones & superseded versions accumulate in L6 (StaleData=0, Score=0), never reclaimed by automatic compaction

[jojojonathan]

Summary

Tombstones and superseded key versions accumulate in the bottom-most level (L6) and are never reclaimed by automatic compaction, even after restart, tuning NumCompactors, or running RunValueLogGC. Only Flatten reclaims the space. I'm reporting this to understand whether the behavior below is expected, a misconfiguration on my side, or a bug — I could not find a config that makes automatic compaction reclaim these entries.

Environment

• badger v4.9.2
• Pure Go build, CGO_ENABLED=0, linux/arm64
• LSM-only workload: values are small (~1 KB each), all inlined in the LSM tree (value log is effectively empty)

Workload shape

• Keys are append-only and monotonically increasing (each key is unique; no overwrites of the same key by the application).
• Typical lifecycle per key: txn.Set(key, ~1KB value) followed some time later by txn.Delete(key) (delete happens in a separate transaction). So each deleted key ends up with 2 versions: the original value + a tombstone.
• Periods of heavy write+delete bursts, followed by long idle periods with little to no new writes.
• NumVersionsToKeep is left at the default (1).

Observed state (representative DB, ~898 MB on disk)

badger info shows almost all data sitting in L6:

Level 5 [B]: NumTables: 19. Size: 19 MiB of 88 MiB. Score: 0.00  StaleData: 44 MiB
Level 6 [ ]: NumTables: 120. Size: 878 MiB of 878 MiB. Score: 0.00  StaleData: 0 B
Total SST size: 898 MiB

A custom read-only scan (iterating with AllVersions=true) over the same DB reports:

• ~1.01M distinct keys
• ~1.84M total entries (versions)
• ~1.00M tombstones
• ~0.84M non-tombstone entries that are older versions of keys whose latest version is a tombstone (i.e. the superseded values that NumVersionsToKeep=1 should allow to be dropped)

Effective (latest-version, non-deleted) data is only ~8 MB, but the DB holds 898 MB.

The notable mismatch: L6 reports StaleData: 0 B while it physically contains ~870 MB of tombstones + superseded versions.

What does NOT reclaim the space

On a copy of this DB I tried, with no effect on L6 (tombstone count and SST size unchanged, L6 Score stays 0.00, StaleData stays 0 B):

1. Opening the DB read/write and letting background compaction run for 120–180 s.
2. Increasing NumCompactors (tried 8).
3. RunValueLogGC repeatedly (returns ErrNoRewrite; expected since vlog is empty, but noting for completeness).
4. Close + reopen (the "restart triggers compaction" path).

During these attempts, no L6 compaction is ever scheduled (no compaction log lines for L6).

Only Flatten reclaims it: the same DB goes from 898 MB / 140 SSTs to 8.9 MB / 5 SSTs.

What I'd like to understand

1. Is it expected that tombstones + superseded versions in the bottom level keep StaleData = 0 and Score = 0.00, so that fillMaxLevelTables never picks them (the StaleDataSize() < 10<<20 guard in levels.go)? If so, what is the intended way for an append-only + per-key "set then delete" workload to get these reclaimed without manual Flatten?
2. Is there a configuration (compaction-related options, NumVersionsToKeep, write pattern) that would let automatic compaction reclaim these entries, or is periodic Flatten/backup-restore the only option for this access pattern?
3. If this is a known limitation, is there a recommended automated maintenance approach for long-running services (something more graceful than scheduling Flatten during low-traffic windows)?

Reproducibility

I could not reproduce the accumulation with a small synthetic DB (tens of thousands of keys) — compaction reclaims tombstones normally at that scale. The accumulation was observed on a DB that had grown to ~900 MB under real write/delete load followed by an idle period. I can try to build a larger reproducer if helpful, but wanted to check first whether the StaleData = 0 vs. physically-present-tombstones mismatch in L6 is already a known/expected condition.

Thank you.