Followers/learners can't catch up under sustained write — snapshot retention (NumLogFiles>4 × maxNumEntries=30000 ≈ 120k) is hardcoded

[MahmoudElhalwany]

Summary

On a large single-group cluster under sustained write, a restarted follower/learner is forced into repeated full snapshot streams and can effectively never catch up, because WAL retention before a forced snapshot is hardcoded to ~120k entries — only minutes of WAL under load.

Environment

• Dgraph v25.3.0
• Single Raft group: 1 alpha leader + 2 learner read-replicas
• ~500 GiB group data, continuous write ingestion (CDC-style)

What happens

1. A learner restarts (or is briefly down).
2. Leader streams a full snapshot (~500 GiB) — takes hours (observed ~37 MiB/s ingest despite idle CPU / 800 MiB/s disk / fast network → the stream, not hardware, is the limit).
3. Writes continue; the leader force-snapshots + truncates its WAL every ~120k entries (minutes).
4. The learner finishes the full snapshot, finds follow-on entries already truncated → re-requests another full snapshot → loops for hours. Never reaches healthy.

Root cause (source, v25.3.0)

• worker/draft.go: calculate := raft.IsEmptySnap(snap) || n.Store.NumLogFiles() > 4
• raftwal/log.go: const maxNumEntries = 30000 (per-file, tied to the fixed 1 MB slot region / entrySize=32)

→ ~4 × 30,000 = 120,000 entries. snapshot-after-entries / snapshot-after-duration can't relax it — the NumLogFiles() > 4 backstop is OR'd in independently. etcd/raft delegates snapshot/compaction to the application, so there's no library knob either.

What we tried

• --raft snapshot-after-entries=200000; snapshot-after-duration=10m → no effect past the backstop.
• Only reliable workaround: pause writes during bootstrap (keeps NumLogFiles() <= 4) — but that halts ingestion for the multi-hour snapshot.

Questions

1. Any supported way to widen this retention that we've missed?
2. Would you accept making the NumLogFiles() > 4 backstop (and/or maxNumEntries) configurable, so large-cluster followers can catch up under sustained write without pausing ingestion?
3. Recommended pattern for (re)bootstrapping a follower on a large group under continuous write, other than pausing writes?