We're trying to move predicates and there's reliability issues where the predicates take too long and timeout and fail to move. Being able to move predicates (any size from 0GB to 1TB) should be durable/reliable/as-fast-as-it-can.
Dgraph version: 25.3.7
Dgraph setup: 1 zero, 3 alphas(each with their own group)
Two separate issues:
- Fix the timeout logic
- Improve the speed/reliability of the predicate moves
For the timeout logic, some apparent items that might be worth addressing:
It appears there's a hardcoded timeout of 120 minutes in the tablet.go and so smaller predicates have no issue, it's the larger ones that hit the ceiling. It appears if there's any hiccup and ctx.Done() the whole move is discarded and cannot be resumed.
It also looks like populateKeyValues uses one badger txn per key in the predicate_move.go and posting/writer.go, looks like TxnWriter.Write loops calling SetAt per KV, and SetAt opens/commits a separate managed txn per key. For a 32MB batch of small posting lists that's tens of thousands of CommitAt calls per proposal.
proposeAndWait is called serially with a 3-message channel in predicate_mov.go if size >= 32<<20....the receiver blocks on each 32mb raft round-trip(propose->replicate->fsync->apply) before accepting the next batch. there is no pipelining. chan buffers=3 head-of-line-blocks the sender's stream.Orchestrate goroutines almost immediately.
Sender-side rollup is done inline, single-value-per-key, on the hot path....predicate_move.go rollup decodes every version,merges, re-encodes. On a heavily-mutated predicate this is cpu-bound and runs inside Stream.KeyToList, which badger parallelizes across numgo routines - but NumGo is left at the default and isn't tuned there.
For the speed/reliability:
Every byte goes through the destination raft log, 3x write amplification, serial, fsync-bound...worker/predicate_move(batchAndProposeKeyValues) an worker/draft.go....each 32mb batch marshalled into a raft entry, appended + fsync to raftwal on the dest leader.....the nreplicaed to and fsynced on ever follower's raftwal....then applied via populateKeyValues -> TxnWriter.SetAt - one badger transaction per key(writer.go), which goes through the LSM write path(memtable->l0) with fll compaction pressure.
startTask(opPredMove) only guards the source leader....predicate_move.go - the receiver never registers a task, so on the destination opRollup/opBackup can run concurrently with ingestion, and the comment at line 258 admits it's a guess.....at larger predicates, concurrent rollup on the dest fights the ingest for LSM write bandwidth
Key-count "verification" is cosmetic
predicate_move.go - the receiver returns a count, the sender logs it, and nothing compares it to anything. there's no send-side count, cno checksum, no failure path.
blockTablet holds a cluster-wide write lock on the predicate for the entire move...tablet.go + oracle.go - every commit touching this predicate is aborted for the full duration...for a large predicate move, that could be hours of write-unavaibility on a hot predicate, and it's held even during a post-move CleanPredicate on the source
Zero leadership change or moveOngoing slot loss silently orphans state...tablet.go uses a channel semaphore in zero memory only. if zero leader changes mid-move, blockCommitsOn is lost and ownership was never flipped so the dest now holds stale garbage that only the next successful move's CleanPredicate will remove.
no resumability/checkpointing, any failures restarts from zero...no progress marker, if the context expires, grpc drops, or re-election, or zero restarts, the next attempt re-issues CleanPredicate on the destination which deletes everything already transferred...it also re-streams from key 0
We're trying to move predicates and there's reliability issues where the predicates take too long and timeout and fail to move. Being able to move predicates (any size from 0GB to 1TB) should be durable/reliable/as-fast-as-it-can.
Dgraph version: 25.3.7
Dgraph setup: 1 zero, 3 alphas(each with their own group)
Two separate issues:
For the timeout logic, some apparent items that might be worth addressing:
It appears there's a hardcoded timeout of 120 minutes in the tablet.go and so smaller predicates have no issue, it's the larger ones that hit the ceiling. It appears if there's any hiccup and ctx.Done() the whole move is discarded and cannot be resumed.
It also looks like populateKeyValues uses one badger txn per key in the predicate_move.go and posting/writer.go, looks like TxnWriter.Write loops calling SetAt per KV, and SetAt opens/commits a separate managed txn per key. For a 32MB batch of small posting lists that's tens of thousands of CommitAt calls per proposal.
proposeAndWait is called serially with a 3-message channel in predicate_mov.go
if size >= 32<<20....the receiver blocks on each 32mb raft round-trip(propose->replicate->fsync->apply) before accepting the next batch. there is no pipelining.chan buffers=3head-of-line-blocks the sender's stream.Orchestrate goroutines almost immediately.Sender-side rollup is done inline, single-value-per-key, on the hot path....predicate_move.go rollup decodes every version,merges, re-encodes. On a heavily-mutated predicate this is cpu-bound and runs inside Stream.KeyToList, which badger parallelizes across numgo routines - but NumGo is left at the default and isn't tuned there.
For the speed/reliability:
Every byte goes through the destination raft log, 3x write amplification, serial, fsync-bound...worker/predicate_move(batchAndProposeKeyValues) an worker/draft.go....each 32mb batch marshalled into a raft entry, appended + fsync to raftwal on the dest leader.....the nreplicaed to and fsynced on ever follower's raftwal....then applied via populateKeyValues -> TxnWriter.SetAt - one badger transaction per key(writer.go), which goes through the LSM write path(memtable->l0) with fll compaction pressure.
startTask(opPredMove) only guards the source leader....predicate_move.go - the receiver never registers a task, so on the destination opRollup/opBackup can run concurrently with ingestion, and the comment at line 258 admits it's a guess.....at larger predicates, concurrent rollup on the dest fights the ingest for LSM write bandwidth
Key-count "verification" is cosmetic
predicate_move.go - the receiver returns a count, the sender logs it, and nothing compares it to anything. there's no send-side count, cno checksum, no failure path.
blockTablet holds a cluster-wide write lock on the predicate for the entire move...tablet.go + oracle.go - every commit touching this predicate is aborted for the full duration...for a large predicate move, that could be hours of write-unavaibility on a hot predicate, and it's held even during a post-move CleanPredicate on the source
Zero leadership change or moveOngoing slot loss silently orphans state...tablet.go uses a channel semaphore in zero memory only. if zero leader changes mid-move, blockCommitsOn is lost and ownership was never flipped so the dest now holds stale garbage that only the next successful move's CleanPredicate will remove.
no resumability/checkpointing, any failures restarts from zero...no progress marker, if the context expires, grpc drops, or re-election, or zero restarts, the next attempt re-issues CleanPredicate on the destination which deletes everything already transferred...it also re-streams from key 0