Skip to content

perf(table): load equality deletes lazily per scan task - #1963

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/lazy-equality-delete-loading
Open

perf(table): load equality deletes lazily per scan task#1963
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/lazy-equality-delete-loading

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Load equality deletes per scan task.
  • Keep a shared, read-once cache for delete files and merged file combinations.
  • Start scan workers when the iterator is consumed, so an unread iterator does no equality-delete I/O.
  • Keep equality-field grouping and delete-set reuse.

The old path read every unique equality-delete file before returning the iterator. This change only reads files needed by tasks that are actually processed.

Benchmark

Setup: 10,000 scan tasks, 1,000 equality-delete files, 10 keys per file, 16 workers. Three runs with -benchtime=1s on an Apple M1 Pro.

case time/op bytes/op allocs/op opens/op
eager/all_tasks 12.8 to 13.7 ms 39.6 MB 299k 1,000
lazy/unread 0.45 to 0.49 ms 285 KB 11,023 0
lazy/first_task 0.50 to 0.54 ms 326 KB 11,298 1
lazy/ten_tasks 0.65 to 0.75 ms 666 KB 13,713 10
lazy/full_scan 13.1 to 14.6 ms 39.2 MB 287k 1,000
  • Unread iterator: no equality-delete opens.
  • Partial scans: I/O follows the tasks reached.
  • Full scan: comparable runtime with fewer allocations.

Tests

  • go test ./table -count=1
  • go test -race ./table -count=1
  • go vet ./table

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant