Every !update pauses searches and requests for the whole rebuild (PAUSE_ON_UPDATE). Most of a rebuild without audio info is the walk. On the operator's 64,136-file NFS library that's about 80 s (Neo's measurements on #914).
update_list.walk_with_sizes() visits one directory at a time. It does scandir(), then entry.stat() for each file, since Linux's d_type gives the type but not the size. On a network mount each of those is a round trip, and none of them overlap. #914 showed what overlapping round trips buys on this mount: 16 → 64 readers went from about 73 to about 236 files a second. QuickList (OmenServe's list maker) walks the same way it reads, ParallelFindNextFile with FileReaderParallelism.
Proposal: walk_with_sizes() lists and stats several directories at once (LIST_SCAN_THREADS, under List rebuild). Each directory is still classified exactly as today: symlinked directories are listed but not descended, an unclassifiable entry goes to onerror, an unstattable file is yielded with size None. Only the order in which directories come back changes, and that already doesn't matter (the rows are sorted before anything is written, #443). LIST_SCAN_THREADS = 1 is today's walk, unchanged. The changelog will carry before/after timings with simulated latency, plus a request for a live measurement.
Every
!updatepauses searches and requests for the whole rebuild (PAUSE_ON_UPDATE). Most of a rebuild without audio info is the walk. On the operator's 64,136-file NFS library that's about 80 s (Neo's measurements on #914).update_list.walk_with_sizes()visits one directory at a time. It doesscandir(), thenentry.stat()for each file, since Linux'sd_typegives the type but not the size. On a network mount each of those is a round trip, and none of them overlap. #914 showed what overlapping round trips buys on this mount: 16 → 64 readers went from about 73 to about 236 files a second. QuickList (OmenServe's list maker) walks the same way it reads,ParallelFindNextFilewithFileReaderParallelism.Proposal:
walk_with_sizes()lists and stats several directories at once (LIST_SCAN_THREADS, under List rebuild). Each directory is still classified exactly as today: symlinked directories are listed but not descended, an unclassifiable entry goes toonerror, an unstattable file is yielded with sizeNone. Only the order in which directories come back changes, and that already doesn't matter (the rows are sorted before anything is written, #443).LIST_SCAN_THREADS = 1is today's walk, unchanged. The changelog will carry before/after timings with simulated latency, plus a request for a live measurement.