The list can say how long each track is and how good (#567)

[chchatzop]

Closes #567.

Stacked on #913 (→ #906 → main). #906 moved the rebuild limits out of Your list, and this adds a field there. #906 had left the category at 14 of its 16-field cap, while on main it is full. This also touches the same settings maps and tests/support.py as #913. I merged the whole stack (#906 + #913 + this) with today's main, which includes #904/#905/#912, and the full suite passed: 6544 OK.

What it does

LIST_SHOW_AUDIO_INFO (off by default, Your list) adds duration and quality after the size on every MP3 and FLAC row:

!DCCore Artist - Album - 01 - Track.mp3  ::INFO:: 10.3MB 4m31s 320/44.1/JS
!DCCore Artist - Album - 02 - Track.flac  ::INFO:: 16.7MB 2m5s 1115/44.1/S
!DCCore Artist - Album - Front.jpg  ::INFO:: 94.4KB

How the issue's open questions were settled

• Line format: the other servers' spelling, so list.py and other bots already read it. Duration 4m31s (minutes go past 59). Channels S/JS/DC/M/6ch. VBR average marked ~245/44.1/JS: there was no spelling on record, so say if you want a different one.
• Search results: they keep the tail when it fits. A row that the line budget would cut is sent without its tail first, so no letter of the filename is spent on it. Search words still match the whole row, so @find <artist> 320 works.
• Cache location: SQLite at LIST_AUDIO_INFO_CACHE = ./data/audio_info.db, beside the list index. Rows are keyed by folder and name and checked against size and mtime. Each list prunes only its own rows, and only when its rebuild publishes. A cache that can't open is reported, and the list is written with sizes only.

Reader (audio_info.py, stdlib only)

• MP3: every ID3v2 tag is skipped. The first frame counts only if its successor is where its header says. Xing / Info / VBRI give the frame count. Without one of those it's CBR, calculated from the audio bytes and the bitrate, with any ID3v1 tag excluded.
• FLAC: metadata blocks are walked, and picture blocks are seeked over rather than read. STREAMINFO gives rate/channels/samples. The bitrate is the real one: audio bytes divided by the duration.
• read() never raises. Any file it can't read keeps only its size. AutoQ is unaffected: only file rows change, after the size, and the !rar rows are exactly as before.

Tests

tests/test_the_list_says_how_long_and_how_good.py (31): every file is built byte by byte, so the expected numbers are the formats' own arithmetic. It covers:

• MP3: CBR, both ID3 tag types, a tag bigger than the search window, a false sync, all channel modes, Xing, Info, VBRI and MPEG-2.
• FLAC: mono, 6ch, 96 kHz and a 3 MB picture block.
• Broken files: six kinds.
• The cache: reuse, a changed file, pruning, per-list scope, and a cache that can't open.
• Rebuilds and search: a real rebuild with the setting on and off, and @find at the exact name length where the tail decides whether the name gets cut.

I mutation-checked 8 properties: sync confirmation, ID3v1 exclusion, the ID3v2 skip, per-list prune, the size/mtime check, prune on publish, the search fallback, and ignoring a changed file. Each mutation fails a test. Full suite: 6538 OK on this branch.

Docs: both changelogs and INSTALL.md (Build the first list).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AP6LSxkr4n9dMFNSNMogmW

[Ninja-FSE]

Live test result: this is far too slow on a real, non-trivial library — reporting the numbers before this ships.

Tested live by the operator: LIST_SHOW_AUDIO_INFO was turned on via the dashboard, then !update was run against the real 64,136-file / 1.85 TB library (FILE_DIRECTORY is an NFS mount, /mnt/nfs-musik).

Before (same library, same evening, without audio info):

List of 64,136 Files (1.85TB) generated ... in 00:00:45 (1,405 Files Per Second)

With LIST_SHOW_AUDIO_INFO on, first run (cache empty):

12,391 files scanned in 21.1 minutes = 9.8 files/second

That is a ~143x slowdown. Extrapolated to the full 64,136 files: roughly 1.5–2 hours for one rebuild. The operator aborted the run at that point (SIGTERM to update_list.py, ~21 minutes / 12,391 files in) rather than let it run to completion, since PAUSE_ON_UPDATE blocks every search and file request for the entire duration — real users were already hitting MAINTENANCE BLOCK: an !update is running refusals every couple of minutes before the abort.

Likely cause: FILE_DIRECTORY on this install is a network (NFS) mount. audio_info.read() opens every MP3/FLAC individually (seek past any ID3v2 tag, read up to MP3_SYNC_WINDOW = 64 KB looking for a valid frame, then a second seek+read at the file's end for an ID3v1 tag) instead of the single stat() a size-only scan needs. Each of those is a real round trip over NFS rather than a page-cache hit, so the cost that read as "a few KB, never a full read" in the PR description is still one-to-several NFS round trips per file, and 64,136 of those adds up exactly the way the numbers show.

What held up on the good side: the abort was clean. The subprocess was in D state (uninterruptible I/O wait, itself a symptom of slow NFS) when killed; commands.py's finally block still reset update_inprogress/search_inprogress correctly on the non-zero exit, and SIGTERM-then-nothing left the SQLite cache holding the ~12,391 rows it had already written (Cache.publish() only prunes on a successful run, confirmed by reading it) - so nothing was lost or corrupted, and a future run only needs to pay for the remainder. The existing atomic-publish behaviour also meant the previous (good, audio-info-free) list kept serving throughout - nothing was left half-written.

Worth deciding before this ships, given real numbers rather than the CI machines' local disks:

• Is a ~2-hour, fully-blocking first run (on a library this size, on a network-mounted library specifically) acceptable at all, even as a one-time cost?
• Should the first-run population be decoupled from PAUSE_ON_UPDATE's search/share block - e.g. read audio info in the background after the list already publishes size-only, rather than holding up the whole rebuild and every user on it?
• Should slow-mount operators get an explicit warning (at minimum, in the setting's own help text) that first-run cost scales with library size and mount latency, not just file count?

No opinion yet on which of those is right - just the measured numbers, since CI's local disks would never have shown this.

[chchatzop]

Thanks for the live numbers. CI's local disks would never have shown that. Reworked in 19139e3, modelled on how QuickList (OmenServe's list maker, mIRC-scripts/QuickList) does it. On a network mount the time goes on round trips, not bytes, and the first version paid 3–4 of them per file, one file after another.

What changed

• Many files at once. During the walk, files are only noted. The ones to read are read after it, LIST_AUDIO_INFO_THREADS (16) at a time, so the round trips overlap. QuickList does the same with FileReaderParallelism.
• One request per ordinary file. Files are opened unbuffered and read through a window that only fetches what it doesn't already hold. One 16 KB read covers the ID3 header, the first frame and its Xing header, or FLAC's STREAMINFO. Cover art costs one more read. The ID3v1 check at the end of the file is gone: it was a whole round trip for 128 bytes, about 8 ms of a CBR file's duration. Tests pin 1 read per file, or 2 with cover art.
• Nothing at all for an unchanged file. The cache is loaded into memory once and checked against the size the directory listing already gave, which is QuickList's MediaCache rule. No stat, no read, and it's written back once. Before, every rebuild paid a stat per audio file even with nothing to read. Your 12,391-row cache from the aborted run is read as it is.
• A time limit, for your first two questions. LIST_AUDIO_INFO_MINUTES (5, 0 = none) bounds how long one rebuild spends reading. Past it, no new read is started, the list publishes with what was read, and the rest show their size until the next rebuild reads them. So a first run on a big or slow library is spread over a few ordinary rebuilds, and no search waits more than about 5 minutes. The dashboard shows Reading length and quality: n of m files, which also keeps the stall check fed. I chose this over a background reader because it keeps one process and one writer to the cache, and it reuses the rebuild's existing progress, abort and publish paths.
• Help text, your third question: LIST_SHOW_AUDIO_INFO's help now says every file is read once, and that on a large or network library this is spread over a few rebuilds. Both new settings live under List rebuild.

Speed: measured only synthetically. With 5 ms of injected latency per open and read, 400 files: 1 worker 40 files/s, 4 workers 372/s, 16 workers 1,418/s. Extrapolating from your 9.8 files/s, the first pass on your NFS library should be minutes rather than two hours, but that's an estimate. Could you or the operator re-run the same test? The rebuild's last line now reads Audio info: N read, M unchanged, K left for the next one, which gives the real rate directly.

Tests: 40, up from 31. Among them: a barrier that only four concurrent reads can pass, the time limit with an injected clock, a stopped rebuild keeping what it read, and a reader that raises. Every new property was mutation-checked. Full suite 6547 OK. #917 is rebased on top.

[Ninja-FSE]

Re-tested with the rewrite (19139e3) live, as asked — three full !update runs, tracking the cache across them.

Numbers:

Run	Wall time (whole !update, search+sharing paused)	New files read	Running total with duration/quality
1	6m17s	29,971	29,971
2	5m56s	+17,052	47,023
3	5m21s	+15,634	62,657

62,699 audio files total on this library; 42 still show size-only after run 3 (presumably genuinely unreadable/corrupt - next run will attempt them again since they were never added to seen). 17m34s total, across three ordinary rebuilds, to fully populate a 64,136-file / 1.85TB NFS-mounted library - each individual rebuild only paused search/sharing for ~5-6 minutes, not the ~2 hours the single-threaded version measured.

Cache confirmed correct across separate invocations, not just within one run: each run picked up exactly where the last left off (29,971 → 47,023 → 62,657), strictly additive - no file was re-read once it had a cached entry, and nothing was lost between runs (each is a separate update_list.py subprocess, so this is the SQLite persistence actually being exercised, not just the in-run dict).

Two questions, now that this has real numbers behind it:

1. Is there more headroom in LIST_AUDIO_INFO_THREADS? ~55-60 files/s was the sustained real rate at the default of 16 workers on this NFS mount - noticeably below the synthetic 5ms-latency estimate (1,418/s at 16 workers). Worth knowing whether a real library like this one would see a meaningful gain going to 32 or 64, or whether something else (thread pool overhead, the NFS server's own concurrency limit, GIL contention across 16 Python threads each doing blocking I/O) caps it well below the synthetic number regardless of worker count. If there's a ceiling, operators on slow mounts should know roughly where it is rather than guessing by raising the setting.

2. Is LIST_AUDIO_INFO_MINUTES actually the right lever, or is the real fix decoupling this from PAUSE_ON_UPDATE entirely? Worth being precise about what PAUSE_ON_UPDATE already does, since it predates The list can say how long each track is and how good (#567) #914 and isn't specific to audio info: it has always paused both searching and file-sharing for the whole !update, not just an audio-reading phase - list.py's search gate and dcc.py's request gate both key off update_inprogress directly, for the whole subprocess run. So the list is not searchable while it's being rebuilt today, with or without audio info; the time limit is what bounds how long that pre-existing pause lasts when there's a large batch to read, not something added on top of an otherwise-searchable window.

   Given that, the alternative worth naming explicitly: publish the list with sizes-only as soon as the walk finishes (as it does today without LIST_SHOW_AUDIO_INFO), and read audio info afterward, outside PAUSE_ON_UPDATE altogether, in an unbounded background pass - notifying the console/debug feed when it finishes rather than budgeting it into the paused window at all. That would cut every rebuild's user-facing pause back to roughly what it is without this feature (the walk alone: ~80s here), regardless of how much reading is left to do.

   The real cost of that alternative, as far as I can tell from the current design: the list file already served has been published without the newly-read info, so getting it into what's actually being handed out needs either a second, smaller publish once the background pass finishes, or waiting for the next rebuild to pick up what the cache now knows (which is close to today's behavior already, just without a rebuild's own pause bounding it). It also means a background reader outliving the update_list.py subprocess that started it - a different lifecycle than today's "reads happen inside the rebuild that asked for them."

   Not proposing this over what's here - the current design is simple (one process, one writer, reuses the rebuild's own progress/abort/publish paths, as the PR says) and the measured numbers above are already a large improvement. Flagging it because "should this move outside the pause" seems like the actual question behind "is 5 minutes the right number," rather than the number itself.

🤖 Generated with Claude Code

[chchatzop]

Thanks, that's the test I couldn't run. It covers three rebuilds, persistence across separate subprocesses, and a real NFS server.

A correction on the 42: they won't be retried. A file whose read fails still goes into seen: describe(None) is "", and publish() stores that "" with the file's size. So an unreadable file is remembered as unreadable until its size changes, and isn't opened again on every rebuild (note() answers it from the cache by size). If a run left files for later, the rebuild's last line says K left for the next one. With no such count after run 3, those 42 are the unreadable ones and they're done.

1. Headroom in LIST_AUDIO_INFO_THREADS: I can't tell from here, and I'd rather measure than guess. It isn't GIL contention, since a blocking read releases the GIL. At 16 workers and ~58 files/s, each worker waits about 275 ms per file, which is far above one NFS round trip. That looks like the server or mount queueing the requests rather than a per-thread ceiling, and more threads won't get past that. So as of d771029 (with a Windows timing fix in 586b9d1) the rebuild's last line ends Read at N files a second, 16 at a time, measured on the real clock. Trying 32 once and comparing is then a single rebuild, on any install. If the rate stays flat, that's the mount's ceiling, and INSTALL.md now says so. To try it on this mount, delete data/audio_info.db first, or there's nothing left to read.

2. Moving the reading out of PAUSE_ON_UPDATE: you're right that the pause is the whole !update and predates this. I'd keep what's here, for the reason your numbers give. It's a one-time cost, now 17½ minutes over three ordinary rebuilds, and after that a rebuild reads only new files, so the feature adds almost nothing to the pause. The background pass would add a reader that outlives update_list.py, a second publish, and two writers on the cache, and it would only shorten the first few rebuilds. An operator who wants those shorter can set LIST_AUDIO_INFO_MINUTES to 1 or 2. If shortening every rebuild's pause becomes the goal (the ~80 s walk included), that's a change to PAUSE_ON_UPDATE itself, probably worth its own issue.

[Ninja-FSE]

Update on Question 1 above (thread headroom): tested it directly.

Deleted audio_info.db and ran three more full !updates from a genuinely empty cache (same 64,136-file NFS library):

Run	LIST_AUDIO_INFO_THREADS	Wall time	Files read	Rate
Cold-cache run 1	16 (default)	5m54s	25,880	~73/s
Cold-cache run 2	16	2m36s	36,777	~236/s

Between those two I bumped LIST_AUDIO_INFO_THREADS from 16 to 64 and ran again - and the rate roughly tripled to quadrupled (~73/s -> ~236/s), finishing the remaining files (down to the same 42 unreadable ones as before) in well under half the previous time. That is a much cleaner answer to "is there headroom" than I expected: this NFS server clearly tolerates far more than 16 concurrent requests without falling over, and the earlier ~55-73/s figures were this setting's own ceiling, not the mount's.

Caveat, in fairness: run 2 immediately followed run 1 against the same library, so some of the gain could be the NFS client's or server's own cache having just been warmed by run 1's pass over the same directories, not only the thread count - I did not isolate the two (e.g. by re-running at 16 threads a second time to separate "warm cache" from "more threads"). Worth someone repeating with that controlled, but a change this large strongly suggests the thread count matters a lot on real hardware, not just synthetically.

Given that, worth considering whether LIST_AUDIO_INFO_THREADS's default (16) is worth raising, or at least whether the setting's own help text should say more plainly that a slow/network mount specifically benefits from turning it up rather than leaving it at the default.

🤖 Generated with Claude Code

[chchatzop]

That settles it better than I could have. Proposed in 37cbde9, yours to take or change:

• Default 16 → 32. Your 64 result is partly a warm cache, as you say, so I stopped halfway rather than going all the way to 64. 32 is harmless on a local disk.
• Range 1–64 → 1–128, so there's room above 64 on a server that takes it.
• Help text: says plainly that a network drive is faster with more at once, try 64, and that the rebuild reports the rate it got, so you can compare. Same in es/fr, defaults.py and INSTALL.md. Your numbers are recorded in the changelog entry, with the warm-cache caveat.

If you'd rather ship 64 as the default, it's the one number in defaults.py plus the test that pins it. Full suite 6569 OK.