Fetching from other bots, the way AutoGet did it: understand their replies, keep the queue working by itself

[chchatzop]

The operator had me read AutoGet 6.1 (^OmeN^) and the Autoget 7.40b4 rebuild (eMpTy, 2005) in full, as the download manager that went with OmeNServE. Hardly anyone runs AutoGet now (AutoQ replaced it; AutoGet breaks on current mIRC), so this is not about being compatible with AutoGet users. It's about taking its practices into DCCore's own fetch side (dcc_fetch.py, list_fetch.py). I checked each item against the code first.

Phase 1: a fetch queue that keeps working by itself (the gap)

1. Understand other servers' replies. Today only "!rar is disabled" is recognised (dcc_fetch.handle_refusal_notice). Everything else waits out FETCH_OFFER_TIMEOUT and ends as "no response". AutoGet 7 carries the exact phrases for OmeNServE (1.31–2.x), SDFind, SpR Jukebox (English and French) and BWI, plus DCCore's own. Each maps to one of four outcomes:
   • accepted, queue position N: shown on the row, and no timeout while queued.
   • bad filename / file deleted / "you need sharing": fail at once, no retry, say why.
   • busy / queue full / "maxed out": back off, retry later.
   • already in your queue: keep waiting.
2. The queue survives a restart (only finished and failed rows are saved today), and a bot that's offline can be queued for (refused today).
3. Re-request when the bot is back (JOIN or an advert), after a random delay, plus a periodic retry sweep. The next file from a bot is requested when one finishes, capped by a per-bot limit on files queued there.
4. Pause a bot after 3 connection failures, with a per-bot on/off switch. Stop fetching when the disk is full (a failed write fails only that row today).

Phase 2: lists

5. Opt-in automatic list grabbing for bots that advertise and have no list held, on AutoGet's rules:

   • a random 5–360 s delay per request, at most one automatic grab every N minutes;
   • cancel our own request if someone else in the channel just asked for that list;
   • 3 tries, then stop, with 30-minute cool-downs;
   • skip bots below a minimum file count or speed, and bots in "servers only" mode.

   Its author's own counter says 37% of list requests arrived (2,175 of 5,835), so the retry limits matter.

6. List expiry (refresh when older than N days) and a "seen" mark per list.

7. "Online only" filter in the cross-list search, since presence is already known.

8. Slots, queue and speed per bot in the List Browser. The advert already carries them; irc._record_bot() drops them.

Each item is its own PR. I'm starting with 1, since it's what lets 2–4 decide between retrying and giving up. Identity across nick changes stays on #376.

[chchatzop]

Taken, starting with item 1.

[chchatzop]

Every item in the plan now has a PR, and all of them pass CI:

• 1–2 (understand replies, take a queued file): A bot's answer to our request is understood, and a queued file is taken when it comes (#926) #927 (merged)
• 3 (queue waits for a bot, per-bot pacing, survives restart): The fetch queue waits for a bot, paces itself per bot, and survives a restart (#926) #928
• 4 (pause an unreachable bot, wait on a full disk): A bot we cannot reach is paused, and a full disk makes fetching wait (#926) #929, stacked on The fetch queue waits for a bot, paces itself per bot, and survives a restart (#926) #928
• 7 (search online bots only): The List Browser's search can ask only the bots that are online (#926) #931
• 8 (slots/queue/speed in the List Browser): Each bot's advertised slots, queue and speed show in the List Browser (#926) #932, stacked on The List Browser's search can ask only the bots that are online (#926) #931
• 6 (undated lists refreshed at 14 days, "New" badge): An undated list is refreshed when old, and a list not opened yet says "New" (#926) #933, stacked on Each bot's advertised slots, queue and speed show in the List Browser (#926) #932
• 5 (automatic list grabbing on AutoGet's rules, off by default): Lists are grabbed automatically, on AutoGet's rules (#926) #934, stacked on An undated list is refreshed when old, and a list not opened yet says "New" (#926) #933

Merge each stack from the bottom up. When the PR below one merges, retarget that PR to main. I've checked each stack merged together with the other open PRs (#925, #930): the full suite passes.

[Ninja-FSE]

All 8 items merged into main. #934 (item 5, the last one) just went in.

• 1–2. Understand other servers' replies, take a queued file when it comes — A bot's answer to our request is understood, and a queued file is taken when it comes (#926) #927 ✅
• 3. Fetch queue waits for a bot, paces itself per bot, survives a restart — The fetch queue waits for a bot, paces itself per bot, and survives a restart (#926) #928 ✅
• 4. Pause an unreachable bot, wait on a full disk — A bot we cannot reach is paused, and a full disk makes fetching wait (#926) #929 ✅
• 5. Automatic list grabbing on AutoGet's rules (off by default) — Lists are grabbed automatically, on AutoGet's rules (#926) #934 ✅
• 6. Undated lists refreshed at 14 days, "New" badge — An undated list is refreshed when old, and a list not opened yet says "New" (#926) #933 ✅
• 7. Search only bots that are online — The List Browser's search can ask only the bots that are online (#926) #931 ✅
• 8. Slots, queue and speed shown in the List Browser — Each bot's advertised slots, queue and speed show in the List Browser (#926) #932 ✅

Each stack merged bottom-up as you said, retargeted to main at every step, with a full-suite run in an isolated clone before every push and green CI (9/9) before every merge.

One thing worth a look separately: #934's own CI run failed once on ubuntu-latest / Python 3.12 only — tests.test_dashboard_routes.TheReadOnlyRoutesAnswer.test_each_one_returns_the_payload_its_builder_produces, /api/stats route - then passed clean on an immediate rerun, and the other 8 platform/version combinations were green on the first try. Looks like the same kind of leaked cross-test state this project has hit before (#920/#921), not a real routing defect. Didn't block the merge since it's non-deterministic and the rerun was clean, but flagging it in case it's worth its own issue.

Closing this tracking issue now that all 8 items are in.