Skip to content

Lists are grabbed automatically, on AutoGet's rules (#926) - #934

Merged
Ninja-FSE merged 2 commits into
mainfrom
feat/926-grab-lists-automatically
Sep 24, 2026
Merged

Ninja-FSE merged 2 commits into
mainfrom
feat/926-grab-lists-automatically

Conversation

@chchatzop

Copy link
Copy Markdown
Collaborator

Item 5 of #926, which is the last item. Stacked on #933 because they touch the same changelog heads. Retarget it to main once the stack below it merges.

New list_grab.py. When AUTO_GRAB_LISTS is on (it is off by default), DCCore asks for the list of any bot that advertises one and whose list we don't hold yet. It follows AutoGet's rules:

  • One at a time: at most one grab every AUTO_GRAB_EVERY_MINUTES (10), biggest list first.
  • Random wait: 5 to 360 seconds before asking, so clients that saw the same advert don't all ask at once.
  • Someone else asked: if another user types @Bot in a channel, a grab waiting on that bot is dropped and the bot is left alone for 10 minutes, because it's busy sending that user's list. This doesn't count as a try.
  • 3 tries, 30 minutes apart, then it stops. AutoGet's own counter shows only 37% of list requests ever arrived.
  • Skipped bots: bots that are offline, our own nick, bots below AUTO_GRAB_MIN_FILES, bots advertising less than AUTO_GRAB_MIN_SPEED_KB (a bot that shows no speed isn't skipped), bots in "servers only" mode, and bots whose list the operator removed by hand. purge_fetched_list() records that. The bulk purge of offline bots doesn't, because that is tidying up, not a decision about the bot.
  • Checked again after the wait: a bot that left in the meantime isn't asked.

Each ask goes through build_list_fetch_enqueue_result(), the same path as the List Browser button, so all the slot limits and duplicate guards still apply. Each grab also writes a console line.

State: the current wait, the time of the last grab and the start guard live in runtime.py, so a rehash doesn't forget a wait or start a second worker. The per-bot tries and the removed-by-hand list are saved in data/list_grabs.json, so a restart doesn't reset the three tries. list_grab_others_asked is bound in defaults.py and listed in PRESERVE_RUNTIME. The worker starts from boot and from rehash, the same way the list refresh does.

Settings: the four settings have their own Grabbing lists category, placed just above Fetching from bots. It sits there so it doesn't collide with #928's Fetch queue, which is added just below that category. They have help text and en/es/fr labels, and settings.conf.sample is regenerated.

Tests: tests/test_lists_are_grabbed_automatically_on_autogets_rules.py has 26 tests. I mutation-checked them: 13 mutations, one per rule, and each one fails a test. The full suite passes on this branch, and also with #925, #929 and #930 merged in.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AP6LSxkr4n9dMFNSNMogmW

Item 5 of #926. With AUTO_GRAB_LISTS on (off by default), the list of a
bot that advertises one and whose list is not held is asked for: one at a
time, at most one every AUTO_GRAB_EVERY_MINUTES, after a random 5-360
second wait, dropped if someone else asks that bot meanwhile, and at most
3 tries 30 minutes apart. Small, slow and servers-only bots are skipped,
and a list removed by hand is not grabbed back. Tries survive a restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AP6LSxkr4n9dMFNSNMogmW
@Ninja-FSE
Ninja-FSE merged commit 1214bfe into main Sep 24, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants