An undated list is refreshed when old, and a list not opened yet says "New" (#926)

defaults.py

@@ -723,7 +723,9 @@
 # one to inherit.
 #
 # The ADVERT decides, not a timer - #286 already worked out what "moved on"
-# means. A timer alone would re-ask every bot for a list we already have.
+# means. A timer alone would re-ask every bot for a list we already have. The
+# one exception is a bot whose advert gives no date to compare (#926): its
+# list is re-asked for once it is 14 days old, or it would never be refreshed.
 AUTO_REFETCH_LISTS: bool = False
 # How stale a held list may get before it is re-asked for, in hours. Not how
 # often the check runs (that is hourly); this is the floor on how often any one

docs/UPDATES-PUBLIC.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- **Added: a "New" mark on fetched lists in the List Browser.** A list you fetched and have not opened yet says *New*; opening it clears the mark, and a refreshed list is new again.
+- **Improved: with `AUTO_REFETCH_LISTS` on, lists from bots that show no date are refreshed too.** Such a bot never says its list changed, so its list was never refreshed; it is now fetched again once it is 14 days old. Bots that do show a date are still refreshed only when it changes.
 - **Added: the List Browser shows each online bot's free slots, queue and speed**, as that bot advertises them - so you can see which one is quick and which one is full before you ask.
 - **Added: "Online only" in the List Browser's search.** Tick it to search only the lists of bots that are in the channel right now - the ones you can actually download from today.
 - **Added: a "Responsible use" note in the README.** DCCore is for sharing files you have the right to share; you decide what your bot offers and are responsible for it; the contributors do not host, control or endorse anything any bot shares.

docs/UPDATES.md

@@ -4,6 +4,30 @@ All version changes, optimizations, and bug fixes made over time in the DCCore p
 
 ## 🟨 Unreleased
 
+### 🆕 Undated lists are refreshed when old, and a list not opened yet says "New" (#926)
+
+Item 6 of #926, AutoGet's list expiry. `list_fetch.lists_worth_refetching()` acted only on freshness "changed" -
+a bot's advert showing a different file count or date than when its list was fetched. A bot that publishes no date,
+or one whose advert has not been seen since starting, is "unknown" and was never refreshed at all, however old its
+list. Age is the only evidence such a bot leaves, so an "unknown" list older than
+`list_fetch.UNKNOWN_LIST_MAX_AGE_DAYS` (14, as AutoGet's default expiry) is now due as well; the interval, the
+per-run cap and the "we already asked" mark still apply. A dated bot is unchanged: an "unchanged" list is never
+refreshed on age. The sweep's console line says which it was - *"X's list is over 14 days old"* rather than *"has
+changed"*, which such a list has not been shown to have (`list_fetch._freshness_of()`). `test_list_freshness`'s
+*unknown is not changed* now holds a list three days old, the case it was about.
+
+Every fetch now stores `seen_at: 0` in the held entry, and opening the list in the List Browser
+(`/api/filelists/bot/<nick>`, on a 200) calls the new `list_fetch.mark_seen()`, which stamps it and saves. The
+sidebar row carries `unseen` - fetched and not opened since - and the page shows a small *New* badge. An entry
+from before this has no `seen_at` and is never marked: nothing says it is new. The setting's help, the
+`defaults.py` comment and `settings.conf.sample` say the 14-day exception.
+
+`tests/test_an_old_list_is_refreshed_and_a_new_one_is_marked.py` (10): an undated list just over and just under the
+limit, an unchanged dated list left alone however old, the sweep saying why, the setting off, a new list unseen, opening it clearing the
+mark and saving (and a second open doing nothing), a legacy entry never marked, and both the fetch and the route
+code. Mutation-checked: no expiry, expiring dated lists too, calling an old list changed, never stamping seen, and
+marking legacy entries each fail a test.
+
 ### 📊 Each bot's advertised slots, queue and speed show in the List Browser (#926)
 
 Item 8 of #926, AutoGet's "slots" page. An OmenServe-style advert says *Slots: 3/10 <> Queued: 12 <> Speed:

list_fetch.py

@@ -734,6 +734,13 @@ def _hours_to_seconds(hours):
         return 0.0
 
 
+# A list whose bot gives no evidence of change - no date in its advert, or no
+# advert seen - is refreshed once it is this old (#926 item 6), the way
+# AutoGet expired lists after N days. Age is the only evidence such a bot
+# leaves; a bot that does publish a date is still refreshed on that alone.
+UNKNOWN_LIST_MAX_AGE_DAYS = 14
+
+
 def lists_worth_refetching(now=None):
     """The bots whose held list their own advert says has moved on.
 
@@ -751,6 +758,9 @@ def lists_worth_refetching(now=None):
     "unknown" is not "changed". A bot that publishes no date, or one whose
     advert we have not seen since starting, gives no evidence either way, and
     acting on no evidence is what makes an automatic feature untrustworthy.
+    EXCEPT AGE (#926): such a list is refreshed once it is older than
+    UNKNOWN_LIST_MAX_AGE_DAYS - otherwise it is never refreshed at all, which
+    is the one thing sure to be wrong about it.
     """
     import webserver
 
@@ -787,14 +797,36 @@ def lists_worth_refetching(now=None):
 
         rows = [row for row in webserver.build_fetched_bot_list_summaries()
                 if str(row.get("bot", "")).strip().lower() == bot.lower()]
-        if not rows or rows[0].get("freshness") != "changed":
+        if not rows:
+            continue
+        freshness = rows[0].get("freshness")
+        too_old = (freshness == "unknown"
+                   and now - float(fetched_at or 0) >= UNKNOWN_LIST_MAX_AGE_DAYS * 86400)
+        if freshness != "changed" and not too_old:
             continue
         due.append((float(fetched_at or 0), bot))
 
     due.sort()
     return [bot for _when, bot in due]
 
 
+def mark_seen(bot):
+    """The operator opened this bot's list (#926 item 6): it is no longer
+    new. Returns whether anything changed."""
+    key = str(bot or "").strip().lower()
+    with _lock():
+        store = _ensure_fetched_bot_lists()
+        entry = store.get(key)
+        if not isinstance(entry, dict) or "seen_at" not in entry:
+            return False
+        if float(entry.get("seen_at") or 0) >= float(entry.get("fetched_at") or 0):
+            return False
+        entry["seen_at"] = time.time()
+        snapshot = dict(store)
+    db.save_fetched_bot_lists(snapshot)
+    return True
+
+
 def _tell_the_console(bot, action, text):
     """One `LISTFETCH` line for a console or the mIRC window (#750). Never raises:
     a console that cannot be told must not fail a fetch."""
@@ -825,6 +857,16 @@ def _note_auto_attempt(bot, when):
     db.save_fetched_bot_lists(snapshot)
 
 
+def _freshness_of(bot):
+    """The List Browser's freshness for `bot`'s held list, or None."""
+    import webserver
+
+    for row in webserver.build_fetched_bot_list_summaries():
+        if str(row.get("bot", "")).strip().lower() == str(bot).strip().lower():
+            return row.get("freshness")
+    return None
+
+
 def refetch_due_lists(log=print, now=None):
     """Ask again for the held lists their own adverts say have changed.
 
@@ -880,9 +922,16 @@ def refetch_due_lists(log=print, now=None):
         if status == 200:
             started.append(bot)
             _note_auto_attempt(bot, time.time() if now is None else now)
-            log(f"[LIST-FETCH] {bot}'s list has changed since we took our copy "
-                f"- asking again automatically.")
-            _tell_the_console(bot, "auto", f"{bot}'s list has changed - asking again automatically")
+            if _freshness_of(bot) != "changed":
+                # #926: no date to compare, so age was the reason.
+                why = f"{bot}'s list is over {UNKNOWN_LIST_MAX_AGE_DAYS} days old"
+                log(f"[LIST-FETCH] {why} and its advert shows no date "
+                    f"- asking again automatically.")
+            else:
+                why = f"{bot}'s list has changed"
+                log(f"[LIST-FETCH] {why} since we took our copy "
+                    f"- asking again automatically.")
+            _tell_the_console(bot, "auto", f"{why} - asking again automatically")
         else:
             # Not an error worth stopping for: the usual reason is that a
             # fetch for that bot is already outstanding, which is the right
@@ -1310,6 +1359,11 @@ def _install_fetched_list(bot, zip_path, extract_dir):
         # absence is the honest answer rather than a zero - see
         # _advert_snapshot().
         "advert_when_fetched": _advert_snapshot(bot),
+        # Not looked at yet (#926 item 6): the List Browser marks it "New"
+        # until the operator opens it - mark_seen(). Set on EVERY fetch, so a
+        # refreshed list is new again. An entry from before this has no
+        # seen_at at all and is not marked: nothing says it is new.
+        "seen_at": 0,
         # EVERY LIST THE ARCHIVE HELD, keyed by a short stable marker. The
         # main one keeps the empty marker and is also mirrored in list_path
         # and entry_count above - which is what every reader written before an

settings.conf.sample

@@ -1059,8 +1059,9 @@
 #MAX_FETCH_SLOTS = 3
 
 # When another bot advertises that its list has changed, fetch the new list
-# automatically. Off by default because it uses the other bot's bandwidth
-# without you asking each time.
+# automatically. A list from a bot whose advert shows no date is fetched again
+# once it is 14 days old. Off by default because it uses the other bot's
+# bandwidth without you asking each time.
 #
 # How long a finished (complete/failed) cross-bot fetch stays in the Downloads
 # table. Age is the primary rule because that table is a recent record of what
@@ -1073,7 +1074,9 @@
 # one to inherit.
 # 
 # The ADVERT decides, not a timer - #286 already worked out what "moved on"
-# means. A timer alone would re-ask every bot for a list we already have.
+# means. A timer alone would re-ask every bot for a list we already have. The
+# one exception is a bot whose advert gives no date to compare (#926): its
+# list is re-asked for once it is 14 days old, or it would never be refreshed.
 #AUTO_REFETCH_LISTS = false
 
 # The least time between two automatic asks for the same bot's list, in hours.

settings_help.py

@@ -208,7 +208,7 @@ def help_text(name):
     'LIST_HEADER_FILE': 'A text file whose contents are printed at the top of your list - a greeting, your channel name, some ASCII art. If the file does not exist, nothing is added.',
     'LIST_HEADER_MAX_BYTES': 'The most of that file that will be used, in bytes, so a wrong file cannot bloat every list.',
     'MAX_FETCH_SLOTS': 'How many downloads FROM other bots you run at the same time. Separate from your own send slots, so your downloading never takes slots away from people downloading from you.',
-    'AUTO_REFETCH_LISTS': "When another bot advertises that its list has changed, fetch the new list automatically. Off by default because it uses the other bot's bandwidth without you asking each time.",
+    'AUTO_REFETCH_LISTS': "When another bot advertises that its list has changed, fetch the new list automatically. A list from a bot whose advert shows no date is fetched again once it is 14 days old. Off by default because it uses the other bot's bandwidth without you asking each time.",
     'AUTO_REFETCH_INTERVAL_HOURS': "The least time between two automatic asks for the same bot's list, in hours. Counted from the last list that arrived or the last time the bot was asked, so a bot that rebuilds hourly - or does not answer - is not asked every hour.",
     'AUTO_REFETCH_MAX_PER_RUN': 'The most lists to re-fetch in one go. If many are out of date at once, the rest are picked up on later rounds, oldest first.',
     'FETCH_MAX_PER_BOT': 'How many files to ask one bot for at once. The next one is asked when one arrives. Servers allow each person only a few; asking for more gets "queue full". 0 means no limit.',

tests/test_an_old_list_is_refreshed_and_a_new_one_is_marked.py

@@ -0,0 +1,132 @@
+"""#926 item 6: a list with no date is refreshed once it is old, and a list
+nobody has opened yet says "New".
+
+- The automatic refresh acts on a bot's advert saying its list changed. A bot
+  that publishes no date never says so, and its list was never refreshed. Age
+  is the only evidence such a bot leaves: past UNKNOWN_LIST_MAX_AGE_DAYS it is
+  refreshed (AutoGet expired lists after N days). A dated bot is unchanged.
+- Every fetch leaves the list unseen; opening it in the List Browser marks it
+  seen. Entries from before this carry no seen_at and are never marked.
+"""
+
+import os
+import sys
+import unittest
+
+REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+if REPO_ROOT not in sys.path:
+    sys.path.insert(0, REPO_ROOT)
+
+import db  # noqa: E402
+import list_fetch  # noqa: E402
+import runtime  # noqa: E402
+import webserver  # noqa: E402
+
+from tests.support import DCCoreTestCase  # noqa: E402
+
+DAY = 86400.0
+NOW = 100 * DAY
+
+
+class ListCase(DCCoreTestCase):
+    def setUp(self):
+        super().setUp()
+        original = dict(runtime.known_bots)
+        runtime.known_bots.clear()
+        self.addCleanup(lambda: (runtime.known_bots.clear(), runtime.known_bots.update(original)))
+        self.set_config(AUTO_REFETCH_LISTS=True, AUTO_REFETCH_INTERVAL_HOURS=24, bot_joined_channel=True)
+        self.saved = []
+        real_save = db.save_fetched_bot_lists
+        db.save_fetched_bot_lists = lambda registry: self.saved.append(registry)
+        self.addCleanup(setattr, db, "save_fetched_bot_lists", real_save)
+        self.store = {}
+        self.set_config(fetched_bot_lists=self.store)
+
+    def hold(self, bot, fetched_at, advert_then=None, advert_now=None, **extra):
+        self.store[bot.lower()] = dict({"bot": bot, "fetched_at": fetched_at, "entry_count": 10,
+                                        "advert_when_fetched": advert_then or {}}, **extra)
+        if advert_now is not None:
+            runtime.known_bots[bot.lower()] = dict(advert_now, nick=bot)
+
+
+class AnUndatedListIsRefreshedWhenOld(ListCase):
+    def test_older_than_the_limit(self):
+        self.hold("QuietBot", NOW - (list_fetch.UNKNOWN_LIST_MAX_AGE_DAYS + 1) * DAY)
+        self.assertEqual(list_fetch.lists_worth_refetching(now=NOW), ["QuietBot"])
+
+    def test_younger_than_the_limit(self):
+        self.hold("QuietBot", NOW - (list_fetch.UNKNOWN_LIST_MAX_AGE_DAYS - 1) * DAY)
+        self.assertEqual(list_fetch.lists_worth_refetching(now=NOW), [])
+
+    def test_a_dated_bot_that_has_not_changed_is_left_alone_however_old(self):
+        same = {"files": 100, "list_date": "Aug 1st"}
+        self.hold("DatedBot", NOW - 60 * DAY, advert_then=same, advert_now=same)
+        self.assertEqual(list_fetch.lists_worth_refetching(now=NOW), [])
+
+    def test_the_sweep_says_why(self):
+        """An undated list did not "change" - the console line says it is old."""
+        asked = []
+        real = webserver.build_list_fetch_enqueue_result
+        webserver.build_list_fetch_enqueue_result = lambda bot: (asked.append(bot), (200, {}))[1]
+        self.addCleanup(setattr, webserver, "build_list_fetch_enqueue_result", real)
+        told = []
+        real_tell = list_fetch._tell_the_console
+        list_fetch._tell_the_console = lambda bot, action, text: told.append(text)
+        self.addCleanup(setattr, list_fetch, "_tell_the_console", real_tell)
+        self.hold("QuietBot", NOW - 60 * DAY)
+        self.hold("DatedBot", NOW - 60 * DAY, advert_then={"files": 1, "list_date": "Aug 1st"},
+                  advert_now={"files": 2, "list_date": "Sep 1st"})
+        lines = []
+        list_fetch.refetch_due_lists(log=lines.append, now=NOW)
+        self.assertEqual(sorted(asked), ["DatedBot", "QuietBot"])
+        self.assertIn("QuietBot's list is over 14 days old - asking again automatically", told)
+        self.assertIn("DatedBot's list has changed - asking again automatically", told)
+        self.assertTrue(any("QuietBot" in line and "shows no date" in line for line in lines))
+
+    def test_off_is_off(self):
+        self.set_config(AUTO_REFETCH_LISTS=False)
+        self.hold("QuietBot", NOW - 60 * DAY)
+        self.assertEqual(list_fetch.lists_worth_refetching(now=NOW), [])
+
+
+class ANewListSaysSo(ListCase):
+    def row(self, bot):
+        return [r for r in webserver.build_fetched_bot_list_summaries()
+                if r.get("bot") == bot][0]
+
+    def test_fetched_and_not_opened_is_unseen(self):
+        self.hold("NewBot", NOW, seen_at=0)
+        self.assertTrue(self.row("NewBot")["unseen"])
+
+    def test_opening_it_marks_it_seen_and_saves(self):
+        self.hold("NewBot", 1000.0, seen_at=0)
+        self.assertTrue(list_fetch.mark_seen("NewBot"))
+        self.assertFalse(self.row("NewBot")["unseen"])
+        self.assertTrue(self.saved)
+        self.assertFalse(list_fetch.mark_seen("NewBot"), "already seen: nothing to do")
+
+    def test_an_entry_from_before_this_is_not_marked(self):
+        self.hold("OldBot", NOW)
+        self.assertFalse(self.row("OldBot")["unseen"])
+        self.assertFalse(list_fetch.mark_seen("OldBot"))
+
+    def test_a_refreshed_list_is_new_again(self):
+        """Every fetch writes seen_at = 0 - read from the code that stores it."""
+        import io
+        with io.open(os.path.join(REPO_ROOT, "list_fetch.py"), encoding="utf-8") as handle:
+            code = handle.read()
+        at = code.index('"advert_when_fetched": _advert_snapshot(bot),')
+        self.assertIn('"seen_at": 0,', code[at:at + 500])
+
+    def test_the_route_marks_it_when_the_list_is_served(self):
+        import io
+        with io.open(os.path.join(REPO_ROOT, "webserver.py"), encoding="utf-8") as handle:
+            code = handle.read()
+        at = code.index("def api_filelists_bot(nick):")
+        body = code[at:at + 900]
+        self.assertIn("if status == 200:", body)
+        self.assertIn("list_fetch.mark_seen(nick)", body)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_list_freshness.py

@@ -362,8 +362,11 @@ def test_a_list_that_has_not_changed_is_left_alone(self):
     def test_unknown_is_not_changed(self):
         """A bot that publishes no date gives no evidence either way, and
         acting on no evidence is what makes an automatic feature
-        untrustworthy."""
-        self.hold("ReelBot", {})
+        untrustworthy. Until the list is old (#926): past
+        UNKNOWN_LIST_MAX_AGE_DAYS age is the evidence, covered in
+        test_an_old_list_is_refreshed_and_a_new_one_is_marked."""
+        fetched = 10 ** 9 - 3 * 86400
+        self.hold("ReelBot", {}, fetched_at=fetched)
         self.advertise("ReelBot", {"files": 250})
 
         self.assertEqual(list_fetch.lists_worth_refetching(now=10 ** 9), [])

web/app.js

@@ -1756,6 +1756,15 @@
       button.appendChild(hand);
     }
 
+    // #926: fetched and not opened yet.
+    if (primary.unseen) {
+      var fresh = document.createElement("span");
+      fresh.className = "bot-row-new";
+      fresh.textContent = t("filelists.newBadge");
+      fresh.title = t("filelists.newBadgeTitle");
+      button.appendChild(fresh);
+    }
+
     if (grouped) {
       var badge = document.createElement("span");
       badge.className = "bot-row-lists-badge";

web/lang/en.json

@@ -109,6 +109,8 @@
   "filelists.liveSlotsFree": "{free}/{total} free",
   "filelists.liveSlotsBusy": "{busy}/{total} busy",
   "filelists.liveQueued": "{count} queued",
+  "filelists.newBadge": "New",
+  "filelists.newBadgeTitle": "Fetched and not opened yet",
   "filelists.legendCannotTell": "cannot tell",
   "filelists.legendHereNow": "here now",
   "filelists.legendListCurrent": "list current",

web/lang/es.json

@@ -109,6 +109,8 @@
   "filelists.liveSlotsFree": "{free}/{total} libres",
   "filelists.liveSlotsBusy": "{busy}/{total} ocupados",
   "filelists.liveQueued": "{count} en cola",
+  "filelists.newBadge": "Nueva",
+  "filelists.newBadgeTitle": "Descargada y aún sin abrir",
   "filelists.legendCannotTell": "no se puede saber",
   "filelists.legendHereNow": "presente ahora",
   "filelists.legendListCurrent": "lista actualizada",