Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ It sits in a channel, advertises a catalogue, and serves files over DCC to whoev

- **Serves files over DCC** — per-user and global queues, configurable slots, `!rar` album packing on demand, and DCC RESUME so an interrupted download continues instead of starting over
- **Keeps a searchable master list** in `.txt`, `.zip` and `.rar`, rebuilt atomically so a failed scan never overwrites a good one
- **Fetches from other bots** — request a file or a whole list, or broadcast one `@find` and collect every reply
- **Fetches from other bots** — request a file or a whole list, or broadcast one `@find` and collect every reply. In this bot's own search, words in quotes must appear together: `@find "metal church"`
- **Remembers people** — a user who disconnects keeps their queue for five minutes and resumes on rejoin
- **Defends itself** — rolling flood windows, escalating mutes, hostmask bans, and every other bot treated as untrusted
- **Is operable** — an authenticated DCC CHAT console, an optional web dashboard, live `!rehash`, and statistics that survive a restart
Expand Down
8 changes: 8 additions & 0 deletions commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,14 @@ def handle_help_request(s, user, target):
f"{bold}{red}@{nick}-remove{reset} to cancel it. "
f"To search every bot at once, type: {bold}{red}@find <words>{reset}")

# #774: a band or title made of common words - "Metal Church" matched
# every file with both words anywhere. Said as this bot's own rule: the
# same @find reaches every bot, and one that does not know quotes simply
# answers nothing to a quoted term.
lines.append(
f"Words in quotes must appear together, in that order, in my list: "
f'{bold}{red}@find "metal church"{reset}')

for line in lines:
# #426: NOT is_vip=True. The VIP lane is strict-priority with no
# aging - a channel advert waiting behind it is the reason it
Expand Down
1 change: 1 addition & 0 deletions docs/UPDATES-PUBLIC.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Unreleased

- **Added: put words in quotes to search for them together.** `@find Metal Church` finds every file with both words anywhere in its name - thousands, in a big library. `@find "Metal Church"` finds only files where those words appear together, in that order (`Metal Church`, `Metal_Church`, `metal.church`), and words outside the quotes still narrow it: `@find "Metal Church" 1986`. Searches without quotes work exactly as before. The same works in the dashboard's Search tab. Other bots in the channel may not understand quotes and answer nothing to a quoted search.
- **Fixed: re-running setup with more than one services host configured (home and phone, say) and leaving the services-host question blank silently kept only the first one.** A blank answer now leaves your services hosts exactly as they are; typing a new one still replaces the list, and now says so first when it is replacing more than one. Typing one you already have changes nothing (it used to drop all the others), and the hosts you have configured are shown to you.
- **Added (mIRC console): the @DCCore window's button now lights up like a channel's.** It turns the message colour (red, by default) when there is new activity - a request, a send, a search - and the highlight colour on a failed transfer, so failures stand out. The five-minute status line and joins, parts and bans do not light it, as they would not in a channel. Needs mIRC 7 or later; update `dccore.mrc` and reload it.
- **Fixed: pasting a batch of requests could mute you and then ban you for an hour, and the mute notice wrongly said your queue had been cleared.** Asking for a file counted toward the flood protection exactly like a search, so pasting about a dozen rows from the list - one album, the ordinary way these lists are used - could get the eleventh line muted and the twelfth banned for an hour, on any client or connection that sends a paste quickly. Asking for files is no longer rate-limited at all: the limit is your queue, as it always should have been (100 files each by default), and past it you are told once rather than once per line. Searches and the other commands are unchanged. The mute notice was also wrong in a way that caused the ban - what it drops is the bot's pending replies to you, never your queued files - so it now says that other commands are ignored for the next 30 seconds and that your queued files are safe.
Expand Down
21 changes: 21 additions & 0 deletions docs/UPDATES.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,27 @@ All version changes, optimizations, and bug fixes made over time in the DCCore p

## 🟨 Unreleased

### 🔎 A "quoted phrase" in a search means those words together, in order (#774)

Asked by the operator after `@find Metal Church` answered 6516 results. A search matched a row when every
word appeared somewhere on that file's line, in any order, as a substring - so a band named with two common
words matched every file holding both (`Metallica ... Church`, `Church Of Heavy Metal`, `Churchill`). A quoted
term was no way out: the quotes were searched for literally and matched nothing, which made the syntax free.

`list.split_search_term()`: a part in double quotes is a **phrase** - its words must appear together, in that
order, separated only by space, `_`, `-`, `.` or `*` (`Metal Church`, `Metal_Church`, `metal.church`) - and
comes back as a tuple inside the same word list, so every caller that passes the list through gets it
unchanged; `find_matching_entries()` compiles each tuple once per list. Words outside the quotes keep the old
rule, so `@find "Metal Church" 1986` narrows further. A term with no quotes splits exactly as before (tested
against a copy of the old rule); a one-word "phrase" is a word; an unpaired quote is dropped rather than
searched for. `webserver.split_list_search_words()` now calls the same function instead of repeating its
code, so `@find`, the dashboard's Search tab and a list's own search cannot drift apart. The List Browser's
filter bar already treats typed words as a phrase (`*` between phrases, #399's beta request) and is untouched.
A phrase is matched against the file's line, like the words - not the folder it sits in. The `-help` reply and
the README say so, and that it is this bot's rule: the same `@find` reaches every bot, and one that does not
know quotes answers nothing to a quoted term. `tests/test_a_quoted_phrase_is_matched_together.py` (14): the
split, the match against a real list file, and `@find` itself.

### 🔒 The library-scan locks live in runtime.py, and the lock guard sees the `or` form (#749)

Follow-up to #731 (#580). `dcc.py` built its scan semaphore and its lookup-memory lock at module level as
Expand Down
62 changes: 58 additions & 4 deletions list.py
Original file line number Diff line number Diff line change
Expand Up @@ -575,6 +575,53 @@ def _split_entry_line(line_strip):
return strip_info_suffix(rest)


# What may sit between the words of a quoted phrase (#774): the same four
# characters an unquoted term is split on, plus the space - so "Metal Church"
# finds Metal Church, Metal_Church, Metal-Church and metal.church, and not
# Metallica ... Church.
_PHRASE_GAP = r"[ _.*\-]+"
_QUOTED = re.compile(r'"([^"]*)"')


def split_search_term(term):
"""A search term as the list find_matching_entries() takes (#774).

Unquoted, exactly the rule @find has always used: `-`, `*`, `_` and `.`
become spaces, the rest is split into lower-cased words, and a row
matches when every word appears on it somewhere, in any order. That is
right for "vivaldi winter" and wrong for a band whose name is two common
words - `@find Metal Church` matched 6516 rows, every file with both
"metal" and "church" in it anywhere.

A part in double quotes is a PHRASE: its words must appear together, in
that order, with only separators between them. It comes back as a tuple
inside the same list, so every caller that passes the list through gets
phrases without changing, and a term with no quotes returns exactly the
list it always did. A one-word "phrase" is just a word, and a stray,
unpaired quote is dropped rather than searched for - a literal quote was
never in a filename anyone searched for, and today it made the whole
search come back empty.

Shared by execute_search() (@find) and webserver.split_list_search_words()
(the dashboard's Search tab and a list's own search), so the two cannot
drift apart - their docstrings have always said they use one rule.
"""
text = str(term or "")
phrases = []

def _take(match):
words = [word for word in re.split(r"[-*_.\s]+", match.group(1).lower()) if word]
if len(words) > 1:
phrases.append(tuple(words))
elif words:
phrases.append(words[0])
return " "

rest = _QUOTED.sub(_take, text).replace('"', " ")
clean_term = re.sub(r'[-*_.]', ' ', rest)
return [w.strip().lower() for w in clean_term.split() if w.strip()] + phrases


def find_matching_entries(search_words, limit=None, list_path=None, name=None):
"""IRC-agnostic core of the master-list search, extracted from execute_search().

Expand Down Expand Up @@ -633,6 +680,12 @@ def find_matching_entries(search_words, limit=None, list_path=None, name=None):
entries = []
total_matches = 0

# A tuple in the list is a quoted phrase (#774), compiled once per list
# rather than once per line; a string is a word, matched as it always was.
plain_words = [item for item in search_words if not isinstance(item, tuple)]
phrase_patterns = [re.compile(_PHRASE_GAP.join(re.escape(word) for word in item))
for item in search_words if isinstance(item, tuple)]

current_list_path = list_path
if not current_list_path or not os.path.exists(current_list_path):
return entries, total_matches
Expand Down Expand Up @@ -689,7 +742,9 @@ def find_matching_entries(search_words, limit=None, list_path=None, name=None):
continue

line_lower = line_strip.lower()
if search_words and not all(word in line_lower for word in search_words):
if plain_words and not all(word in line_lower for word in plain_words):
continue
if phrase_patterns and not all(pattern.search(line_lower) for pattern in phrase_patterns):
continue

total_matches += 1
Expand Down Expand Up @@ -1231,9 +1286,8 @@ def execute_search(irc_sock, user, search_term, channel):
# Strip mIRC colour codes and control characters from the search terms
raw_clean = strip_control_codes(search_term)

# Split the search terms
clean_term = re.sub(r'[-*_.]', ' ', raw_clean)
search_words = [w.strip().lower() for w in clean_term.split() if w.strip()]
# Split the search terms - words, and "quoted phrases" (#774)
search_words = split_search_term(raw_clean)

# ---------------------------------------------------------------------
# Straight copy: no reformatting, the file row is sent raw
Expand Down
147 changes: 147 additions & 0 deletions tests/test_a_quoted_phrase_is_matched_together.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
"""A "quoted phrase" in a search means those words together, in order (#774).

Asked by the operator after `@find Metal Church` answered 6516 results:
every word of a search had to appear somewhere on a file's line, in any
order, so a band named with two common words matched every file holding
both - Metallica, Heavy Metal, Churchill. A quoted term was no help: the
quotes were searched for literally and matched nothing.

Now a part in double quotes is a phrase - its words adjacent, in order,
with only separators between them - and words outside the quotes keep the
old rule. A term with no quotes is split exactly as before; that is the
first thing checked, against a copy of the old rule.
"""

import os
import re
import sys
import unittest

REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if REPO_ROOT not in sys.path:
sys.path.insert(0, REPO_ROOT)

import list as list_mod # noqa: E402
import webserver # noqa: E402

from tests.support import DCCoreTestCase, RecordingSocket # noqa: E402
from tests.test_webserver import write_master_list # noqa: E402


def the_old_rule(term):
"""execute_search()'s and split_list_search_words()'s split before #774."""
clean_term = re.sub(r'[-*_.]', ' ', str(term or ""))
return [w.strip().lower() for w in clean_term.split() if w.strip()]


ROWS = [
("Metal Church - The Dark.mp3", "5.1MB"), # phrase, spaced
("Metal_Church-Watch The Children Pray.flac", "31MB"), # underscore
("metal.church.live.1986.mp3", "7MB"), # dots, and 1986
("Metallica - Church Bells.mp3", "4MB"), # both words, not adjacent
("Church Of Heavy Metal.mp3", "3MB"), # both words, wrong order
("Vivaldi - The Four Seasons - Winter.flac", "40MB"),
]


class TheSplit(unittest.TestCase):

def test_a_term_without_quotes_splits_exactly_as_it_always_did(self):
for term in ("Metal Church", "vivaldi winter", "AC-DC_Back.in*Black", "Björk Jóga",
" ", "", "---", "a.b-c_d*e", "Metal Church 1986"):
with self.subTest(term=term):
self.assertEqual(list_mod.split_search_term(term), the_old_rule(term))

def test_a_quoted_part_is_one_phrase(self):
self.assertEqual(list_mod.split_search_term('"Metal Church"'), [("metal", "church")])

def test_words_outside_the_quotes_keep_the_old_rule(self):
self.assertEqual(list_mod.split_search_term('"Metal Church" 1986'),
["1986", ("metal", "church")])

def test_separators_inside_the_quotes_split_the_phrase_the_same_way(self):
self.assertEqual(list_mod.split_search_term('"Metal_Church-Live"'),
[("metal", "church", "live")])

def test_a_one_word_phrase_is_just_a_word(self):
self.assertEqual(list_mod.split_search_term('"metal"'), ["metal"])

def test_an_unpaired_quote_is_dropped_not_searched_for(self):
"""It used to be part of the word, so `@find "metal` matched nothing."""
self.assertEqual(list_mod.split_search_term('"metal'), ["metal"])

def test_the_dashboard_splits_with_the_same_function(self):
for term in ('"Metal Church" 1986', "plain words", '"a b"'):
with self.subTest(term=term):
self.assertEqual(webserver.split_list_search_words(term),
list_mod.split_search_term(term))


class TheMatch(DCCoreTestCase):

def setUp(self):
super().setUp()
self.tree = self.make_tree()
os.makedirs(self.tree.lists, exist_ok=True)
self.path = write_master_list(self.tree.lists, "SomeBot", [(None, ROWS)])

def matched(self, term):
entries, total = list_mod.find_matching_entries(
list_mod.split_search_term(term), list_path=self.path)
self.assertEqual(total, len(entries))
return sorted(entry["filename"] for entry in entries)

def test_unquoted_it_matches_both_words_anywhere_as_before(self):
self.assertEqual(self.matched("Metal Church"), sorted([
"Church Of Heavy Metal.mp3", "Metal Church - The Dark.mp3",
"Metal_Church-Watch The Children Pray.flac", "Metallica - Church Bells.mp3",
"metal.church.live.1986.mp3"]))

def test_quoted_it_matches_only_the_words_together_in_order(self):
self.assertEqual(self.matched('"Metal Church"'), sorted([
"Metal Church - The Dark.mp3", "Metal_Church-Watch The Children Pray.flac",
"metal.church.live.1986.mp3"]))

def test_a_word_outside_the_phrase_narrows_it_further(self):
self.assertEqual(self.matched('"Metal Church" 1986'), ["metal.church.live.1986.mp3"])

def test_two_phrases_must_both_be_there(self):
self.assertEqual(self.matched('"four seasons" "the four"'),
["Vivaldi - The Four Seasons - Winter.flac"])
self.assertEqual(self.matched('"four seasons" "metal church"'), [])

def test_an_empty_search_still_matches_everything(self):
self.assertEqual(len(self.matched("")), len(ROWS))


class ThroughFind(DCCoreTestCase):
"""@find itself, against the same list, as a user would type it."""

def setUp(self):
super().setUp()
self.tree = self.make_tree()
os.makedirs(self.tree.lists, exist_ok=True)
write_master_list(self.tree.lists, "DCCoreTest", [(None, ROWS)])
self.set_config(FILE_DIRECTORY=self.tree.music, LOCAL_LIST_DIR=self.tree.lists,
LIST_BASE_NAME="DCCoreTest", NICKNAME="DCCoreTest", CHANNEL="#chan",
search_inprogress=False, update_inprogress=False)

def find(self, term):
self.oserve.queued.clear()
list_mod.execute_search(RecordingSocket(), "dave", term, "#chan")
return "".join(m for _u, m, *_ in self.oserve.queued)

def test_the_quoted_search_answers_only_the_band(self):
reply = self.find('"Metal Church"')

self.assertIn("Found: ", reply)
self.assertIn("3 Match(es)", reply)
self.assertNotIn("Metallica", reply)
self.assertNotIn("Church Of Heavy Metal", reply)

def test_the_unquoted_search_answers_as_it_always_did(self):
self.assertIn("5 Match(es)", self.find("Metal Church"))


if __name__ == "__main__":
unittest.main()
9 changes: 4 additions & 5 deletions webserver.py
Original file line number Diff line number Diff line change
Expand Up @@ -696,11 +696,10 @@ def split_list_search_words(query):
results, not the whole list) - so each caller decides what to do with an
empty return rather than this function guessing for all of them.
"""
import re

raw_clean = str(query or "")
clean_term = re.sub(r'[-*_.]', ' ', raw_clean)
return [w.strip().lower() for w in clean_term.split() if w.strip()]
# One rule, not a copy of it (#774): a "quoted phrase" means the same
# here as in @find, and the two cannot drift apart.
import list as list_mod
return list_mod.split_search_term(query)


def build_search_payload(query):
Expand Down
Loading