Modernize tooling: uv + Selenium 4

.gitignore

@@ -3,3 +3,16 @@ geckodriver.log
 data
 
 .venv
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+
+# Build artifacts
+dist/
+build/
+
+# Worktrees (per superpowers using-git-worktrees)
+.worktrees/

Makefile

@@ -1,14 +1,14 @@
-REQUIRED_BINS := geckodriver
+REQUIRED_BINS := geckodriver uv
 $(foreach bin,$(REQUIRED_BINS),\
     $(if $(shell command -v $(bin) 2> /dev/null),$(info Found required `$(bin)`),$(error Please install `$(bin)`)))
 
 -include .env
+export
 
+.PHONY: install scrape
 
 install:
-	@pipenv install
+	@uv sync --extra dev
 
-
-.PHONY: scrape
 scrape:
-	@pipenv run scrape
+	@uv run scrape

README.md

@@ -0,0 +1,131 @@
+# scrape
+
+A Python CLI that downloads MP3 tracks from the [Radio Nat Turner](https://www.radionatturner.com) record pool. It logs into each genre page with Selenium, iterates over the embedded audio blocks, and streams every track to disk with a progress bar.
+
+> **Heads up:** this scraper is purpose-built for the Radio Nat Turner site. The selectors (`.password-input`, `.audio-block`, `.title`, `.artistName`, `.secondary-controls .download a`) are specific to that site's markup. It is not a general-purpose scraper.
+
+---
+
+## Features
+
+- Logs into password-protected Radio Nat Turner genre pages via Selenium (Firefox)
+- Downloads every track on 11 genre/category pages out of the box
+- Skips files that already exist locally (resumable runs)
+- Streams downloads in 1 KB chunks with per-file and per-folder progress bars (via [`enlighten`](https://pypi.org/project/enlighten/))
+- Backs off and restarts the browser session on HTTP 429 (rate-limited)
+- Collects non-429 failures and prints a summary at the end
+
+---
+
+## Requirements
+
+| Dependency | Version | Notes |
+|---|---|---|
+| Python | 3.12+ | Declared in `pyproject.toml` |
+| [`uv`](https://docs.astral.sh/uv/) | latest | Used to manage the venv and run the package |
+| Firefox | recent | The scraper drives a real Firefox window |
+| `geckodriver` | on `$PATH` | The `Makefile` aborts if it can't find this binary |
+| A Radio Nat Turner account | — | You need the site password to log in |
+
+Install everything on macOS:
+
+```bash
+brew install uv geckodriver
+brew install --cask firefox
+```
+
+---
+
+## Installation
+
+```bash
+git clone https://github.com/benpetty/scrape.git
+cd scrape
+make install
+# …or, equivalently:
+uv sync --extra dev
+```
+
+`uv sync` creates a `.venv/` from `pyproject.toml` + `uv.lock`, pinning exact versions for reproducible installs.
+
+---
+
+## Configuration
+
+The scraper reads the site password from the `RADIO_NAT_TURNER_PASSWORD` environment variable. Create a `.env` file at the repo root (it's gitignored):
+
+```bash
+# .env
+RADIO_NAT_TURNER_PASSWORD=your-password-here
+```
+
+The `Makefile` automatically loads and exports `.env`, so `make scrape` will see the variable.
+
+---
+
+## Usage
+
+```bash
+make scrape
+# …or:
+uv run scrape
+```
+
+Tracks are saved under `data/<category-slug>/` (gitignored). For example, the `disco` category writes to `data/disco/`.
+
+A run will:
+
+1. Iterate over the 11 hardcoded category URLs in `scrape/radio_nat_turner.py` (`URLS`).
+2. Open Firefox, log into each page, and locate every `.audio-block`.
+3. Download each track's MP3, skipping anything already on disk.
+4. On HTTP 429, close the browser, reopen it, and restart the current category.
+5. Print a `💀` summary of any tracks that failed for non-rate-limit reasons.
+
+Press **Ctrl-C** to abort cleanly — the program exits with status `130`.
+
+---
+
+## Project layout
+
+```
+scrape/
+├── Makefile                       # Wraps install + run; checks for uv + geckodriver
+├── pyproject.toml                 # Project metadata, dependencies, entry point
+├── uv.lock                        # Pinned dependency tree (committed)
+└── scrape/                        # The package
+    ├── __init__.py                # Package metadata
+    ├── __main__.py                # Entry point (handles KeyboardInterrupt → exit 130)
+    ├── radio_nat_turner.py        # Main scraper: URLs, login, downloads, retries
+    └── core/
+        ├── exit_status.py         # IntEnum of exit codes
+        ├── normalize_filename.py  # strip_accents() helper
+        └── progress_bars.py       # enlighten wrapper with named color formats
+```
+
+---
+
+## How it works
+
+1. **`scrape/__main__.py`** is the entry point (`uv run scrape` → `python -m scrape`). It wraps the run in a `try/except KeyboardInterrupt` so Ctrl-C produces exit code 130.
+2. **`RadioNatTurner(url).scrape()`** in `radio_nat_turner.py`:
+   - Spawns a Firefox WebDriver (Selenium 4 auto-resolves `geckodriver` via Selenium Manager if not on `$PATH`).
+   - Visits the URL, types the password into `.password-input`, and presses Return.
+   - Waits for `.audio-block` elements to render.
+   - For each block, pulls the title, artist, and download URL, then either skips (already on disk) or streams the MP3.
+3. **`Writer`** is a context manager that wraps each download: it opens the destination file, attaches an `enlighten` progress counter, and writes the response stream in 1 KB chunks.
+4. **Rate-limit handling** — on `429 Too Many Requests`, the scraper closes the browser, spawns a fresh one, and recursively re-invokes `scrape()` for the same category. There is no backoff or attempt cap, so a persistently rate-limited category can loop indefinitely; interrupt with Ctrl-C if that happens.
+5. **Other failures** are collected in a module-level `FAILURES` list and printed at the end of the run as a `💀` summary block.
+
+---
+
+## Known limitations
+
+- **No retry cap on 429** — see "How it works" above.
+- **Empty password** is silently allowed: `os.environ.get("RADIO_NAT_TURNER_PASSWORD")` returns `None` if the env var is missing, and the login submits nothing.
+- **`scrape/core/normalize_filename.py`** is currently unused. Accented track-title filenames pass through unchanged.
+
+---
+
+## License
+
+MIT — see package metadata in `pyproject.toml`.

pyproject.toml

@@ -0,0 +1,31 @@
+[project]
+name = "scrape"
+version = "0.0.0"
+description = "Downloader for the Radio Nat Turner record pool"
+readme = "README.md"
+requires-python = ">=3.12"
+license = "MIT"
+authors = [
+    { name = "Audeos", email = "benny@audeos.cloud" },
+]
+dependencies = [
+    "colorama>=0.4.4",
+    "enlighten>=1.8.0",
+    "requests>=2.32.0",
+    "selenium>=4.27.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "ipython>=8.10.0",
+]
+
+[project.scripts]
+scrape = "scrape.__main__:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["scrape"]

scrape/radio_nat_turner.py

@@ -1,7 +1,6 @@
 #!/usr/bin/python3
 
 import os
-import random
 import json
 
 from urllib.parse import urlparse
@@ -11,15 +10,13 @@
 
 from selenium import webdriver
 from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support.ui import Select
 from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.common.by import By
 from selenium.webdriver.support import expected_conditions as EC
 
 from colorama import Fore, Back, Style
 
 from scrape.core.progress_bars import ProgressBars
-from scrape.core.normalize_filename import strip_accents
 
 
 URLS = [
@@ -92,14 +89,13 @@ def __init__(self, url: str):
         self.browser = webdriver.Firefox()
         self.url = url
         self.folder_name = f"data{urlparse(self.url).path}"
-        if not os.path.isdir(self.folder_name):
-            os.mkdir(self.folder_name)
+        os.makedirs(self.folder_name, exist_ok=True)
 
     def login(self):
         self.browser.get(self.url)
         self.browser.set_window_size(200, 200)
         print(f"{Style.RESET_ALL}signing in @ {Fore.YELLOW}{self.url}")
-        password_input = self.browser.find_element_by_class_name("password-input")
+        password_input = self.browser.find_element(By.CLASS_NAME, "password-input")
         password_input.send_keys(PASSWORD)
         password_input.send_keys(Keys.RETURN)
 
@@ -113,7 +109,7 @@ def scrape(self):
             EC.presence_of_element_located((By.CLASS_NAME, "audio-block"))
         )
 
-        tracks = self.browser.find_elements_by_class_name("audio-block")
+        tracks = self.browser.find_elements(By.CLASS_NAME, "audio-block")
         _ = f"{Fore.YELLOW}{len(tracks)}{Style.RESET_ALL}"
         print(f"found {_} tracks")
         print(f"saving to {Fore.YELLOW}{self.folder_name}")
@@ -130,12 +126,12 @@ def scrape(self):
                 EC.presence_of_element_located((By.CLASS_NAME, "title"))
             )
 
-            title = track.find_element_by_class_name("title").text
-            artist = track.find_element_by_class_name("artistName").text
+            title = track.find_element(By.CLASS_NAME, "title").text
+            artist = track.find_element(By.CLASS_NAME, "artistName").text
             track_url = (
-                track.find_element_by_class_name("secondary-controls")
-                .find_element_by_class_name("download")
-                .find_element_by_tag_name("a")
+                track.find_element(By.CLASS_NAME, "secondary-controls")
+                .find_element(By.CLASS_NAME, "download")
+                .find_element(By.TAG_NAME, "a")
                 .get_property("href")
                 .split("?")[0]
             )