Describe --max-pages as a queue budget, not a render count

CHANGELOG.md

@@ -18,8 +18,11 @@ All notable changes to kage are recorded here. The format follows
 - `--exclude` and `--scope-prefix` now match complete path prefixes and their
   descendants rather than arbitrary substrings. If you relied on the old
   `--exclude` behaviour, pass the full path prefix.
-- `--max-pages` is documented as attempting at most N page renders; failed
-  renders count toward the cap.
+- `--max-pages` is documented as a cap on how many page URLs are queued rather
+  than how many render. The budget is spent in `enqueuePage`, so a page that
+  fails to render, that `robots.txt` disallows, or that turns out not to be HTML
+  has already taken its slot by the time kage finds out, and a run can save
+  fewer pages than the number asked for.
 
 ### Deprecated
 

README.md

@@ -114,7 +114,7 @@ The flags you'll actually reach for:
 | Flag | Default | Meaning |
 |------|---------|---------|
 | `-o, --out` | `$HOME/data/kage` | Output root; the mirror lands in `<out>/<host>/` |
-| `-p, --max-pages` | `0` | Attempt at most N page renders (0 = no limit); failed renders count toward the cap |
+| `-p, --max-pages` | `0` | Queue at most N page URLs (0 = no limit); pages that fail, are disallowed, or are not HTML still count |
 | `-d, --max-depth` | `0` | How many links deep to follow (0 = no limit) |
 | `--scope-prefix` | | Only crawl this path and its descendants, not similar path names |
 | `--subdomains` | `false` | Treat subdomains of the seed host as in scope |

cli/clone.go

@@ -69,7 +69,7 @@ func newCloneCmd() *cobra.Command {
 	fs.IntVar(&f.workers, "workers", 4, "concurrent page render workers")
 	fs.IntVar(&f.assetWorkers, "asset-workers", 8, "concurrent asset download workers")
 	fs.IntVar(&f.browserPages, "browser-pages", 4, "Chrome page-pool size")
-	fs.IntVarP(&f.maxPages, "max-pages", "p", 0, "attempt at most N page renders (0 = unlimited)")
+	fs.IntVarP(&f.maxPages, "max-pages", "p", 0, "queue at most N page URLs (0 = unlimited)")
 	fs.IntVarP(&f.maxDepth, "max-depth", "d", 0, "link-follow depth cap (0 = unlimited)")
 	fs.StringVar(&f.traversal, "traversal", "bfs", "frontier order (ignored; the crawl is always breadth-first)")
 	_ = fs.MarkDeprecated("traversal", "the crawl is always breadth-first; this flag was never read")

clone/config.go

@@ -28,10 +28,14 @@ type Config struct {
 	OutDir   string // output root; the mirror lands in <OutDir>/<host>/
 	Reserved string // reserved dir name for assets and state (default "_kage")
 
-	Workers       int // page render workers
-	AssetWorkers  int // HTTP asset download workers
-	BrowserPages  int // Chrome page-pool size
-	MaxPages      int // attempt at most N page renders (0 = unlimited)
+	Workers      int // page render workers
+	AssetWorkers int // HTTP asset download workers
+	BrowserPages int // Chrome page-pool size
+	// MaxPages caps how many page URLs are queued, not how many render (0 =
+	// unlimited). The budget is spent in enqueuePage, so a page that fails,
+	// that robots.txt disallows, or that turns out not to be HTML has already
+	// taken its slot by the time the crawl finds out.
+	MaxPages      int
 	MaxDepth      int // BFS/DFS depth cap (0 = unlimited)
 	Traversal     string
 	MaxAssetBytes int64

docs/content/guides/scoping-a-crawl.md

@@ -11,16 +11,22 @@ the crawl.
 ## Limit by count and depth
 
 ```bash
-# Attempt at most 200 page renders
+# Queue at most 200 page URLs
 kage clone example.com --max-pages 200
 
 # Only follow links three hops from the seed
 kage clone example.com --max-depth 3
 ```
 
 `--max-depth 0` (the default) means unlimited depth; `--max-pages 0` means
-unlimited attempts. Failed renders count toward the page cap. Combine the flags
-to put a hard ceiling on a run.
+unlimited pages. Combine the flags to put a hard ceiling on a run.
+
+The `--max-pages` budget is spent when a URL is queued, not when it renders, so
+a run can save fewer pages than the number you asked for. A page that fails to
+render, that `robots.txt` disallows, or that turns out not to be HTML has
+already taken its slot by the time kage finds out. Pages discovered after the
+budget runs out are still written to `state.json`, so raising the cap and
+running again continues where the last run stopped rather than starting over.
 
 ## Limit by path
 

docs/content/reference/cli.md

@@ -35,7 +35,7 @@ images, and fonts, and writes a browsable mirror to `<out>/<host>/`.
 
 | Flag | Default | Meaning |
 |------|---------|---------|
-| `-p, --max-pages` | `0` | Attempt at most N page renders (0 = unlimited); failures count toward the cap |
+| `-p, --max-pages` | `0` | Queue at most N page URLs (0 = unlimited); pages that fail, are disallowed by `robots.txt`, or turn out not to be HTML still count |
 | `-d, --max-depth` | `0` | Link-follow depth cap (0 = unlimited) |
 | `--scope-prefix` | | Only crawl the path prefix and its descendants, not similar path names |
 | `--subdomains` | `false` | Treat subdomains of the seed host as in scope |

docs/content/reference/release-notes.md

@@ -10,8 +10,9 @@ The authoritative, commit-level history lives in [`CHANGELOG.md`](https://github
 
 - **Crawl path controls use path boundaries.** `--exclude` and
   `--scope-prefix` match a path and its descendants without catching unrelated
-  names that merely contain the same text. `--max-pages` now accurately says
-  that failed render attempts count toward its cap.
+  names that merely contain the same text. `--max-pages` is now described
+  accurately as a cap on queued page URLs rather than on renders, so a page that
+  fails, that `robots.txt` disallows, or that is not HTML still spends its slot.
 - **Multi-page ZIM packs get a usable landing page.** Mirrors without a root
   `index.html` list pages by title instead of opening an arbitrary first page;
   single-page archives still open directly on their article ([#62](https://github.com/tamnd/kage/issues/62)).