diff --git a/README.md b/README.md index 97ec1c1..dd874d7 100644 --- a/README.md +++ b/README.md @@ -730,10 +730,15 @@ docker run -d --name brewshot-worker --restart unless-stopped \ ``` Watch mode accepts complete, visible, regular, direct-child `.html` and `.htm` -files only. They should be self-contained: claiming moves the file into a -job-specific processing directory, so sibling asset paths are not a worker -contract. URL manifests and arbitrary option manifests are intentionally not -accepted; use the unchanged CLI for URLs and advanced capture options. +files for the existing PNG path. They should be self-contained: claiming moves +an individual file into a job-specific processing directory. + +For GIFs or configurable stills, drop a versioned `*.brewshot.json` website job, +or a `*.brewshot.zip` containing root `job.json`, local HTML and relative assets. +A website job contains a URL and typed capture settings; a local ZIP is claimed +as one atomic package. No per-capture Docker command is needed. See the +[watch job schema, examples and limits](docs/watch-jobs.md). Plain HTML drops +continue producing PNGs with the existing defaults. Publish a complete job with a hidden sibling plus atomic rename so the worker never observes a partial upload: diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 7eb7e56..1bfd0d4 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -2,6 +2,16 @@ ## Unreleased +- **Watched GIF jobs from website URLs or local ZIP bundles** (plan a15a0d3b). + Drop a versioned `*.brewshot.json` URL job or a `*.brewshot.zip` containing + `job.json`, HTML and relative assets. Typed options delegate to the existing + Main CLI; PNG/GIF selection, element crop, scale and preparation scripts are + supported. Existing HTML drops retain their PNG behavior. JSON/ZIP jobs run + with a whole-job deadline, bounded ZIP extraction and the existing atomic + no-replacement publication/recovery rules. Local bundles use `file://` with + its local fetch/XHR limitations. See `docs/watch-jobs.md`. The real worker + smoke now also requires Python 3 (stdlib only). + - **The pinned Chrome CI job reaps Chrome's helpers like the runtime image does** (plan a7074a87). The chrome-test build stage ran the real-Chrome suite with no init, so helpers orphaned during a test stayed as zombies, and the two real cleanup tests (CleanupReportingTest, which pin the diff --git a/SECURITY.md b/SECURITY.md index 82810ed..798b113 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -121,6 +121,28 @@ harness for TRUSTED pages driven by a trusted operator** — calibrate to that. host-scoped credentials prefer `cookie()`, which the browser applies under its own same-domain rules. +## Watched job trust and networking + +Anyone who can write a `*.brewshot.json` or `*.brewshot.zip` into the watched +input folder can run trusted page JavaScript and make the container access its +reachable network, including host services. Website sources must explicitly +use HTTP(S); `file:`, `data:` and browser-internal URL schemes are refused as +manifest URL sources. This is a source-scheme check, not an SSRF allowlist: +HTTP redirects and page subresources retain ordinary browser network reach. +Do not expose the drop folder to untrusted producers. Do not put credentials +in URL references. The worker suppresses raw child output and logs bounded +job codes rather than page text or URL queries. + +ZIP extraction has finite compressed/expanded-byte and entry-count limits and +rejects traversal, links/special entries and duplicates. Extracted local HTML +is opened with `file://`; local fetch/XHR and ES modules have Chrome's file-origin +limitations. These checks protect the job packaging contract; they do not +sandbox page code or prevent it from fetching network resources. Preparation +scripts run in the page through the existing eval path, with no shell argument +interpolation. The job subprocess has an outer deadline (120 seconds by +default, plus bounded shutdown); an expired job is failed, never described as +successful or as proof that all browser resources are gone. + ## Reporting This is a small, single-maintainer utility provided under Apache-2.0 on an diff --git a/build.gradle.kts b/build.gradle.kts index 7145280..d62af6d 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -39,6 +39,10 @@ application { mainClass = "com.brewshot.Main" } +// The container worker is a separate runtime jar, but its orchestration tests +// must compile and execute in the browser-free lane too. +sourceSets.test { java.srcDir("docker") } + tasks.jar { manifest { attributes("Main-Class" to "com.brewshot.Main") diff --git a/docker/BrewShotFolderWorker.java b/docker/BrewShotFolderWorker.java index a1bd4e3..0d0f986 100644 --- a/docker/BrewShotFolderWorker.java +++ b/docker/BrewShotFolderWorker.java @@ -25,14 +25,14 @@ import java.util.Locale; import java.util.Objects; import java.util.UUID; +import java.util.Map; +import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicLong; /** - * Container-only folder worker for self-contained local HTML files. Every - * capture calls the shipped {@link Main} entry point with fixed defaults, so - * this class owns mounted-file orchestration rather than a second browser or - * argument parser. URL and advanced-option jobs remain available through the - * ordinary one-shot CLI. + * Container-only folder worker for local HTML stills and versioned URL/ZIP jobs. + * Manifest jobs map to the shipped {@link Main} entry point in a bounded subprocess; + * this class owns mounted-file orchestration rather than a second recorder or CLI grammar. * *

Claims are atomic renames to the original filename below a UUID-named * directory in {@code input/processing}. Keeping the UUID and source name in @@ -55,6 +55,8 @@ public final class BrewShotFolderWorker { private static final int CLAIM_ID_LENGTH = 32; private static final int MAX_DERIVED_NAME_BYTES = 255; private static final int DIAGNOSTIC_LIMIT = 512; + private static final long MAX_JOB_BYTES = 16L * 1024 * 1024; + private static final long JOB_TIMEOUT_MILLIS = jobTimeoutMillis(); private static final byte[] PNG_SIGNATURE = { (byte) 0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a }; @@ -332,50 +334,39 @@ private void process(Claim claim, FileChannel source, String attemptId) + "-" + attemptId + ".png"); Files.deleteIfExists(renderTemp); + BrewShotWatchCommit commit = null; try { - render(claim, source, renderTemp); - publishComplete(renderTemp, output.resolve( - derivedName(claim, ".png"))); - } catch (JobFailure failure) { - Files.deleteIfExists(renderTemp); - if (completedByAnotherWorker(claim, source)) { - logAlreadyCompleted(claim); - return; - } - if (fail(claim, source, attemptId, failure.code())) { - System.err.println("brewshot-watch: job " + claim.jobId() - + " failed (" + failure.code() + ")"); + if (isManifestName(claim.originalName())) { + commit = new BrewShotWatchCommit(output, claim.jobId(), + sourceHash(source)); + Path chosen = commit.recover(); + String suffix; + if (chosen == null) { + suffix = render(claim, source, renderTemp); + chosen = commit.record(renderTemp, suffix); + } else { + suffix = BrewShotWatchCommit.suffix(chosen); + } + if (suffix.equals(".gif")) { requireGif(chosen); } + else { requirePng(chosen); } + try { BrewShotWatchCommit.publish(chosen, output.resolve(derivedName(claim, suffix))); } + catch (BrewShotWatchCommit.Collision collision) { throw new JobFailure("output-collision"); } + Files.deleteIfExists(renderTemp); } else { - logAlreadyCompleted(claim); + String suffix = render(claim, source, renderTemp); + publishComplete(renderTemp, output.resolve(derivedName(claim, suffix))); } + } catch (JobFailure failure) { + failedAttempt(claim, source, attemptId, renderTemp, commit, failure.code()); return; } catch (IOException failure) { - Files.deleteIfExists(renderTemp); - if (completedByAnotherWorker(claim, source)) { - logAlreadyCompleted(claim); - return; - } - String code = "io-" + failure.getClass().getSimpleName(); - if (fail(claim, source, attemptId, code)) { - System.err.println("brewshot-watch: job " + claim.jobId() - + " failed (" + code + ")"); - } else { - logAlreadyCompleted(claim); - } + failedAttempt(claim, source, attemptId, renderTemp, commit, + failure instanceof BrewShotWatchCommit.Failed ? failure.getMessage() + : "io-" + failure.getClass().getSimpleName()); return; } catch (RuntimeException failure) { - Files.deleteIfExists(renderTemp); - if (completedByAnotherWorker(claim, source)) { - logAlreadyCompleted(claim); - return; - } - String code = "runtime-" + failure.getClass().getSimpleName(); - if (fail(claim, source, attemptId, code)) { - System.err.println("brewshot-watch: job " + claim.jobId() - + " failed (" + code + ")"); - } else { - logAlreadyCompleted(claim); - } + failedAttempt(claim, source, attemptId, renderTemp, commit, + "runtime-" + failure.getClass().getSimpleName()); return; } @@ -386,6 +377,37 @@ private void process(Claim claim, FileChannel source, String attemptId) } } + private void failedAttempt(Claim claim, FileChannel source, String attemptId, + Path renderTemp, BrewShotWatchCommit commit, String code) throws IOException { + Files.deleteIfExists(renderTemp); + if (completedByAnotherWorker(claim, source)) { + logAlreadyCompleted(claim); + return; + } + if (commit != null) { + try { + // A late failing attempt must honor a selected success; a failure + // selected first must prevent a late renderer publishing success. + Path chosen = commit.fail(code); + String suffix = BrewShotWatchCommit.suffix(chosen); + if (suffix.equals(".gif")) { requireGif(chosen); } else { requirePng(chosen); } + BrewShotWatchCommit.publish(chosen, output.resolve(derivedName(claim, suffix))); + archive(claim, source, finished); + logAlreadyCompleted(claim); + return; + } catch (BrewShotWatchCommit.Failed selectedFailure) { + code = selectedFailure.getMessage(); + } catch (BrewShotWatchCommit.Collision unrelatedOutput) { + code = "output-collision"; + } catch (JobFailure invalidImage) { + code = invalidImage.code(); + } + } + if (fail(claim, source, attemptId, code)) { + System.err.println("brewshot-watch: job " + claim.jobId() + " failed (" + code + ")"); + } else { logAlreadyCompleted(claim); } + } + private static void logAlreadyCompleted(Claim claim) { System.err.println("brewshot-watch: job " + claim.jobId() + " already completed by another worker"); @@ -397,8 +419,11 @@ private boolean completedByAnotherWorker(Claim claim, FileChannel source) && archivedAnywhere(claim, source); } - private void render(Claim claim, FileChannel source, Path renderTemp) + private String render(Claim claim, FileChannel source, Path renderTemp) throws IOException, JobFailure { + if (isManifestName(claim.originalName())) { + return renderManifest(claim, source, renderTemp); + } long size = source.size(); if (size == 0) { throw new JobFailure("empty-input"); @@ -431,6 +456,190 @@ private void render(Claim claim, FileChannel source, Path renderTemp) throw new JobFailure("capture-exit-" + exit); } requirePng(renderTemp); + return ".png"; + } + + private static String sourceHash(FileChannel source) throws IOException { + if (source.size() == 0 || source.size() > MAX_JOB_BYTES) { + throw new IOException("job input size"); + } + try { + var digest = java.security.MessageDigest.getInstance("SHA-256"); + ByteBuffer buffer = ByteBuffer.allocate(8192); + long position = 0; + while (position < source.size()) { + buffer.clear(); + int read = source.read(buffer, position); + if (read < 0) { throw new IOException("source truncated"); } + buffer.flip(); + digest.update(buffer); + position += read; + } + return java.util.HexFormat.of().formatHex(digest.digest()); + } catch (java.security.NoSuchAlgorithmException impossible) { + throw new IOException("SHA-256 unavailable", impossible); + } + } + + private static boolean isManifestName(String name) { + String lower = name.toLowerCase(Locale.ROOT); + return lower.endsWith(".brewshot.json") || lower.endsWith(".brewshot.zip"); + } + + private String renderManifest(Claim claim, FileChannel source, Path renderTemp) + throws IOException, JobFailure { + long started = System.nanoTime(); + if (source.size() == 0 || source.size() > MAX_JOB_BYTES) { + throw new JobFailure("job-input-size"); + } + Path scratch = Files.createTempDirectory("brewshot-watch-job-"); + Path captureImagePng = renderTemp.resolveSibling(renderTemp.getFileName() + ".png"); + Path captureImageGif = renderTemp.resolveSibling(renderTemp.getFileName() + ".gif"); + try { + // Copy from the OPEN CLAIM, not a producer-controlled pathname. The + // claimed input remains intact for archival and restart recovery. + Path snapshot = scratch.resolve(claim.originalName()); + try (var destination = FileChannel.open(snapshot, StandardOpenOption.CREATE_NEW, + StandardOpenOption.WRITE)) { + ByteBuffer buffer = ByteBuffer.allocate(8192); + long position = 0; + while (position < source.size()) { + if (TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started) + >= JOB_TIMEOUT_MILLIS) { throw new JobFailure("job-deadline"); } + buffer.clear(); + int read = source.read(buffer, position); + if (read < 0) { throw new JobFailure("job-input-truncated"); } + buffer.flip(); + while (buffer.hasRemaining()) { destination.write(buffer); } + position += read; + if (position > MAX_JOB_BYTES) { throw new JobFailure("job-input-size"); } + } + } + Path bundleScratch = Files.createDirectory(scratch.resolve("bundle")); + Path resultPath = scratch.resolve("result.json"); + String java = Path.of(System.getProperty("java.home"), "bin", "java").toString(); + long remaining = JOB_TIMEOUT_MILLIS + - TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started); + if (remaining <= 0) { throw new JobFailure("job-deadline"); } + int exit = runJobProcess(List.of(java, "-Djava.awt.headless=true", "-cp", + System.getProperty("java.class.path"), BrewShotWatchCapture.class.getName(), + snapshot.toString(), renderTemp.toString(), resultPath.toString(), + bundleScratch.toString()), remaining); + Map result = readJobResult(resultPath); + if (exit == Main.EXIT_CLEANUP_UNCONFIRMED) { + CleanupOutcome cleanup = cleanupFromJobResult(result); + noteSurvivingChrome(claim.jobId(), cleanup); + if (cleanup != null) { System.err.println("brewshot-watch: " + cleanup.warning()); } + } else if (exit != 0) { + Object code = result.get("code"); + throw new JobFailure(code instanceof String text + && text.matches("[A-Za-z0-9._-]{1,80}") ? text : "job-exit-" + exit); + } + String format = String.valueOf(result.get("format")); + if (format.equals("png")) { requirePng(renderTemp); return ".png"; } + if (format.equals("gif")) { requireGif(renderTemp); return ".gif"; } + throw new JobFailure("job-result-format"); + } finally { + Files.deleteIfExists(captureImagePng); + Files.deleteIfExists(captureImageGif); + // Only this attempt's freshly created private scratch is removed. + // The claimed original is never beneath this directory. + try (var paths = Files.walk(scratch)) { + for (Path path : paths.sorted(Comparator.reverseOrder()).toList()) { + Files.deleteIfExists(path); + } + } + } + } + + static int runJobProcess(List command, long timeoutMillis) + throws IOException, JobFailure { + Process process = new ProcessBuilder(command) + .redirectOutput(ProcessBuilder.Redirect.DISCARD) + .redirectError(ProcessBuilder.Redirect.DISCARD).start(); + try { + if (process.waitFor(timeoutMillis, TimeUnit.MILLISECONDS)) { + return process.exitValue(); + } + throw new JobFailure("job-deadline"); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new JobFailure("job-interrupted"); + } finally { + if (process.isAlive()) { + // Give Main's existing bounded shutdown hook a chance first. This + // is scoped to the child created above, never a shared browser. + List descendants; + try { + descendants = process.descendants().toList(); + } catch (RuntimeException unavailable) { + // Some restricted hosts deny process enumeration. Still stop + // the exact child we launched; never let a diagnostic failure + // prevent the primary shutdown signal or imply tree release. + descendants = List.of(); + System.err.println("brewshot-watch: child tree unavailable; " + + "release remains unconfirmed"); + } + process.destroy(); + try { + if (!process.waitFor(10, TimeUnit.SECONDS)) { + descendants.forEach(ProcessHandle::destroyForcibly); + process.destroyForcibly(); + process.waitFor(5, TimeUnit.SECONDS); + } + } catch (InterruptedException interrupted) { + descendants.forEach(ProcessHandle::destroyForcibly); + process.destroyForcibly(); + Thread.currentThread().interrupt(); + } + if (process.isAlive()) { + // Do not accept more jobs beside a child we could not stop. + Thread.currentThread().interrupt(); + } + System.err.println("brewshot-watch: bounded job interrupted; child shutdown " + + "requested; Chrome release is not proven by timeout"); + } + } + } + + private static Map readJobResult(Path path) throws IOException, JobFailure { + if (!Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS) || Files.size(path) > 65536) { + throw new JobFailure("job-result-missing"); + } + Object parsed = MiniJson.parseStrict(BoundedUtf8.read(path, 65536, "job result")); + if (!(parsed instanceof Map map)) { throw new JobFailure("job-result-invalid"); } + return map; + } + + private static CleanupOutcome cleanupFromJobResult(Map result) { + if (!(result.get("cleanup") instanceof Map cleanup)) { return null; } + if (!(cleanup.get("livePids") instanceof List pids) + || !(cleanup.get("liveWithoutPid") instanceof Number unnamed)) { return null; } + var live = pids.stream().map(value -> ((Number) value).longValue()).toList(); + return new CleanupOutcome(CleanupOutcome.State.PROCESS_UNCONFIRMED, live, + unnamed.intValue(), List.of(), null, "job capture cleanup"); + } + + static void requireGif(Path rendered) throws IOException, JobFailure { + if (!Files.isRegularFile(rendered, LinkOption.NOFOLLOW_LINKS) || Files.size(rendered) < 14) { + throw new JobFailure("empty-gif-output"); + } + byte[] header; + try (InputStream in = Files.newInputStream(rendered)) { header = in.readNBytes(6); } + String version = new String(header, StandardCharsets.US_ASCII); + if (!version.equals("GIF87a") && !version.equals("GIF89a")) { + throw new JobFailure("invalid-gif-output"); + } + } + + private static long jobTimeoutMillis() { + String value = System.getenv("BREWSHOT_WATCH_JOB_TIMEOUT_MS"); + if (value == null || value.isBlank()) { return 120_000; } + try { + long parsed = Long.parseLong(value); + if (parsed >= 1_000 && parsed <= 600_000) { return parsed; } + } catch (NumberFormatException ignored) { } + throw new IllegalArgumentException("BREWSHOT_WATCH_JOB_TIMEOUT_MS must be 1000..600000"); } /** @@ -788,7 +997,7 @@ private static boolean eligible(Path path) { return false; } String lower = name.toLowerCase(Locale.ROOT); - return lower.endsWith(".html") || lower.endsWith(".htm"); + return eligibleName(lower); } private static Claim parseClaim(Path path) throws IOException { @@ -848,7 +1057,8 @@ private static boolean validJobId(String value) { private static boolean eligibleName(String name) { String lower = name.toLowerCase(Locale.ROOT); return !name.startsWith(".") - && (lower.endsWith(".html") || lower.endsWith(".htm")); + && (lower.endsWith(".html") || lower.endsWith(".htm") + || isManifestName(lower)); } private static void ensureDirectory(Path dir) throws IOException { @@ -921,7 +1131,7 @@ public void close() throws IOException { } } - private static final class JobFailure extends Exception { + static final class JobFailure extends Exception { private final String code; JobFailure(String code) { diff --git a/docker/smoke-test.sh b/docker/smoke-test.sh index 83dee24..1e551b6 100755 --- a/docker/smoke-test.sh +++ b/docker/smoke-test.sh @@ -647,4 +647,7 @@ step "no complete-looking hidden artifacts survive a terminal path" # of a race against it. wait_for_no_worker_temps "$output" "$race_output" "$unreadable_output" +step "versioned URL and ZIP watch jobs" +python3 docker/watch-job-smoke.py "$image" + echo 'brewshot Docker smoke: PASS' diff --git a/docker/watch-job-smoke.py b/docker/watch-job-smoke.py new file mode 100644 index 0000000..7110998 --- /dev/null +++ b/docker/watch-job-smoke.py @@ -0,0 +1,293 @@ +#!/usr/bin/env python3 +"""Real watched job integration smoke. Python stdlib only; pass the built image.""" +import hashlib +import http.server +import io +import json +import os +from pathlib import Path +import shutil +import struct +import subprocess +import sys +import tempfile +import threading +import time +import uuid +import zipfile + + +def docker(*args, check=True): + return subprocess.run(["docker", *map(str, args)], check=check, + stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout + + +def gif_info(data): + """Independent structural reader; distinct encoded images, not a pixel verdict.""" + assert data[:6] in (b"GIF87a", b"GIF89a") + width, height, packed = struct.unpack_from(" 1 else "brewshot:local" + root = Path(tempfile.mkdtemp(prefix="brewshot-job-smoke-")) + root.chmod(0o755) + input_dir, output_dir, site = (root / name for name in ("in", "out", "site")) + for path in (input_dir, output_dir, site): + path.mkdir(mode=0o777) + path.chmod(0o777) + identity = uuid.uuid4().hex[:12] + names = [f"brewshot-jobs-{identity}-{index}" for index in range(2)] + step = "setup" + started = [] + handler = lambda *args, **kwargs: http.server.SimpleHTTPRequestHandler( + *args, directory=str(site), **kwargs) + server = http.server.ThreadingHTTPServer(("0.0.0.0", 0), handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + + def start(name): + docker("run", "-d", "--name", name, + "--add-host", "host.docker.internal:host-gateway", + "-e", "BREWSHOT_WATCH_POLL_MS=20", + "-e", "BREWSHOT_WATCH_JOB_TIMEOUT_MS=15000", + "-v", f"{input_dir}:/brewshot/input", + "-v", f"{output_dir}:/brewshot/output", image, "watch") + started.append(name) + + def wait(relative, timeout=60): + limit = time.monotonic() + timeout + while time.monotonic() < limit: + if (root / relative).exists(): + return + time.sleep(0.05) + raise AssertionError(f"missing {relative}") + + def read_output(name): + return docker("exec", names[0], "cat", f"/brewshot/output/{name}") + + def submit(name, content): + hidden = input_dir / ("." + name + ".upload") + hidden.write_bytes(content) + hidden.chmod(0o644) + hidden.rename(input_dir / name) + + def manifest(source, fmt="gif", **capture): + rules = {"format": fmt, "viewport": {"width": 320, "height": 240}, + "dpr": 1, "settleMs": 50, "selector": ".stage"} + if fmt == "gif": + rules.update(frames=4, delayMs=100) + rules.update(capture) + return {"version": 1, "source": source, "capture": rules} + + html = b''' +

+ ''' + css = b"body{margin:0}.stage{width:160px;height:100px;background:#123456}" + js = b'''let n=0; setInterval(()=>{ + document.querySelector('.stage').style.backgroundColor= + `rgb(${(++n*47)%255},${(n*83)%255},${(n*131)%255})`; + document.querySelector('#tick').textContent=n; + },40);''' + svg = b'' + for name, data in (("index.html", html), ("style.css", css), + ("animation.js", js), ("button.svg", svg)): + (site / name).write_bytes(data) + (site / name).chmod(0o644) + + def bundle(job, extra=None): + data = io.BytesIO() + with zipfile.ZipFile(data, "w", zipfile.ZIP_DEFLATED) as archive: + archive.writestr("job.json", json.dumps(job)) + for name in ("index.html", "style.css", "animation.js", "button.svg"): + archive.writestr(name, (site / name).read_bytes()) + archive.writestr("prepare.js", "document.querySelector('.stage').style.width='128px';") + if extra: + archive.writestr(*extra) + return data.getvalue() + + try: + step = "URL GIF and ignored non-job JSON" + start(names[0]) + (input_dir / "notes.json").write_text('{"ignored":true}') + url = f"http://host.docker.internal:{server.server_port}/index.html" + submit("url.brewshot.json", json.dumps(manifest({"url": url})).encode()) + wait("in/finished/url.brewshot.json") + info = gif_info(read_output("url.brewshot.json.gif")) + assert info[:3] == (160, 100, 4) and info[3] == [100] * 4 and info[4] >= 2, info + assert (input_dir / "notes.json").is_file() + + step = "ZIP assets, preparation script, selector and scale" + job = manifest({"file": "index.html"}, scale=0.5) + job["prepareScript"] = "prepare.js" + package = bundle(job) + submit("bundle.brewshot.zip", package) + wait("in/finished/bundle.brewshot.zip") + gif = read_output("bundle.brewshot.zip.gif") + info = gif_info(gif) + assert info[:3] == (64, 50, 4) and info[3] == [100] * 4 and info[4] >= 2, info + # Preserve artifacts outside owner-only mount for independent local decoding. + (root / "bundle-decoded-input.gif").write_bytes(gif) + + step = "format PNG and validation failures followed by later progress" + submit("still.brewshot.zip", bundle(manifest({"file": "index.html"}, "png"))) + wait("in/finished/still.brewshot.zip") + png = read_output("still.brewshot.zip.png") + assert png[:8] == b"\x89PNG\r\n\x1a\n" + assert struct.unpack_from(">II", png, 16) == (160, 100) + failures = { + "scheme.brewshot.json": json.dumps(manifest({"url": "file:///etc/passwd"})).encode(), + "range.brewshot.json": json.dumps(manifest({"url": url}, dpr=5)).encode(), + "unknown.brewshot.json": b'{"version":1,"unexpected":true}', + "selector.brewshot.json": json.dumps(manifest({"url": url}, selector=".absent")).encode(), + "unreachable.brewshot.json": json.dumps(manifest({"url": "http://127.0.0.1:1/"})).encode(), + "broken.brewshot.zip": b"not a ZIP", + "traversal.brewshot.zip": bundle(job, ("../escape", "bad")), + } + for name, data in failures.items(): + submit(name, data) + wait("in/failed/" + name) + error = read_output(name + ".error.txt") + assert 0 < len(error) <= 600 + assert not (output_dir / (name + ".gif")).exists() + + step = "outer deadline and worker progress afterwards" + submit("deadline.brewshot.json", json.dumps(manifest({"url": url}, settleMs=60000)).encode()) + wait("in/failed/deadline.brewshot.json", timeout=40) + assert b"job-deadline" in read_output("deadline.brewshot.json.error.txt") + submit("after.html", b"

Still alive

") + wait("in/finished/after.html") + assert read_output("after.html.png")[:8] == b"\x89PNG\r\n\x1a\n" + + step = "no replacement on same-name GIF resend" + before = hashlib.sha256(gif).hexdigest() + submit("bundle.brewshot.zip", package) + wait("in/failed/bundle.brewshot.zip") + assert hashlib.sha256(read_output("bundle.brewshot.zip.gif")).hexdigest() == before + + step = "restart recovery and two-worker single-job convergence" + docker("stop", "-t", "15", names[0]) + docker("rm", names[0]) + started.remove(names[0]) + claim_id = uuid.uuid4().hex + processing = input_dir / "processing" / claim_id + # Directories are worker-owned; create the synthetic recovered claim + # using the worker uid through its own image rather than host root assumptions. + docker("run", "--rm", "-v", f"{input_dir}:/brewshot/input", + "--entrypoint", "/bin/sh", image, "-c", + f"mkdir -p /brewshot/input/processing/{claim_id}") + staged = input_dir / ".recovered.upload" + staged.write_bytes(package) + staged.chmod(0o644) + docker("run", "--rm", "--user", "0:0", "-v", f"{input_dir}:/brewshot/input", + "--entrypoint", "/bin/sh", image, "-c", + f"cp /brewshot/input/.recovered.upload /brewshot/input/processing/{claim_id}/recovered.brewshot.zip") + start(names[0]) + start(names[1]) + wait("in/finished/recovered.brewshot.zip") + assert gif_info(read_output("recovered.brewshot.zip.gif"))[:3] == (64, 50, 4) + assert not processing.exists() + step = "restart after publication reuses the durable selected GIF" + for name in list(started): + docker("stop", "-t", "15", name) + docker("rm", name) + started.remove(name) + published_id = uuid.uuid4().hex + source_hash = hashlib.sha256(package).hexdigest() + # Model the exact durable filesystem state between publish and archive: + # immutable claim, committed image and its public hard link already exist. + docker("run", "--rm", "--user", "0:0", + "-v", f"{input_dir}:/brewshot/input", "-v", f"{output_dir}:/brewshot/output", + "--entrypoint", "/bin/sh", image, "-ec", + f"mkdir -p /brewshot/input/processing/{published_id} " + f"/brewshot/output/.brewshot-state/{published_id}; " + f"chown 10001:10001 /brewshot/input/processing/{published_id} " + f"/brewshot/output/.brewshot-state/{published_id}; " + f"cp /brewshot/input/.recovered.upload /brewshot/input/processing/{published_id}/published.brewshot.zip; " + f"cp /brewshot/output/bundle.brewshot.zip.gif /brewshot/output/.brewshot-state/{published_id}/{source_hash}.decision; " + f"chown 10001:10001 /brewshot/output/.brewshot-state/{published_id}/{source_hash}.decision; " + f"chmod 600 /brewshot/output/.brewshot-state/{published_id}/{source_hash}.decision; " + f"ln /brewshot/output/.brewshot-state/{published_id}/{source_hash}.decision /brewshot/output/published.brewshot.zip.gif") + start(names[0]) + start(names[1]) + wait("in/finished/published.brewshot.zip") + assert read_output("published.brewshot.zip.gif") == gif + assert not (input_dir / "failed/published.brewshot.zip").exists() + for name in started: + assert docker("inspect", "-f", "{{.State.Running}}", name).strip() == b"true" + print("watch job smoke: PASS — URL/ZIP GIFs, PNG, refusals, deadline, collision, recovery and two workers") + print(json.dumps({"bundleGif": info, "sha256": before, "image": image})) + preserve = os.environ.get("BREWSHOT_SMOKE_EVIDENCE_DIR") + if preserve: + destination = Path(preserve) + destination.mkdir(parents=True, exist_ok=True) + (destination / "watch-bundle.gif").write_bytes(gif) + (destination / "watch-bundle.png").write_bytes(png) + except Exception: + print(f"watch job smoke: FAILED during {step}", file=sys.stderr) + for name in started: + print(docker("logs", name, check=False).decode(errors="replace"), file=sys.stderr) + raise + finally: + server.shutdown() + server.server_close() + thread.join(timeout=2) + for name in started: + docker("rm", "-f", name, check=False) + # Fixed uid outputs are owner-only. Match the existing smoke's + # deliberate container-side reader/cleanup instead of weakening modes. + docker("run", "--rm", "--user", "0:0", "-v", f"{root}:/cleanup", + "--entrypoint", "/bin/sh", image, "-c", + "rm -rf /cleanup/in /cleanup/out", check=False) + shutil.rmtree(root) + + +if __name__ == "__main__": + main() diff --git a/docs/watch-jobs.md b/docs/watch-jobs.md new file mode 100644 index 0000000..f2efd73 --- /dev/null +++ b/docs/watch-jobs.md @@ -0,0 +1,140 @@ +# Watched PNG and GIF jobs + +The container's `watch` worker accepts three complete, visible, regular files +directly inside `/brewshot/input`. Publish with a hidden sibling and an atomic +rename; do not write directly into a visible job while it is being consumed. + +| Drop | Meaning | Successful output | +| --- | --- | --- | +| `page.html` or `page.htm` | Existing self-contained HTML still | `page.html.png` | +| `capture.brewshot.json` | Website URL plus capture settings | `capture.brewshot.json.gif` or `.png` | +| `capture.brewshot.zip` | Root `job.json`, local HTML and relative assets | `capture.brewshot.zip.gif` or `.png` | + +Other JSON/ZIP names are ignored. Outputs go to `/brewshot/output`. The +original submitted file moves to `input/finished`; failure produces a bounded +`.error.txt` and archives the input under `input/failed`. Existing outputs and +archives are never replaced. The whole ZIP is one claimed job, so its assets +cannot arrive after capture has started. Each worker processes jobs serially. + +Manifest jobs retain the selected complete capture beneath +`output/.brewshot-state//.decision`. A success decision +is the complete image; a failure decision is a bounded diagnostic. This is durable +recovery state: competing attempts honor the same first terminal result, and a +restart after publication does not record a changing page again. The public +image is a hard link, so its bytes are not duplicated. Keep this state with the +job archives; remove it only during deliberate offline retention maintenance +with all workers stopped. Ordinary top-level output names remain unchanged. + +## A website job + +Save this as `capture.brewshot.json`, substituting a website you trust: + +```json +{ + "version": 1, + "source": {"url": "https://example.com/animation"}, + "capture": { + "format": "gif", + "frames": 40, + "delayMs": 100, + "viewport": {"width": 1380, "height": 920}, + "dpr": 1, + "settleMs": 500, + "selector": ".stage", + "scale": 0.85 + } +} +``` + +Remove `selector` to film the full page. Set `format` to `png` and remove +`frames` and `delayMs` for a still image. A website needs no ZIP. + +The URL is fetched **from inside the container**. `localhost` names that +container, not the host. On Docker Desktop, use an explicit host address such +as `http://host.docker.internal:8080/...` for a host service. On Linux configure +a reachable hostname/network explicitly. BrewShot does not rewrite URLs. + +## A local bundle + +The ZIP root contains `job.json`, an HTML entry and any assets it references: + +```text +job.json +index.html +style.css +animation.js +images/button.png +prepare.js # optional +``` + +Use the same manifest with `"source": {"file": "index.html"}`. A bundle may +also set the root field `"prepareScript": "prepare.js"` to run a local script +after page load and settling, immediately before the first frame. This is +JavaScript in the page, with Main's existing eval semantics; it is not a host +command. An asynchronous preparation should return a promise. Keep preparation +bounded: the entire job shares one deadline. + +For v1, the entry is opened through **`file://`**, exactly as Main opens local +HTML. Relative CSS, images and ordinary script references are supported. Chrome +restricts local-file `fetch()`/XHR and ES module loading; a ZIP does not create +an HTTP origin. For an app needing HTTP semantics, serve it yourself and submit +a URL manifest. External resources still depend on the container's network. + +Create a normal single-disk ZIP containing the directory's contents, with +`job.json` at its root. ZIP64, encrypted archives, links/special files, duplicate +entries and unsafe paths are rejected. Stage the complete package before the +visible rename: + +```sh +cp capture.brewshot.zip Input/.capture.upload +mv Input/.capture.upload Input/capture.brewshot.zip +``` + +## Version 1 rules + +Unknown fields, duplicate JSON keys, wrong JSON types and unsupported versions +are rejected. Loose JSON accepts a URL source; ZIP accepts a local file source. +There is exactly one source. No output pathname or free-form CLI arguments are +accepted from a manifest. `format` is required; GIF also requires `frames`. + +| Manifest field | Main CLI mapping | +| --- | --- | +| `source.url` / `source.file` | Positional URL / extracted local HTML path | +| `capture.format = "gif"`, `capture.frames` | `--gif N`, GIF output extension | +| `capture.format = "png"` | PNG output extension | +| `capture.delayMs` | `--gif-delay` (GIF only) | +| `capture.viewport.width`, `height` | `--size WxH` (both required together) | +| `capture.dpr` | `--dpr` | +| `capture.settleMs` | `--settle` | +| `capture.selector` | GIF: `--gif-element`; PNG: `--clip-selector` | +| `capture.scale` | `--scale` | +| `prepareScript` | `--eval-file` (bundle only) | + +The manifest checks its schema, then delegates numeric ranges and capture +options to Main's own parser and validators. Defaults are Main defaults. +`scale` applies to the page or selector capture using the existing recorder. +Navigation happens first, followed by settling, preparation, then capture. +Requested GIF delay controls capture cadence; encoded playback delay rounds to +centiseconds (nearest 10 ms, half up, minimum 20 ms). A 40-frame, 100 ms recording +has four seconds of encoded playback; elapsed job time also includes navigation, +settling, encoding and cleanup. + +## Limits and failures + +The worker bounds each JSON/ZIP job to 120 seconds, including subprocess +preparation, navigation and capture. `BREWSHOT_WATCH_JOB_TIMEOUT_MS` configures +1000 through 600000 ms; termination can take up to 15 additional seconds for +bounded shutdown. A deadline failure publishes no success image. Main's +existing GIF frame/decoded-image budgets continue to apply. Archive input is +limited to 16 MiB, root JSON to 64 KiB, ZIP entries to 256, one expanded entry +to 16 MiB, and all expanded entries to 64 MiB. + +The worker publishes a fully written hidden image with the existing atomic, +no-replacement mechanism. A completed capture reporting exit 6 still publishes +its image and records its surviving-Chrome warning and counter. A timeout is +a failure and is **not** proof that Chrome resources were released. No unrelated +browser/service is stopped. Original job bytes remain available for diagnosis +or resubmission in the terminal archive. + +The input folder is a trusted-operator interface. See +[SECURITY.md](../SECURITY.md) for URL reach and executable-content boundaries. diff --git a/src/main/java/com/brewshot/BrewShotWatchCapture.java b/src/main/java/com/brewshot/BrewShotWatchCapture.java new file mode 100644 index 0000000..7345461 --- /dev/null +++ b/src/main/java/com/brewshot/BrewShotWatchCapture.java @@ -0,0 +1,73 @@ +package com.brewshot; + +import java.io.OutputStream; +import java.io.PrintStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.Map; + +/** Private subprocess entry point: the folder worker bounds this entire JVM's lifetime. */ +final class BrewShotWatchCapture { + private BrewShotWatchCapture() { } + + public static void main(String[] args) throws Exception { + if (args.length != 4) { System.exit(2); } + Path source = Path.of(args[0]); + Path target = Path.of(args[1]); + Path receipt = Path.of(args[2]); + Path scratch = Path.of(args[3]); + // URL queries, page text and eval results may contain secrets. Only the bounded + // structured result below crosses the subprocess boundary into watcher logs. + System.setOut(new PrintStream(OutputStream.nullOutputStream())); + System.setErr(new PrintStream(OutputStream.nullOutputStream())); + Map result = new LinkedHashMap<>(); + int exit = 1; + Path image = null; + try { + BrewShotWatchJob.Prepared job = BrewShotWatchJob.prepare(source, scratch); + image = target.resolveSibling(target.getFileName() + "." + job.format()); + var cli = literalArguments(job.arguments()); + cli.add("-o"); + cli.add(image.toString()); + // Keep diagnostics private to the job; Main supplies requested/encoded GIF + // timing and typed cleanup in its own receipt, with no second implementation. + cli.add("--json"); + cli.add(scratch.resolve("capture.json").toString()); + CleanupOutcome[] cleanup = new CleanupOutcome[1]; + exit = Main.run(cli.toArray(String[]::new), value -> cleanup[0] = value); + result.put("format", job.format()); + if (cleanup[0] != null) { result.put("cleanup", cleanup[0].toJson()); } + if (exit == 0 || exit == Main.EXIT_CLEANUP_UNCONFIRMED) { + // Both are hidden worker temporaries on the SAME output mount. Main + // created the image owner-only; rename preserves that permission mode. + Files.move(image, target, StandardCopyOption.ATOMIC_MOVE); + } + result.put("code", "capture-exit-" + exit); + } catch (Exception | LinkageError failure) { + // The parser's detail is intentionally not logged here, even on a future + // path that accidentally includes source content in an exception message. + String message = failure.getMessage(); + result.put("code", message != null && message.matches("JOB_[A-Z_]{1,48}") + ? message : "job-" + failure.getClass().getSimpleName()); + exit = 1; + } finally { + if (image != null) { Files.deleteIfExists(image); } + Files.writeString(receipt, MiniJson.stringify(result)); + } + System.exit(exit); + } + + static ArrayList literalArguments(java.util.List arguments) { + var result = new ArrayList(); + result.add(arguments.get(0)); + // Main supports --name=value. Use that spelling so its normalization + // cannot reinterpret a VALUE beginning "--x=y" as another option pair. + for (int index = 1; index < arguments.size(); index += 2) { + result.add(arguments.get(index) + "=" + arguments.get(index + 1)); + } + return result; + } +} diff --git a/src/main/java/com/brewshot/BrewShotWatchCommit.java b/src/main/java/com/brewshot/BrewShotWatchCommit.java new file mode 100644 index 0000000..fe84c0a --- /dev/null +++ b/src/main/java/com/brewshot/BrewShotWatchCommit.java @@ -0,0 +1,117 @@ +package com.brewshot; + +import java.io.IOException; +import java.nio.file.FileAlreadyExistsException; +import java.nio.file.Files; +import java.nio.file.LinkOption; +import java.nio.file.Path; + +/** + * Durable first-complete-capture selection for one immutable manifest claim. + * State is retained: an unlocked late attempt must see the same decision even + * after the winning attempt has archived its input and removed its temporary. + */ +final class BrewShotWatchCommit { + private final Path directory; + private final String sourceHash; + + BrewShotWatchCommit(Path output, String jobId, String sourceHash) throws IOException { + if (!jobId.matches("[a-f0-9]{32}") || !sourceHash.matches("[a-f0-9]{64}")) { + throw new IOException("invalid capture identity"); + } + Path state = output.resolve(".brewshot-state"); + ensureDirectory(state); + this.directory = state.resolve(jobId); + ensureDirectory(directory); + this.sourceHash = sourceHash; + } + + Path recover() throws IOException { + Path decision = directory.resolve(sourceHash + ".decision"); + if (!regularOrMissing(decision)) { return null; } + try (var input = Files.newInputStream(decision)) { + byte[] prefix = input.readNBytes(5); + if (java.util.Arrays.equals(prefix, "FAIL ".getBytes(java.nio.charset.StandardCharsets.US_ASCII))) { + byte[] code = input.readNBytes(81); + String text = new String(code, java.nio.charset.StandardCharsets.US_ASCII); + if (!text.matches("[A-Za-z0-9._-]{1,80}")) { + throw new IOException("invalid failure decision"); + } + throw new Failed(text); + } + } + return decision; + } + + Path record(Path complete, String suffix) throws IOException { + if (!suffix.equals(".png") && !suffix.equals(".gif")) { + throw new IOException("invalid capture format"); + } + Path target = directory.resolve(sourceHash + ".decision"); + try { + // A complete, owner-only image is committed BEFORE public output. + // Concurrent unlocked attempts choose exactly the same first image. + Files.createLink(target, complete); + } catch (FileAlreadyExistsException winner) { + if (!regularOrMissing(target)) { throw new IOException("capture disappeared"); } + } + return recover(); + } + + /** First terminal result wins, including a failing competing attempt. */ + Path fail(String code) throws IOException { + if (!code.matches("[A-Za-z0-9._-]{1,80}")) { code = "job-failed"; } + Path candidate = Files.createTempFile(directory, ".failure-", ".tmp"); + try { + Files.writeString(candidate, "FAIL " + code); + return record(candidate, ".png"); + } finally { Files.deleteIfExists(candidate); } + } + + static String suffix(Path committed) throws IOException { + try (var input = Files.newInputStream(committed)) { + byte[] header = input.readNBytes(8); + if (header.length >= 6 && header[0] == 'G' && header[1] == 'I' && header[2] == 'F') { + return ".gif"; + } + if (header.length == 8 && header[0] == (byte) 0x89 && header[1] == 'P' + && header[2] == 'N' && header[3] == 'G') { return ".png"; } + throw new IOException("invalid committed image"); + } + } + + static void publish(Path committed, Path target) throws IOException { + try { + Files.createLink(target, committed); + } catch (FileAlreadyExistsException existing) { + // Same claim retries compare against the durable image, never a new + // recording. Different claims still cannot replace an old output. + if (!Files.isRegularFile(target, LinkOption.NOFOLLOW_LINKS) + || Files.mismatch(committed, target) != -1) { + throw new Collision(); + } + } + } + + private static boolean regularOrMissing(Path path) throws IOException { + if (!Files.exists(path, LinkOption.NOFOLLOW_LINKS)) { return false; } + if (!Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS)) { + throw new IOException("invalid capture state"); + } + return true; + } + + private static void ensureDirectory(Path path) throws IOException { + try { Files.createDirectory(path); } + catch (FileAlreadyExistsException exists) { + if (!Files.isDirectory(path, LinkOption.NOFOLLOW_LINKS)) { + throw new IOException("invalid capture state directory"); + } + } + } + + static final class Collision extends IOException { } + static final class Failed extends IOException { + Failed(String code) { super(code); } + } +} diff --git a/src/main/java/com/brewshot/BrewShotWatchJob.java b/src/main/java/com/brewshot/BrewShotWatchJob.java new file mode 100644 index 0000000..7928272 --- /dev/null +++ b/src/main/java/com/brewshot/BrewShotWatchJob.java @@ -0,0 +1,415 @@ +package com.brewshot; + +import java.io.IOException; +import java.io.InputStream; +import java.math.BigDecimal; +import java.net.URI; +import java.nio.ByteBuffer; +import java.nio.charset.CharacterCodingException; +import java.nio.charset.CodingErrorAction; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.LinkOption; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.zip.CRC32; +import java.util.zip.ZipEntry; +import java.util.zip.ZipFile; + +/** Preparation only: worker-owned immutable input and empty, private scratch. */ +final class BrewShotWatchJob { + private static final int MAX_ARCHIVE = 16 * 1024 * 1024; + private static final int MAX_MANIFEST = 64 * 1024; + private static final int MAX_ENTRIES = 256; + private static final long MAX_EXPANDED = 64L * 1024 * 1024; + private static final int MAX_ENTRY = 16 * 1024 * 1024; + + private BrewShotWatchJob() { } + + record Prepared(String format, List arguments) { + Prepared { arguments = List.copyOf(arguments); } + } + + /** Exceptions deliberately contain codes only, never caller-controlled text. */ + static Prepared prepare(Path source, Path scratch) throws IOException { + try { + if (!Files.isRegularFile(source, LinkOption.NOFOLLOW_LINKS) + || !Files.isDirectory(scratch, LinkOption.NOFOLLOW_LINKS)) { + throw failure("JOB_IO"); + } + try (var children = Files.list(scratch)) { + if (children.findAny().isPresent()) { throw failure("JOB_SCRATCH"); } + } + String name = source.getFileName().toString().toLowerCase(Locale.ROOT); + Path assets = null; + String text; + if (name.endsWith(".brewshot.zip")) { + byte[] archive = readBounded(source, MAX_ARCHIVE); + List members = inspectArchive(archive); + assets = scratch.toAbsolutePath().resolve("assets"); + extract(source, assets, members); + text = BoundedUtf8.readStrict(assets.resolve("job.json"), + MAX_MANIFEST, "manifest"); + } else if (name.endsWith(".brewshot.json")) { + if (Files.size(source) > MAX_MANIFEST) { throw failure("JOB_LIMIT"); } + text = BoundedUtf8.readStrict(source, MAX_MANIFEST, "manifest"); + } else { + throw failure("JOB_EXTENSION"); + } + Object parsed; + try { parsed = MiniJson.parseStrict(text); } + catch (IllegalArgumentException malformed) { throw failure("JOB_JSON_INVALID"); } + return manifest(parsed, assets); + } catch (JobFailure controlled) { + throw controlled; + } catch (IOException unreadable) { + throw failure("JOB_IO"); + } catch (RuntimeException malformed) { + throw failure("JOB_SCHEMA"); + } + } + + private static Prepared manifest(Object parsed, Path assets) throws IOException { + Map root = object(parsed, Set.of("version", "source", "capture", "prepareScript")); + if (!(root.get("version") instanceof Double version) || version != 1.0) { + throw failure("JOB_SCHEMA"); + } + Map source = object(root.get("source"), + assets == null ? Set.of("url") : Set.of("file")); + if (source.size() != 1) { throw failure("JOB_SCHEMA"); } + String input; + if (assets == null) { + input = string(source.get("url")); + URI uri; + try { uri = new URI(input); } + catch (java.net.URISyntaxException malformed) { throw failure("JOB_URL"); } + if (!("https".equalsIgnoreCase(uri.getScheme()) + || "http".equalsIgnoreCase(uri.getScheme())) + || uri.getHost() == null || uri.getRawUserInfo() != null) { + throw failure("JOB_URL"); + } + // Main's URL-mode recognizer expects a lower-case scheme. Preserve + // the rest verbatim, including escaped path and query bytes. + input = uri.getScheme().toLowerCase(Locale.ROOT) + input.substring(input.indexOf(':')); + } else { + Path file = bundleFile(assets, string(source.get("file"))); + String lower = file.getFileName().toString().toLowerCase(Locale.ROOT); + if (!(lower.endsWith(".html") || lower.endsWith(".htm"))) { + throw failure("JOB_PATH"); + } + input = file.toString(); + } + Map capture = object(root.get("capture"), Set.of("format", "frames", "delayMs", + "viewport", "dpr", "settleMs", "selector", "scale")); + String format = string(capture.get("format")); + if (!(format.equals("gif") || format.equals("png"))) { throw failure("JOB_SCHEMA"); } + List args = new ArrayList<>(); + args.add(input); + if (format.equals("gif")) { + add(args, "--gif", integer(capture.get("frames"))); + optionalNumber(capture, "delayMs", "--gif-delay", true, args); + } else if (capture.containsKey("frames") || capture.containsKey("delayMs")) { + throw failure("JOB_SCHEMA"); + } + if (capture.containsKey("viewport")) { + Map viewport = object(capture.get("viewport"), Set.of("width", "height")); + add(args, "--size", integer(viewport.get("width")) + "x" + + integer(viewport.get("height"))); + } + optionalNumber(capture, "dpr", "--dpr", true, args); + optionalNumber(capture, "settleMs", "--settle", true, args); + if (capture.containsKey("selector")) { + add(args, format.equals("gif") ? "--gif-element" : "--clip-selector", + string(capture.get("selector"))); + } + optionalNumber(capture, "scale", "--scale", false, args); + if (root.containsKey("prepareScript")) { + if (assets == null) { throw failure("JOB_SCHEMA"); } + Path script = bundleFile(assets, string(root.get("prepareScript"))); + if (!script.getFileName().toString().toLowerCase(Locale.ROOT).endsWith(".js")) { + throw failure("JOB_PATH"); + } + // Same byte cap and decoding as Main's --eval-file; no independent JS grammar. + if (Files.size(script) > Main.MAX_EVAL_FILE_BYTES) { throw failure("JOB_LIMIT"); } + BoundedUtf8.read(script, Main.MAX_EVAL_FILE_BYTES, "script"); + add(args, "--eval-file", script.toString()); + } + return new Prepared(format, args); + } + + private static Map object(Object value, Set keys) throws IOException { + if (!(value instanceof Map map) || !keys.containsAll(map.keySet())) { + throw failure("JOB_SCHEMA"); + } + return map; + } + + private static String string(Object value) throws IOException { + if (!(value instanceof String string)) { throw failure("JOB_SCHEMA"); } + return string; + } + + private static String integer(Object value) throws IOException { + if (!(value instanceof Double number) || !Double.isFinite(number) + || number != Math.rint(number)) { throw failure("JOB_SCHEMA"); } + // MiniJson's numeric domain is Double. Emit plain decimal integer tokens, + // including out-of-range values, for Main's single CLI validator to decide. + return BigDecimal.valueOf(number).toBigIntegerExact().toString(); + } + + private static void optionalNumber(Map map, String key, String flag, + boolean integral, List args) throws IOException { + if (!map.containsKey(key)) { return; } + Object value = map.get(key); + String token; + if (integral) { token = integer(value); } + else { + if (!(value instanceof Double number) || !Double.isFinite(number)) { + throw failure("JOB_SCHEMA"); + } + token = Double.toString(number); + } + add(args, flag, token); + } + + private static void add(List args, String flag, String value) { + args.add(flag); + args.add(value); + } + + private static Path bundleFile(Path assets, String name) throws IOException { + safeName(name, false); + Path path = assets.resolve(name); + if (!Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS)) { throw failure("JOB_PATH"); } + return path; + } + + private record Member(String name, boolean directory, long size, long compressed, + long crc, int localOffset, int method, int flags, byte[] rawName) { } + + /** + * Inspect the bounded central directory because ZipEntry does not expose Unix + * file modes. Crosscheck local headers as well: no alternate names, encryption, + * hidden local members, ZIP64, special files, or overlapping records. + */ + private static List inspectArchive(byte[] bytes) throws IOException { + int end = -1; + for (int p = bytes.length - 22; p >= Math.max(0, bytes.length - 65557); p--) { + if (u32(bytes, p) == 0x06054b50L && p + 22L + u16(bytes, p + 20) == bytes.length) { + end = p; + break; + } + } + if (end < 0 || u16(bytes, end + 4) != 0 || u16(bytes, end + 6) != 0 + || u16(bytes, end + 8) != u16(bytes, end + 10)) { + throw failure("JOB_ZIP_INVALID"); + } + int count = u16(bytes, end + 10); + long centralSize = u32(bytes, end + 12); + long centralStart = u32(bytes, end + 16); + if (count == 0xffff || centralSize == 0xffffffffL || centralStart == 0xffffffffL) { + throw failure("JOB_ZIP_INVALID"); + } + if (count == 0 || count > MAX_ENTRIES) { throw failure("JOB_LIMIT"); } + if (centralStart + centralSize != end) { throw failure("JOB_ZIP_INVALID"); } + int at = (int) centralStart; + List members = new ArrayList<>(); + Map names = new HashMap<>(); + long total = 0; + for (int index = 0; index < count; index++) { + require(bytes, at, 46); + if (u32(bytes, at) != 0x02014b50L) { throw failure("JOB_ZIP_INVALID"); } + int flags = u16(bytes, at + 8); + int method = u16(bytes, at + 10); + long crc = u32(bytes, at + 16); + long compressed = u32(bytes, at + 20); + long size = u32(bytes, at + 24); + int nameLength = u16(bytes, at + 28); + int extraLength = u16(bytes, at + 30); + int commentLength = u16(bytes, at + 32); + long localOffset = u32(bytes, at + 42); + if ((flags & ~(0x800 | 8)) != 0 || (method != 0 && method != 8) + || u16(bytes, at + 34) != 0 || compressed == 0xffffffffL + || size == 0xffffffffL || localOffset == 0xffffffffL) { + throw failure("JOB_ZIP_INVALID"); + } + require(bytes, at + 46, nameLength + extraLength + commentLength); + byte[] rawName = Arrays.copyOfRange(bytes, at + 46, at + 46 + nameLength); + String name = decodeName(rawName); + boolean directory = name.endsWith("/"); + safeName(name, directory); + int mode = (int) ((u32(bytes, at + 38) >>> 16) & 0xffff); + int type = mode & 0170000; + if (type != 0 && type != (directory ? 0040000 : 0100000) + || !directory && (u32(bytes, at + 38) & 0x10) != 0) { + throw failure("JOB_ZIP_INVALID"); + } + String key = directory ? name.substring(0, name.length() - 1) : name; + if (names.putIfAbsent(key, directory) != null) { throw failure("JOB_ZIP_INVALID"); } + extras(bytes, at + 46 + nameLength, extraLength); + if (size > MAX_ENTRY || (total += size) > MAX_EXPANDED + || name.equals("job.json") && size > MAX_MANIFEST) { + throw failure("JOB_LIMIT"); + } + if (directory && (size != 0 || crc != 0)) { throw failure("JOB_ZIP_INVALID"); } + members.add(new Member(name, directory, size, compressed, crc, + (int) localOffset, method, flags, rawName)); + at += 46 + nameLength + extraLength + commentLength; + } + if (at != end || !Boolean.FALSE.equals(names.get("job.json"))) { + throw failure("JOB_ZIP_INVALID"); + } + for (String name : names.keySet()) { + for (int slash = name.indexOf('/'); slash >= 0; slash = name.indexOf('/', slash + 1)) { + if (Boolean.FALSE.equals(names.get(name.substring(0, slash)))) { + throw failure("JOB_ZIP_INVALID"); + } + } + } + List localOrder = new ArrayList<>(members); + localOrder.sort(java.util.Comparator.comparingInt(Member::localOffset)); + long next = 0; + for (Member member : localOrder) { + int local = member.localOffset; + if (next != local) { throw failure("JOB_ZIP_INVALID"); } + require(bytes, local, 30); + int nameLength = u16(bytes, local + 26); + int extraLength = u16(bytes, local + 28); + require(bytes, local + 30, nameLength + extraLength); + if (u32(bytes, local) != 0x04034b50L || u16(bytes, local + 6) != member.flags + || u16(bytes, local + 8) != member.method + || u32(bytes, local + 18) == 0xffffffffL + || u32(bytes, local + 22) == 0xffffffffL + || !Arrays.equals(member.rawName, + Arrays.copyOfRange(bytes, local + 30, local + 30 + nameLength))) { + throw failure("JOB_ZIP_INVALID"); + } + extras(bytes, local + 30 + nameLength, extraLength); + next = local + 30L + nameLength + extraLength + member.compressed; + if (next > centralStart) { throw failure("JOB_ZIP_INVALID"); } + if ((member.flags & 8) == 0) { + if (u32(bytes, local + 14) != member.crc + || u32(bytes, local + 18) != member.compressed + || u32(bytes, local + 22) != member.size) { + throw failure("JOB_ZIP_INVALID"); + } + } else { + int descriptor = (int) next; + if (u32(bytes, descriptor) == 0x08074b50L) { descriptor += 4; } + if (u32(bytes, descriptor) != member.crc + || u32(bytes, descriptor + 4) != member.compressed + || u32(bytes, descriptor + 8) != member.size) { + throw failure("JOB_ZIP_INVALID"); + } + next = descriptor + 12L; + } + } + if (next != centralStart) { throw failure("JOB_ZIP_INVALID"); } + return members; + } + + private static void extras(byte[] bytes, int at, int length) throws IOException { + int end = at + length; + while (at < end) { + if (at + 4 > end) { throw failure("JOB_ZIP_INVALID"); } + int id = u16(bytes, at); + int size = u16(bytes, at + 2); + if (id == 1 || at + 4L + size > end) { throw failure("JOB_ZIP_INVALID"); } + at += 4 + size; + } + } + + private static String decodeName(byte[] bytes) throws IOException { + try { + return StandardCharsets.UTF_8.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(bytes)).toString(); + } catch (CharacterCodingException malformed) { throw failure("JOB_ZIP_INVALID"); } + } + + private static void safeName(String name, boolean directory) throws IOException { + if (name.isEmpty() || name.startsWith("/") || name.indexOf('\\') >= 0 + || name.indexOf(':') >= 0 || name.chars().anyMatch(c -> c < 32 || c == 127)) { + throw failure("JOB_PATH"); + } + String stripped = directory ? name.substring(0, name.length() - 1) : name; + for (String component : stripped.split("/", -1)) { + if (component.isEmpty() || component.equals(".") || component.equals("..")) { + throw failure("JOB_PATH"); + } + } + } + + private static void extract(Path source, Path assets, List members) throws IOException { + Files.createDirectory(assets); + long total = 0; + try (ZipFile zip = new ZipFile(source.toFile(), StandardCharsets.UTF_8)) { + for (Member member : members) { + Path target = assets.resolve(member.name); + if (member.directory) { Files.createDirectories(target); } + else { Files.createDirectories(target.getParent()); } + ZipEntry entry = zip.getEntry(member.name); + if (entry == null) { throw failure("JOB_ZIP_INVALID"); } + CRC32 crc = new CRC32(); + long count = 0; + try (InputStream in = zip.getInputStream(entry); + var out = member.directory ? java.io.OutputStream.nullOutputStream() + : Files.newOutputStream(target, StandardOpenOption.CREATE_NEW)) { + byte[] buffer = new byte[8192]; + int n; + while ((n = in.read(buffer)) != -1) { + count += n; + total += n; + if (count > MAX_ENTRY || count > member.size || total > MAX_EXPANDED) { + throw failure("JOB_LIMIT"); + } + crc.update(buffer, 0, n); + out.write(buffer, 0, n); + } + } + if (count != member.size || crc.getValue() != member.crc) { + throw failure("JOB_ZIP_INVALID"); + } + } + } + } + + private static byte[] readBounded(Path source, int max) throws IOException { + if (Files.size(source) > max) { throw failure("JOB_LIMIT"); } + try (InputStream in = Files.newInputStream(source)) { + byte[] bytes = in.readNBytes(max + 1); + if (bytes.length > max) { throw failure("JOB_LIMIT"); } + return bytes; + } + } + + private static int u16(byte[] bytes, int offset) throws IOException { + require(bytes, offset, 2); + return (bytes[offset] & 255) | ((bytes[offset + 1] & 255) << 8); + } + + private static long u32(byte[] bytes, int offset) throws IOException { + require(bytes, offset, 4); + return (long) u16(bytes, offset) | ((long) u16(bytes, offset + 2) << 16); + } + + private static void require(byte[] bytes, int offset, int size) throws IOException { + if (offset < 0 || size < 0 || offset + (long) size > bytes.length) { + throw failure("JOB_ZIP_INVALID"); + } + } + + private static JobFailure failure(String code) { return new JobFailure(code); } + + private static final class JobFailure extends IOException { + JobFailure(String code) { super(code); } + } +} diff --git a/src/test/java/com/brewshot/BrewShotWatchCommitTest.java b/src/test/java/com/brewshot/BrewShotWatchCommitTest.java new file mode 100644 index 0000000..08344bf --- /dev/null +++ b/src/test/java/com/brewshot/BrewShotWatchCommitTest.java @@ -0,0 +1,81 @@ +package com.brewshot; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.Executors; +import static org.junit.jupiter.api.Assertions.*; + +class BrewShotWatchCommitTest { + @TempDir Path directory; + private static final String JOB = "a".repeat(32), SOURCE = "b".repeat(64); + + @Test void restartAfterPublicationReusesOriginalImageEvenWhenNewRecordingDiffers() throws Exception { + Path first = Files.writeString(directory.resolve("first"), "first complete capture"); + var beforeCrash = new BrewShotWatchCommit(directory, JOB, SOURCE); + Path committed = beforeCrash.record(first, ".gif"); + Path output = directory.resolve("job.gif"); + BrewShotWatchCommit.publish(committed, output); + Files.delete(first); // process exited before input archive + var afterRestart = new BrewShotWatchCommit(directory, JOB, SOURCE); + assertEquals(committed, afterRestart.recover()); + Path different = Files.writeString(directory.resolve("new"), "different animation frame"); + assertEquals(committed, afterRestart.record(different, ".gif")); + BrewShotWatchCommit.publish(committed, output); + assertEquals("first complete capture", Files.readString(output)); + } + + @Test void concurrentUnlockedAttemptsChooseOneCaptureBeforePublishing() throws Exception { + Path one = Files.writeString(directory.resolve("one"), "one"); + Path two = Files.writeString(directory.resolve("two"), "two"); + var start = new CountDownLatch(1); + Path output = directory.resolve("race.gif"); + try (var pool = Executors.newFixedThreadPool(2)) { + var first = pool.submit(() -> { + var commit = new BrewShotWatchCommit(directory, JOB, SOURCE); + start.await(); + Path chosen = commit.record(one, ".gif"); + BrewShotWatchCommit.publish(chosen, output); + return chosen; + }); + var second = pool.submit(() -> { + var commit = new BrewShotWatchCommit(directory, JOB, SOURCE); + start.await(); + Path chosen = commit.record(two, ".gif"); + BrewShotWatchCommit.publish(chosen, output); + return chosen; + }); + start.countDown(); + assertEquals(first.get(), second.get()); + } + assertTrue(Files.readString(output).equals("one") || Files.readString(output).equals("two")); + assertEquals(Files.readString(new BrewShotWatchCommit(directory, JOB, SOURCE).recover()), + Files.readString(output)); + } + + @Test void foreignOutputAndDifferentSourceAreNotMistakenForTheSameCommit() throws Exception { + Path complete = Files.writeString(directory.resolve("complete"), "image"); + var commit = new BrewShotWatchCommit(directory, JOB, SOURCE); + Path chosen = commit.record(complete, ".png"); + Path foreign = Files.writeString(directory.resolve("foreign.png"), "foreign bytes"); + assertThrows(BrewShotWatchCommit.Collision.class, + () -> BrewShotWatchCommit.publish(chosen, foreign)); + assertEquals("foreign bytes", Files.readString(foreign)); + assertNull(new BrewShotWatchCommit(directory, JOB, "c".repeat(64)).recover()); + } + + @Test void aLateFailureCannotOverruleSuccessAndALateSuccessCannotOverruleFailure() throws Exception { + Path image = Files.writeString(directory.resolve("image"), "image bytes"); + var success = new BrewShotWatchCommit(directory, JOB, SOURCE); + Path selected = success.record(image, ".gif"); + assertEquals(selected, success.fail("job-deadline")); + var failure = new BrewShotWatchCommit(directory, "d".repeat(32), SOURCE); + assertEquals("job-deadline", assertThrows(BrewShotWatchCommit.Failed.class, + () -> failure.fail("job-deadline")).getMessage()); + assertEquals("job-deadline", assertThrows(BrewShotWatchCommit.Failed.class, + () -> failure.record(image, ".gif")).getMessage()); + assertThrows(BrewShotWatchCommit.Failed.class, failure::recover); + } +} diff --git a/src/test/java/com/brewshot/BrewShotWatchJobTest.java b/src/test/java/com/brewshot/BrewShotWatchJobTest.java new file mode 100644 index 0000000..dc30090 --- /dev/null +++ b/src/test/java/com/brewshot/BrewShotWatchJobTest.java @@ -0,0 +1,434 @@ +package com.brewshot; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.zip.CRC32; +import java.util.zip.ZipEntry; +import java.util.zip.ZipOutputStream; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** Pure file/schema preparation; no Chrome, subprocesses, or network. */ +class BrewShotWatchJobTest { + @TempDir Path temporary; + private int sequence; + + @Test + void urlGifMapsOnlyDocumentedOptionsAndPreservesCliDefaults() throws Exception { + BrewShotWatchJob.Prepared job = json(""" + {"version":1,"source":{"url":"https://example.test/scene?q=1"}, + "capture":{"format":"gif","frames":40,"delayMs":100, + "viewport":{"width":1380,"height":920},"dpr":2,"settleMs":500, + "selector":".stage","scale":0.85}} + """); + assertEquals("gif", job.format()); + assertEquals(List.of("https://example.test/scene?q=1", "--gif", "40", + "--gif-delay", "100", "--size", "1380x920", "--dpr", "2", + "--settle", "500", "--gif-element", ".stage", "--scale", "0.85"), + job.arguments()); + assertFalse(job.arguments().contains("-o")); + assertFalse(job.arguments().contains("--json")); + assertThrows(UnsupportedOperationException.class, + () -> job.arguments().add("--unexpected")); + assertEquals(List.of("http://localhost:8080/"), json(urlJob("png", "")).arguments()); + assertEquals(List.of("http://localhost:8080/", "--gif", "3"), + json(urlJob("gif", ",\"frames\":3")).arguments()); + } + + @Test + void pngSelectorUsesStillCaptureFlagAndLeavesSelectorLiteral() throws Exception { + assertEquals(List.of("http://localhost:8080/", "--clip-selector", "#animation", + "--scale", "2.0"), json(urlJob("png", + ",\"selector\":\"#animation\",\"scale\":2")).arguments()); + assertEquals(List.of("http://localhost:8080/", "--clip-selector", "--out"), + json(urlJob("png", ",\"selector\":\"--out\"")).arguments(), + "selector is one argv value, never interpreted as caller-supplied flags"); + } + + @Test + void numericRangesAndCombinationsRemainTheCliValidatorsResponsibility() throws Exception { + BrewShotWatchJob.Prepared job = json(urlJob("gif", """ + ,"frames":0,"delayMs":-2,"viewport":{"width":-1,"height":0}, + "dpr":99,"settleMs":-1,"scale":0 + """)); + assertEquals(List.of("http://localhost:8080/", "--gif", "0", "--gif-delay", "-2", + "--size", "-1x0", "--dpr", "99", "--settle", "-1", "--scale", "0.0"), + job.arguments()); + assertEquals("100000000000000000000", json(urlJob("gif", ",\"frames\":1e20")) + .arguments().get(2), "integer token, not overflow/clamping or exponent syntax"); + assertEquals("-1", json(urlJob("gif", ",\"frames\":-1.0")) + .arguments().get(2)); + } + + @Test + void schemaRefusesUnknownFieldsWrongTypesMissingFieldsAndPngGifOptions() throws Exception { + for (String content : List.of( + "[]", "null", "{}", + "{\"version\":\"1\",\"source\":{\"url\":\"https://example.test\"},\"capture\":{\"format\":\"png\"}}", + "{\"version\":2,\"source\":{\"url\":\"https://example.test\"},\"capture\":{\"format\":\"png\"}}", + urlJob("jpg", ""), urlJob("gif", ""), + urlJob("png", ",\"frames\":1"), urlJob("png", ",\"delayMs\":1"), + urlJob("gif", ",\"frames\":\"40\""), + urlJob("gif", ",\"frames\":true"), + urlJob("gif", ",\"frames\":1.5"), + urlJob("png", ",\"dpr\":1.5"), + urlJob("png", ",\"settleMs\":null"), + urlJob("png", ",\"scale\":\"2\""), + urlJob("png", ",\"selector\":false"), + urlJob("png", ",\"viewport\":{\"width\":390}"), + urlJob("png", ",\"viewport\":{\"width\":390,\"height\":800,\"auto\":true}"), + urlJob("png", ",\"out\":\"escape.png\""), + urlJob("png", "").replace("\"version\":1", "\"version\":1,\"args\":[\"--eval\",\"x\"]"), + urlJob("png", "").replace("\"url\":\"http://localhost:8080/\"", + "\"url\":\"http://localhost:8080/\",\"file\":\"page.html\""), + urlJob("png", "").replace("\"url\":\"http://localhost:8080/\"", + "\"file\":\"page.html\""), + urlJob("png", "").replace("\"capture\":", "\"prepareScript\":\"prepare.js\",\"capture\":"))) { + assertCode("JOB_SCHEMA", () -> json(content)); + } + } + + @Test + void strictJsonRefusesDuplicatesInvalidNumbersAndMalformedUtf8() throws Exception { + for (String content : List.of( + "{\"version\":1,\"version\":1}", + urlJob("png", ",\"dpr\":+1"), + urlJob("png", ",\"dpr\":01"), + urlJob("png", ",\"scale\":1e309"), + urlJob("png", ",\"selector\":\"\\ud800\""), + urlJob("png", ",\"selector\":\"raw\ncontrol\""), + urlJob("png", "") + " garbage")) { + assertCode("JOB_JSON_INVALID", () -> json(content)); + } + Path source = input(".brewshot.json", new byte[] {(byte) 0xc3, 0x28}); + assertCode("JOB_IO", () -> BrewShotWatchJob.prepare(source, scratch())); + } + + @Test + void urlIntakeRequiresHttpHostAndNoEmbeddedCredentials() throws Exception { + for (String url : List.of("file:///etc/passwd", "data:text/html,hello", + "chrome://version", "javascript:alert(1)", "//example.test/path", + "http:example.test", "https:///missing", "https://user:secret@example.test/", + "https://@example.test/", "https://example.test/a b")) { + String content = urlJob("png", "").replace("http://localhost:8080/", url); + assertCode("JOB_URL", () -> json(content)); + } + assertEquals("http://example.test/path", json(urlJob("png", "") + .replace("http://localhost:8080/", "HTTP://example.test/path")).arguments().get(0)); + } + + @Test + void bundleRetainsRelativeAssetsAndMapsBoundedPreparationFile() throws Exception { + String manifest = """ + {"version":1,"source":{"file":"site/index.html"}, + "capture":{"format":"gif","frames":4},"prepareScript":"prepare.js"} + """; + byte[] image = {0, 1, 2, 3}; + Path source = input(".brewshot.zip", deflatedZip(List.of( + new Fixture("job.json", utf8(manifest), 0100644), + new Fixture("site/", new byte[0], 0040755), + new Fixture("site/index.html", utf8(""), 0100644), + new Fixture("site/image.png", image, 0100644), + new Fixture("site/scene.js", utf8("document.body.dataset.ready='yes'"), 0100644), + new Fixture("prepare.js", utf8("document.body.click()"), 0100644)))); + Path scratch = scratch(); + BrewShotWatchJob.Prepared prepared = BrewShotWatchJob.prepare(source, scratch); + assertEquals("gif", prepared.format()); + assertEquals(List.of(scratch.resolve("assets/site/index.html").toString(), + "--gif", "4", "--eval-file", scratch.resolve("assets/prepare.js").toString()), + prepared.arguments()); + assertArrayEquals(image, Files.readAllBytes(scratch.resolve("assets/site/image.png"))); + assertEquals("document.body.dataset.ready='yes'", + Files.readString(scratch.resolve("assets/site/scene.js"))); + } + + @Test + void storedZipAndPlainIntegerVersionAreSupported() throws Exception { + BrewShotWatchJob.Prepared prepared = bundle(storedZip(basicMembers())); + assertEquals("png", prepared.format()); + assertEquals(1, prepared.arguments().size()); + assertTrue(prepared.arguments().get(0).endsWith("/assets/page.html")); + } + + @Test + void bundleSourceAndPreparationMustBeExistingLocalHtmlAndJs() throws Exception { + for (String replacement : List.of("missing.html", "job.json", "../page.html", + "/page.html", "C:/page.html", "folder/./page.html", "folder//page.html")) { + List members = new ArrayList<>(basicMembers()); + members.set(0, fixture("job.json", fileJob(replacement))); + assertBoundedFailure(() -> bundle(storedZip(members))); + } + List members = new ArrayList<>(basicMembers()); + members.set(0, fixture("job.json", """ + {"version":1,"source":{"url":"https://example.test"},"capture":{"format":"png"}} + """)); + assertCode("JOB_SCHEMA", () -> bundle(storedZip(members))); + for (String script : List.of("absent.js", "page.html", "../prepare.js")) { + List scriptMembers = new ArrayList<>(basicMembers()); + scriptMembers.set(0, fixture("job.json", fileJob("page.html") + .replace("\"version\":1", "\"version\":1,\"prepareScript\":\"" + script + "\""))); + assertCode("JOB_PATH", () -> bundle(storedZip(scriptMembers))); + } + } + + @Test + void rejectsTraversalDuplicateCollisionsAndSpecialFileModesBeforeExtraction() throws Exception { + for (String name : List.of("../escape", "a/../escape", "a/./escape", "/escape", + "C:/escape", "a\\escape", "a//escape", "a/", "bad\u0000name")) { + List members = new ArrayList<>(basicMembers()); + // a/ is a normal directory only with directory metadata; here it is regular. + members.add(new Fixture(name, utf8("payload"), 0100644)); + Path scratch = scratch(); + Path source = input(".brewshot.zip", storedZip(members)); + assertBoundedFailure(() -> BrewShotWatchJob.prepare(source, scratch)); + assertFalse(Files.exists(scratch.resolve("assets")), "inspect before extracting"); + } + for (List extra : List.of( + List.of(fixture("page.html", "duplicate")), + List.of(fixture("job.json", "duplicate manifest")), + List.of(fixture("page.html/child", "file parent collision")), + List.of(new Fixture("page.html/", new byte[0], 0040755)), + List.of(new Fixture("link", utf8("/etc/passwd"), 0120777)), + List.of(new Fixture("fifo", new byte[0], 0010644)), + List.of(new Fixture("device", new byte[0], 0020644)))) { + List members = new ArrayList<>(basicMembers()); + members.addAll(extra); + assertCode("JOB_ZIP_INVALID", () -> bundle(storedZip(members))); + } + assertFalse(Files.exists(temporary.resolve("escape"))); + } + + @Test + void archiveRequiresExactlyOneRootManifestAndRejectsCorruption() throws Exception { + assertCode("JOB_ZIP_INVALID", () -> bundle(storedZip(List.of(fixture("page.html", "html"))))); + assertCode("JOB_ZIP_INVALID", () -> bundle(storedZip(List.of( + fixture("nested/job.json", fileJob("page.html")), fixture("page.html", "html"))))); + byte[] valid = storedZip(basicMembers()); + assertBoundedFailure(() -> bundle(Arrays.copyOf(valid, valid.length - 1))); + assertBoundedFailure(() -> bundle(new byte[] {1, 2, 3})); + byte[] badData = valid.clone(); + int payload = 30 + "job.json".length(); + badData[payload] ^= 1; + assertCode("JOB_ZIP_INVALID", () -> bundle(badData)); + byte[] differentLocalName = valid.clone(); + differentLocalName[30] = 'x'; + assertCode("JOB_ZIP_INVALID", () -> bundle(differentLocalName)); + byte[] descriptor = deflatedZip(basicMembers()); + int descriptorAt = firstSignature(descriptor, 0x08074b50); + descriptor[descriptorAt + 4] ^= 1; + assertCode("JOB_ZIP_INVALID", () -> bundle(descriptor)); + byte[] trailingJunk = Arrays.copyOf(valid, valid.length + 1); + assertCode("JOB_ZIP_INVALID", () -> bundle(trailingJunk)); + } + + @Test + void refusesEncryptedZip64AndCentralLocalMetadataDisagreements() throws Exception { + byte[] original = storedZip(basicMembers()); + int central = firstSignature(original, 0x02014b50); + for (int mutation = 0; mutation < 7; mutation++) { + byte[] changed = original.clone(); + switch (mutation) { + case 0 -> { put16(changed, 6, 1); put16(changed, central + 8, 1); } + case 1 -> put32(changed, central + 24, 0xffffffffL); + case 2 -> put16(changed, original.length - 22 + 10, 0xffff); + case 3 -> put16(changed, central + 34, 1); + case 4 -> put32(changed, central + 42, 1); + case 5 -> put16(changed, 8, 8); + case 6 -> put32(changed, central + 16, 0); + default -> throw new AssertionError(); + } + assertCode("JOB_ZIP_INVALID", () -> bundle(changed)); + } + // A ZIP64 extra must be rejected even without sentinel size fields. + // Fixtures with a four-byte local extra are easier to construct explicitly. + byte[] ordinaryExtra = zipWithExtra(new byte[] {2, 0, 0, 0}); + assertEquals("png", bundle(ordinaryExtra).format()); + // ZipOutputStream rewrites its own ZIP64 fields, so mutate an ordinary + // local extra after writing instead of asking that writer for ZIP64. + byte[] zip64 = ordinaryExtra.clone(); + put16(zip64, 30 + "job.json".length(), 1); + assertCode("JOB_ZIP_INVALID", () -> bundle(zip64)); + } + + @Test + void enforcesManifestCompressedInputEntryCountAndExpandedLimits() throws Exception { + String small = urlJob("png", ""); + assertEquals("png", json(small + " ".repeat(64 * 1024 - utf8(small).length)).format()); + assertCode("JOB_LIMIT", () -> json(" ".repeat(64 * 1024 + 1))); + assertCode("JOB_LIMIT", () -> bundle(new byte[16 * 1024 * 1024 + 1])); + List exactCount = new ArrayList<>(basicMembers()); + for (int i = 0; i < 254; i++) { exactCount.add(fixture("asset" + i, "")); } + assertEquals("png", bundle(storedZip(exactCount)).format()); + List tooMany = new ArrayList<>(basicMembers()); + for (int i = 0; i < 255; i++) { tooMany.add(fixture("asset" + i, "")); } + assertCode("JOB_LIMIT", () -> bundle(storedZip(tooMany))); + List giantManifest = new ArrayList<>(basicMembers()); + giantManifest.set(0, new Fixture("job.json", new byte[64 * 1024 + 1], 0100644)); + assertCode("JOB_LIMIT", () -> bundle(deflatedZip(giantManifest))); + List giantEntry = new ArrayList<>(basicMembers()); + giantEntry.add(new Fixture("big", new byte[16 * 1024 * 1024 + 1], 0100644)); + assertCode("JOB_LIMIT", () -> bundle(deflatedZip(giantEntry))); + List giantTotal = new ArrayList<>(basicMembers()); + byte[] sixteenMib = new byte[16 * 1024 * 1024]; + for (int i = 0; i < 4; i++) { giantTotal.add(new Fixture("big" + i, sixteenMib, 0100644)); } + assertCode("JOB_LIMIT", () -> bundle(deflatedZip(giantTotal))); + } + + @Test + void usesMainsPreparationScriptByteCapAndRequiresEmptyPrivateScratch() throws Exception { + for (int size : new int[] {Main.MAX_EVAL_FILE_BYTES, Main.MAX_EVAL_FILE_BYTES + 1}) { + List members = new ArrayList<>(basicMembers()); + members.set(0, fixture("job.json", fileJob("page.html") + .replace("\"version\":1", "\"version\":1,\"prepareScript\":\"prepare.js\""))); + members.add(new Fixture("prepare.js", new byte[size], 0100644)); + if (size == Main.MAX_EVAL_FILE_BYTES) { + assertTrue(bundle(deflatedZip(members)).arguments().contains("--eval-file")); + } else { assertCode("JOB_LIMIT", () -> bundle(deflatedZip(members))); } + } + Path scratch = scratch(); + Files.writeString(scratch.resolve("preexisting"), "keep"); + Path source = input(".brewshot.json", utf8(urlJob("png", ""))); + assertCode("JOB_SCRATCH", () -> BrewShotWatchJob.prepare(source, scratch)); + assertEquals("keep", Files.readString(scratch.resolve("preexisting"))); + } + + private BrewShotWatchJob.Prepared json(String content) throws Exception { + return BrewShotWatchJob.prepare(input(".brewshot.json", utf8(content)), scratch()); + } + + private BrewShotWatchJob.Prepared bundle(byte[] bytes) throws Exception { + return BrewShotWatchJob.prepare(input(".brewshot.zip", bytes), scratch()); + } + + private Path input(String suffix, byte[] bytes) throws IOException { + Path path = temporary.resolve("job-" + sequence++ + suffix); + return Files.write(path, bytes); + } + + private Path scratch() throws IOException { + return Files.createDirectory(temporary.resolve("scratch-" + sequence++)); + } + + private static String urlJob(String format, String captureFields) { + return "{\"version\":1,\"source\":{\"url\":\"http://localhost:8080/\"}," + + "\"capture\":{\"format\":\"" + format + "\"" + captureFields + "}}"; + } + + private static String fileJob(String path) { + return "{\"version\":1,\"source\":{\"file\":\"" + path + + "\"},\"capture\":{\"format\":\"png\"}}"; + } + + private static byte[] utf8(String string) { return string.getBytes(StandardCharsets.UTF_8); } + private static Fixture fixture(String name, String content) { + return new Fixture(name, utf8(content), 0100644); + } + private static List basicMembers() { + return List.of(fixture("job.json", fileJob("page.html")), fixture("page.html", "

hello

")); + } + private record Fixture(String name, byte[] content, int unixMode) { } + + /** Minimal stored writer allows intentionally duplicate/hostile member names. */ + private static byte[] storedZip(List members) throws IOException { + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + List offsets = new ArrayList<>(); + List crcs = new ArrayList<>(); + for (Fixture member : members) { + offsets.add(bytes.size()); + CRC32 crc = new CRC32(); + crc.update(member.content); + crcs.add(crc.getValue()); + byte[] name = utf8(member.name); + le32(bytes, 0x04034b50); le16(bytes, 20); le16(bytes, 0x800); le16(bytes, 0); + le16(bytes, 0); le16(bytes, 0); le32(bytes, crc.getValue()); + le32(bytes, member.content.length); le32(bytes, member.content.length); + le16(bytes, name.length); le16(bytes, 0); bytes.write(name); bytes.write(member.content); + } + int start = bytes.size(); + for (int i = 0; i < members.size(); i++) { + Fixture member = members.get(i); + byte[] name = utf8(member.name); + le32(bytes, 0x02014b50); le16(bytes, 0x0314); le16(bytes, 20); + le16(bytes, 0x800); le16(bytes, 0); le16(bytes, 0); le16(bytes, 0); + le32(bytes, crcs.get(i)); le32(bytes, member.content.length); le32(bytes, member.content.length); + le16(bytes, name.length); le16(bytes, 0); le16(bytes, 0); le16(bytes, 0); le16(bytes, 0); + le32(bytes, (long) member.unixMode << 16); le32(bytes, offsets.get(i)); bytes.write(name); + } + int size = bytes.size() - start; + le32(bytes, 0x06054b50); le16(bytes, 0); le16(bytes, 0); + le16(bytes, members.size()); le16(bytes, members.size()); + le32(bytes, size); le32(bytes, start); le16(bytes, 0); + return bytes.toByteArray(); + } + + private static byte[] deflatedZip(List members) throws IOException { + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (ZipOutputStream zip = new ZipOutputStream(bytes, StandardCharsets.UTF_8)) { + for (Fixture member : members) { + zip.putNextEntry(new ZipEntry(member.name)); + zip.write(member.content); + zip.closeEntry(); + } + } + return bytes.toByteArray(); + } + + private static byte[] zipWithExtra(byte[] extra) throws IOException { + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (ZipOutputStream zip = new ZipOutputStream(bytes, StandardCharsets.UTF_8)) { + for (Fixture fixture : basicMembers()) { + ZipEntry entry = new ZipEntry(fixture.name); + entry.setExtra(extra); + zip.putNextEntry(entry); + zip.write(fixture.content); + zip.closeEntry(); + } + } + return bytes.toByteArray(); + } + + private static int firstSignature(byte[] bytes, int signature) { + for (int p = 0; p < bytes.length - 3; p++) { + if ((bytes[p] & 255) == (signature & 255) + && (bytes[p + 1] & 255) == ((signature >>> 8) & 255) + && (bytes[p + 2] & 255) == ((signature >>> 16) & 255) + && (bytes[p + 3] & 255) == ((signature >>> 24) & 255)) { return p; } + } + throw new AssertionError("fixture missing signature"); + } + private static void le16(ByteArrayOutputStream out, int value) { + out.write(value & 255); out.write((value >>> 8) & 255); + } + private static void le32(ByteArrayOutputStream out, long value) { + le16(out, (int) value); le16(out, (int) (value >>> 16)); + } + private static void put16(byte[] bytes, int offset, int value) { + bytes[offset] = (byte) value; bytes[offset + 1] = (byte) (value >>> 8); + } + private static void put32(byte[] bytes, int offset, long value) { + put16(bytes, offset, (int) value); put16(bytes, offset + 2, (int) (value >>> 16)); + } + + @FunctionalInterface private interface Action { void run() throws Exception; } + private static void assertCode(String expected, Action action) { + assertEquals(expected, assertThrows(IOException.class, action::run).getMessage()); + } + private static void assertBoundedFailure(Action action) { + IOException failure = assertThrows(IOException.class, action::run); + assertTrue(failure.getMessage().matches("JOB_[A-Z_]+"), failure.getMessage()); + assertTrue(failure.getMessage().length() < 64); + assertEquals(null, failure.getCause(), "no nested path/URL leak through exception causes"); + } +} diff --git a/src/test/java/com/brewshot/BrewShotWatchWorkerTest.java b/src/test/java/com/brewshot/BrewShotWatchWorkerTest.java new file mode 100644 index 0000000..7559f95 --- /dev/null +++ b/src/test/java/com/brewshot/BrewShotWatchWorkerTest.java @@ -0,0 +1,74 @@ +package com.brewshot; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import static org.junit.jupiter.api.Assertions.*; + +class BrewShotWatchWorkerTest { + @TempDir Path scratch; + + @Test void mainNormalizationCannotPromoteAManifestValueIntoArguments() throws Exception { + assertEquals(List.of("https://example.test/", "--gif=4", + "--gif-element=--out=/tmp/foreign", "--scale=0.5"), + BrewShotWatchCapture.literalArguments(List.of("https://example.test/", + "--gif", "4", "--gif-element", "--out=/tmp/foreign", "--scale", "0.5"))); + boolean[] launcherReached = { false }; + Main.sessionLauncherForTests = (width, height, dpr) -> { + launcherReached[0] = true; + throw new java.io.IOException("scripted stop before browser launch"); + }; + try { + var cli = BrewShotWatchCapture.literalArguments(List.of("https://example.test/", + "--clip-selector", "--out=/a-nonexistent-manifest-value")); + cli.add("-o"); cli.add(scratch.resolve("literal.png").toString()); + assertThrows(java.io.IOException.class, () -> Main.run(cli.toArray(String[]::new))); + assertTrue(launcherReached[0], "a value must not replace the positional URL"); + assertFalse(Files.exists(scratch.resolve("literal.png"))); + } finally { Main.sessionLauncherForTests = null; } + } + + @Test void manifestNumericOptionsAreRejectedByMainBeforeAnyLaunch() throws Exception { + Path input = scratch.resolve("invalid.brewshot.json"); + Files.writeString(input, """ + {"version":1,"source":{"url":"https://example.test/"}, + "capture":{"format":"png","dpr":5}} + """); + var job = BrewShotWatchJob.prepare(input, Files.createDirectory(scratch.resolve("job"))); + var cli = BrewShotWatchCapture.literalArguments(job.arguments()); + cli.add("-o"); cli.add(scratch.resolve("invalid.png").toString()); + Main.sessionLauncherForTests = (width, height, dpr) -> { + throw new AssertionError("Main must reject the invalid option before launch"); + }; + try { + assertEquals(2, Main.run(cli.toArray(String[]::new))); + assertFalse(Files.exists(scratch.resolve("invalid.png"))); + } finally { Main.sessionLauncherForTests = null; } + } + + @Test void subprocessPreservesExitAndContinuesAfterDeadline() throws Exception { + assertEquals(7, BrewShotFolderWorker.runJobProcess( + List.of("/bin/sh", "-c", "exit 7"), 2000)); + long started = System.nanoTime(); + var failure = assertThrows(BrewShotFolderWorker.JobFailure.class, + () -> BrewShotFolderWorker.runJobProcess( + List.of("/bin/sh", "-c", "exec sleep 30"), 100)); + assertTrue((System.nanoTime() - started) / 1_000_000 < 17000, + "deadline plus the bounded shutdown grace must return"); + assertEquals("job-deadline", failure.getMessage()); + assertEquals(0, BrewShotFolderWorker.runJobProcess( + List.of("/bin/sh", "-c", "exit 0"), 2000)); + } + + @Test void rejectsMislabeledAndTruncatedGif() throws Exception { + Path bad = scratch.resolve("bad.gif"); + Files.writeString(bad, "this is a PNG mislabeled as GIF"); + assertThrows(BrewShotFolderWorker.JobFailure.class, + () -> BrewShotFolderWorker.requireGif(bad)); + Files.writeString(bad, "GIF89a"); + assertThrows(BrewShotFolderWorker.JobFailure.class, + () -> BrewShotFolderWorker.requireGif(bad)); + } +}