diff --git a/CHANGELOG.md b/CHANGELOG.md index f0687e97..f485fdce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,9 @@ - LLM input guard (`fuckmark-guard-v1`). `protect()` / `inspect()` / `Guard` / `@Guard().wrap` sanitize strings, lists, and dicts (OpenAI-style `messages` included) before a model call. Default action is strip; `on_findings="refuse"` raises `HiddenTextRefused` without calling the inner function; `report` scans only. Unicode tag characters are decoded to ASCII as `tag_payload` on the receipt (`extract_tag_payload`). CLI: `fuckmark guard` (`--json`, `--refuse`, `--report`, `--receipt`, `--select`). HTTP: `POST /api/guard`. JS: `editors/vscode/guard.js`. This does not detect semantic prompt injection. Default categories are the shared `SECURITY_SCAN_CATEGORIES` also used by `fuckmark lint`. Reference: `docs/guard.md`. - Hidden-scan `format` classification uses the same explicit Cf ranges as `editors/vscode/scan.js` (including U+13430-U+1343F and U+0890-U+0891) instead of only `unicodedata.category`, so Python 3.11 (Unicode 14) agrees with the JS port and with Python 3.12+ (Unicode 15). - Hidden-scan findings now carry **context**, **severity**, and a one-line **why/remedy**. A bidi override inside an identifier is `critical` (Trojan Source); a ZWJ inside an emoji cluster is `info`. Lint JSON, `--scan` reports, and the VS Code hover/Problems panel all show those fields. Diagnostic rank: `critical` is Error, `high` is Warning. -- Frozen classification spec `fuckmark-hidden-scan-v1` under `specs/` (`protocol.md`, `vectors.json`, `freeze.json`). Conformance vectors are codepoint arrays only; `tests/test_hidden_scan_spec.py` replays them on the Python engine and on `editors/vscode/scan.js`. +- Frozen classification spec `fuckmark-hidden-scan-v1` under `specs/` (`protocol.md`, `vectors.json`, `freeze.json`). Conformance vectors are codepoint arrays only; `tests/test_hidden_scan_spec.py` replays them on the Python engine and on `editors/vscode/scan.js`. Freeze record **v3** adds language-aware comment/string roles: a bidi override in a comment or string is `critical` even next to emoji, `#` is a comment only for `python`, and `//` after `:` is not a comment. +- Language-aware Trojan Source analysis and autofix. `scan_hidden_characters(..., language=)` (and lint, from the file suffix) classifies source as `comment` / `string` / code. `autofix_trojan_source` / `autofixTrojanSource` strip only `bidi_control`. +- Zero-install hidden-Unicode scan page: `docs/scan.html` (packaged as `fuckmark/webui/scan.html`, engine copy `scan.js`). `fuckmark web` serves it at `/scan.html`. After deploy: `mark.q1z.org/scan.html`. Scanning runs in the browser; a local Python `/api/scan` confirm is optional when the CLI server is up. - `fuckmark normalize` (`fuckmark-normalize-v1`): NFC, optional identifier lookalike fold (UTS #39-inspired subset, not a full confusable map), then strip of the security category set, plus a JSON receipt (`input_sha256` / `output_sha256` / `steps` / `report_hash`). `--clean` remains the blunt strip with no receipt. HTTP: `POST /api/normalize`. Python: `normalize_text`, `skeleton_fold`. Reference: `docs/normalize.md`. - VS Code / Cursor: `fuckmark.cleanOnSave` (default off) strips hidden Unicode on save. - Frozen confirmation files, hashes, mix mechanisms, and the published wheel are unchanged. Package version stays 0.4.1. diff --git a/README.md b/README.md index a487f75f..69d417bc 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ Opens a local page at `http://127.0.0.1:8765/mark.html`. Paste text in the brows `fuckmark web` also serves a local Python API. Detect and strip POST to `/api/remove-marks` and run the same engine as `fuckmark --detect`. Opening `mark.html` with `file://` or the static website has no Python process, so those paths keep the in-browser fallback. - Product paste UI source: [`docs/mark.html`](docs/mark.html) (packaged as `fuckmark/webui/mark.html`) — after deploy: [mark.q1z.org/mark.html](https://mark.q1z.org/mark.html) +- Hidden-Unicode scan (no install): [`docs/scan.html`](docs/scan.html) — `file://` works; `fuckmark web` serves `/scan.html`; after deploy: [mark.q1z.org/scan.html](https://mark.q1z.org/scan.html) - Research demo: [`docs/demo.html`](docs/demo.html) — `file://` works; after deploy: [mark.q1z.org/demo.html](https://mark.q1z.org/demo.html) `mark.html` runs a local closed-set FuckMark scan, strips approved insertions when found, and on a miss shows an English no-watermark card with contact to `Fhelp@q1z.org`. Under `fuckmark web` that scan is the Python detector; the JS copy is the fallback. @@ -132,7 +133,7 @@ printf 'if (x != \u202eadmin\u202c) {\n' | .venv/bin/fuckmark --scan .venv/bin/fuckmark --clean --file suspect.txt -o clean.txt ``` -`--scan` prints a human report by default, a machine line with `-q`, and a `fuckmark-scan ...` status line to stderr with `--status`. Findings include context (`identifier` / `emoji` / `string` / `prose`) and severity (`critical` / `high` / `medium` / `info`): a bidi override inside an identifier is critical; a ZWJ inside an emoji cluster is info. `--clean` removes every flagged category (including FuckMark's own carriers, so `--clean` reverses a mix back to the visible text) and reports how many characters it removed. The same engine is available in the browser tool via `POST /api/scan`, and in Python through `scan_hidden_characters`, `clean_hidden_characters`, and `classify_hidden_codepoint`. Frozen spec: [`specs/fuckmark-hidden-scan-v1.protocol.md`](specs/fuckmark-hidden-scan-v1.protocol.md). +`--scan` prints a human report by default, a machine line with `-q`, and a `fuckmark-scan ...` status line to stderr with `--status`. Findings include context (`identifier` / `comment` / `string` / `emoji` / `prose`) and severity (`critical` / `high` / `medium` / `info`): a bidi override inside an identifier, comment, or string is critical (Trojan Source); a ZWJ inside an emoji cluster is info. Lint infers comment syntax from the file suffix (`javascript`, `python`, `sql`, `html`). `--clean` removes every flagged category (including FuckMark's own carriers, so `--clean` reverses a mix back to the visible text) and reports how many characters it removed. `autofix_trojan_source` strips only bidirectional controls. The same engine is in the browser at [`docs/scan.html`](docs/scan.html) (`fuckmark web` → `/scan.html`), via `POST /api/scan`, and in Python through `scan_hidden_characters`, `clean_hidden_characters`, and `classify_hidden_codepoint`. Frozen spec: [`specs/fuckmark-hidden-scan-v1.protocol.md`](specs/fuckmark-hidden-scan-v1.protocol.md). Whitespace tab, newline, carriage return, and space are never flagged, and ordinary combining accents (`Mn`) are left alone. `--clean` strips emoji zero-width joiners and variation selectors as well, so pass a category subset to `clean_hidden_characters(...)` in Python if you need to keep emoji sequences intact. @@ -176,7 +177,7 @@ Binary, non-UTF-8, oversized, and vendored/VCS paths are skipped; `--exclude GLO ## See it in your editor (VS Code / Cursor) -The [`editors/vscode`](editors/vscode) extension reveals hidden Unicode inline as you read code: a red box and a visible `‹U+202E›` badge on every hidden character, hover text with why it matters, diagnostics in the Problems panel (`critical` as Error), a status-bar count, one-click clean, and optional clean-on-save (`fuckmark.cleanOnSave`). It is zero-dependency plain JavaScript with no build step, and its scanner is a faithful port of `fuckmark-hidden-scan-v1` — pinned to the Python engine across every Unicode codepoint by [`tests/test_vscode_scanner_parity.py`](tests/test_vscode_scanner_parity.py), so the editor and the CLI agree exactly. Load it with `code --extensionDevelopmentPath=editors/vscode` or open the folder and press `F5`. +The [`editors/vscode`](editors/vscode) extension reveals hidden Unicode inline as you read code: a red box and a visible `‹U+202E›` badge on every hidden character, hover text with why it matters, diagnostics in the Problems panel (`critical` as Error, including Trojan Source in comments and strings), a status-bar count, one-click clean, and optional clean-on-save (`fuckmark.cleanOnSave`). It is zero-dependency plain JavaScript with no build step, and its scanner is a faithful port of `fuckmark-hidden-scan-v1` — pinned to the Python engine across every Unicode codepoint by [`tests/test_vscode_scanner_parity.py`](tests/test_vscode_scanner_parity.py), so the editor and the CLI agree exactly. Load it with `code --extensionDevelopmentPath=editors/vscode` or open the folder and press `F5`. ## Guard model input (LLM prompt-injection smuggling) diff --git a/docs/cli.md b/docs/cli.md index 98f3add5..d7919a20 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -116,9 +116,9 @@ Existing files are read as UTF-8 bytes with no newline conversion. LF, CRLF, CR, `--scan` and `--clean` are the defensive inverse of the mix. The scan is general, not FuckMark-only. Flagged categories: `bidi_control` (Trojan Source, CVE-2021-42574), `zero_width`, `variation_selector`, `tag` (hidden-text / prompt-injection smuggling), `enclosing_mark`, `line_separator`, `deprecated` (interlinear annotation and deprecated format controls), `format` (other `Cf`), `control` (C0/C1), `private_use`, `noncharacter`, and `surrogate`. Tab, newline, carriage return, and space are never flagged; ordinary combining accents (`Mn`) are left alone. -Findings include context (`identifier`, `emoji`, `string`, `prose`) and severity (`critical`, `high`, `medium`, `info`). A bidi override next to an identifier is `critical`; a ZWJ inside an emoji cluster is `info`. The frozen table is [`specs/fuckmark-hidden-scan-v1.protocol.md`](../specs/fuckmark-hidden-scan-v1.protocol.md). +Findings include context (`identifier`, `comment`, `string`, `emoji`, `prose`) and severity (`critical`, `high`, `medium`, `info`). A bidi override in an identifier, comment, or string is `critical` (Trojan Source); a ZWJ inside an emoji cluster is `info`. Comment syntax follows `language` (`auto` treats `//` and `/* */` as comments; `python` also treats `#`; `sql` treats `--`; `html` treats ``). The frozen table is [`specs/fuckmark-hidden-scan-v1.protocol.md`](../specs/fuckmark-hidden-scan-v1.protocol.md). -`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. The browser tool exposes the same engine at `POST /api/scan`. +`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. `autofix_trojan_source` strips only `bidi_control`. The browser scan page is `docs/scan.html` (`fuckmark web` serves `/scan.html`); the API is `POST /api/scan` with optional `"language"`. ### `fuckmark normalize` @@ -143,7 +143,7 @@ fuckmark guard --refuse --receipt --json < messages.json ### `fuckmark web` -Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`. +Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`. Hidden-Unicode reveal is `/scan.html` (same engine as `--scan`; zero-install copy at `docs/scan.html`). ```text fuckmark web diff --git a/docs/lint.md b/docs/lint.md index dc46f825..04a32c32 100644 --- a/docs/lint.md +++ b/docs/lint.md @@ -34,7 +34,9 @@ Pass `--select all` to include `variation_selector`, `enclosing_mark`, `line_separator`, `deprecated`, `format`, and `private_use`, or pass an explicit comma-separated subset. See [`cli.md`](cli.md) for the full category list. JSON locations include `context`, `severity`, `why`, and `remedy` from -`fuckmark-hidden-scan-v1`. +`fuckmark-hidden-scan-v1`. Context uses the file suffix: `.js`/`.ts` get +JavaScript comments (`//`, `/* */`), `.py` gets `#` comments, `.sql` gets `--`, +`.html` gets ``. A bidi override in a comment or string is `critical`. ### What is skipped diff --git a/docs/scan.html b/docs/scan.html new file mode 100644 index 00000000..0b007977 --- /dev/null +++ b/docs/scan.html @@ -0,0 +1,606 @@ + + + + + +FuckMark — Hidden Unicode scan + + + + + + + + + + +
+
+ +

See the bytes.

+

Paste source or a prompt. This page reveals Trojan Source bidi overrides, zero-width characters, and Unicode tag smuggling without sending the text anywhere.

+ + +
+ + + + Engine: browser +
+ + + +
+
+ + +
+ +

Paste text to scan.

+
+ +

+ Scanning is fuckmark-hidden-scan-v1. Copy uses the security category set + (bidi_control, zero_width, tag, control, + noncharacter, surrogate). Fix Trojan Source strips only + bidirectional controls. Nothing is uploaded from a static page or file://. + Under fuckmark web an optional local Python API can confirm the same verdicts. + Contact Fhelp@q1z.org. +

+
+
+ + + + diff --git a/docs/scan.js b/docs/scan.js new file mode 100644 index 00000000..f28c001f --- /dev/null +++ b/docs/scan.js @@ -0,0 +1,569 @@ +// Faithful JavaScript port of fuckmark-hidden-scan-v1 (fuckmark/product/scan.py). +// Kept in lockstep with the Python engine by tests/test_vscode_scanner_parity.py. + +"use strict"; + +const SCAN_ALGORITHM_VERSION = "fuckmark-hidden-scan-v1"; + +const CATEGORY_BIDI_CONTROL = "bidi_control"; +const CATEGORY_ZERO_WIDTH = "zero_width"; +const CATEGORY_VARIATION_SELECTOR = "variation_selector"; +const CATEGORY_TAG = "tag"; +const CATEGORY_ENCLOSING_MARK = "enclosing_mark"; +const CATEGORY_LINE_SEPARATOR = "line_separator"; +const CATEGORY_DEPRECATED = "deprecated"; +const CATEGORY_FORMAT = "format"; +const CATEGORY_CONTROL = "control"; +const CATEGORY_PRIVATE_USE = "private_use"; +const CATEGORY_NONCHARACTER = "noncharacter"; +const CATEGORY_SURROGATE = "surrogate"; + +const SCAN_CATEGORIES = [ + CATEGORY_BIDI_CONTROL, + CATEGORY_ZERO_WIDTH, + CATEGORY_VARIATION_SELECTOR, + CATEGORY_TAG, + CATEGORY_ENCLOSING_MARK, + CATEGORY_LINE_SEPARATOR, + CATEGORY_DEPRECATED, + CATEGORY_FORMAT, + CATEGORY_CONTROL, + CATEGORY_PRIVATE_USE, + CATEGORY_NONCHARACTER, + CATEGORY_SURROGATE, +]; + +const CATEGORY_DESCRIPTIONS = { + [CATEGORY_BIDI_CONTROL]: "bidirectional override/isolate (Trojan Source reordering)", + [CATEGORY_ZERO_WIDTH]: "zero-width or invisible spacing character", + [CATEGORY_VARIATION_SELECTOR]: "variation selector (glyph/steganography carrier)", + [CATEGORY_TAG]: "Unicode tag character (hidden text / prompt-injection smuggling)", + [CATEGORY_ENCLOSING_MARK]: "enclosing combining mark (alters surrounding glyph)", + [CATEGORY_LINE_SEPARATOR]: "line/paragraph separator (breaks parsers, invisible)", + [CATEGORY_DEPRECATED]: "deprecated format control or interlinear annotation", + [CATEGORY_FORMAT]: "general Unicode format control (Cf)", + [CATEGORY_CONTROL]: "C0/C1 control character", + [CATEGORY_PRIVATE_USE]: "private-use codepoint (renderer-defined)", + [CATEGORY_NONCHARACTER]: "Unicode noncharacter or unassigned codepoint", + [CATEGORY_SURROGATE]: "lone surrogate codepoint", +}; + +const DEFAULT_SECURITY_CATEGORIES = [ + CATEGORY_BIDI_CONTROL, + CATEGORY_ZERO_WIDTH, + CATEGORY_TAG, + CATEGORY_CONTROL, + CATEGORY_NONCHARACTER, + CATEGORY_SURROGATE, +]; + +const TROJAN_SOURCE_CATEGORIES = [CATEGORY_BIDI_CONTROL]; + +const ALLOWED_WHITESPACE = new Set([0x09, 0x0a, 0x0d, 0x20]); + +function inRange(cp, lo, hi) { + return cp >= lo && cp <= hi; +} + +const BIDI_CONTROL = new Set([0x061c, 0x200e, 0x200f]); +for (let cp = 0x202a; cp <= 0x202e; cp += 1) BIDI_CONTROL.add(cp); +for (let cp = 0x2066; cp <= 0x2069; cp += 1) BIDI_CONTROL.add(cp); + +const DEPRECATED = new Set([0xfff9, 0xfffa, 0xfffb]); +for (let cp = 0x206a; cp <= 0x206f; cp += 1) DEPRECATED.add(cp); + +const ZERO_WIDTH = new Set([ + 0x00ad, 0x034f, 0x115f, 0x1160, 0x17b4, 0x17b5, 0x180e, 0x200b, 0x200c, + 0x200d, 0x2060, 0x2061, 0x2062, 0x2063, 0x2064, 0x3164, 0xfeff, 0xffa0, +]); + +function isVariationSelector(cp) { + return inRange(cp, 0xfe00, 0xfe0f) || inRange(cp, 0xe0100, 0xe01ef); +} + +function isTag(cp) { + return inRange(cp, 0xe0000, 0xe007f); +} + +function isNoncharacter(cp) { + if (inRange(cp, 0xfdd0, 0xfdef)) return true; + const low = cp & 0xffff; + return low === 0xfffe || low === 0xffff; +} + +function isEnclosingMark(cp) { + // Unicode general category Me. Enumerated to avoid a runtime Unicode DB. + return ( + inRange(cp, 0x0488, 0x0489) || + cp === 0x1abe || + inRange(cp, 0x20dd, 0x20e0) || + inRange(cp, 0x20e2, 0x20e4) || + inRange(cp, 0xa670, 0xa672) + ); +} + +function isControl(cp) { + return inRange(cp, 0x00, 0x1f) || inRange(cp, 0x7f, 0x9f); +} + +function isLineSeparator(cp) { + return cp === 0x2028 || cp === 0x2029; +} + +// General category Cf ranges (Unicode 15-era), excluding codepoints already +// classified above. Mirrors what unicodedata.category returns as "Cf". +const FORMAT_RANGES = [ + [0x0600, 0x0605], + [0x06dd, 0x06dd], + [0x070f, 0x070f], + [0x0890, 0x0891], + [0x08e2, 0x08e2], + [0x110bd, 0x110bd], + [0x110cd, 0x110cd], + [0x13430, 0x1343f], + [0x1bca0, 0x1bca3], + [0x1d173, 0x1d17a], +]; + +function isFormat(cp) { + for (const [lo, hi] of FORMAT_RANGES) { + if (inRange(cp, lo, hi)) return true; + } + return false; +} + +function isPrivateUse(cp) { + return ( + inRange(cp, 0xe000, 0xf8ff) || + inRange(cp, 0xf0000, 0xffffd) || + inRange(cp, 0x100000, 0x10fffd) + ); +} + +function classify(cp) { + if (ALLOWED_WHITESPACE.has(cp)) return null; + if (BIDI_CONTROL.has(cp)) return CATEGORY_BIDI_CONTROL; + if (DEPRECATED.has(cp)) return CATEGORY_DEPRECATED; + if (ZERO_WIDTH.has(cp)) return CATEGORY_ZERO_WIDTH; + if (isVariationSelector(cp)) return CATEGORY_VARIATION_SELECTOR; + if (isTag(cp)) return CATEGORY_TAG; + if (isNoncharacter(cp)) return CATEGORY_NONCHARACTER; + if (cp >= 0xd800 && cp <= 0xdfff) return CATEGORY_SURROGATE; + if (isEnclosingMark(cp)) return CATEGORY_ENCLOSING_MARK; + if (isLineSeparator(cp)) return CATEGORY_LINE_SEPARATOR; + if (isControl(cp)) return CATEGORY_CONTROL; + if (isFormat(cp)) return CATEGORY_FORMAT; + if (isPrivateUse(cp)) return CATEGORY_PRIVATE_USE; + return null; +} + +function codepointToken(cp) { + return "U+" + cp.toString(16).toUpperCase().padStart(4, "0"); +} + +function isEmojiish(cp) { + if (cp < 0) return false; + if (cp >= 0x1f1e6 && cp <= 0x1f1ff) return true; + if (cp >= 0x1f000 && cp <= 0x1faff) return true; + if (cp >= 0x2600 && cp <= 0x27bf) return true; + return cp === 0x200d || [0x00a9, 0x00ae, 0x203c, 0x2049, 0x2122, 0x2139, 0x3030, 0x303d, 0x3297, 0x3299].includes(cp); +} + +function isIdentChar(ch) { + return ch === "_" || /\p{L}|\p{N}/u.test(ch); +} + +function neighborBefore(text, offset) { + if (offset <= 0) return { cp: -1, ch: "" }; + if (offset >= 2) { + const cp = text.codePointAt(offset - 2); + if (cp > 0xffff) return { cp, ch: String.fromCodePoint(cp) }; + } + const cp = text.codePointAt(offset - 1); + return { cp, ch: String.fromCodePoint(cp) }; +} + +function neighborAfter(text, offset, length) { + const nextIndex = offset + length; + if (nextIndex >= text.length) return { cp: -1, ch: "" }; + const cp = text.codePointAt(nextIndex); + return { cp, ch: String.fromCodePoint(cp) }; +} + +const LANGUAGE_ALIASES = { + auto: "auto", + javascript: "javascript", + js: "javascript", + ts: "javascript", + typescript: "javascript", + jsx: "javascript", + tsx: "javascript", + c: "c", + h: "c", + cc: "c", + cpp: "c", + cxx: "c", + java: "c", + go: "c", + rs: "c", + rust: "c", + cs: "c", + css: "c", + jsonc: "c", + python: "python", + py: "python", + pyi: "python", + hash: "python", + sh: "python", + bash: "python", + zsh: "python", + shell: "python", + shellscript: "python", + yaml: "python", + yml: "python", + rb: "python", + ruby: "python", + toml: "python", + html: "html", + htm: "html", + xml: "html", + sql: "sql", +}; + +function normalizeLanguage(language) { + if (!language) return "auto"; + const key = String(language).trim().toLowerCase(); + return LANGUAGE_ALIASES[key] || "auto"; +} + +function slashComments(language) { + return language === "auto" || language === "javascript" || language === "c" || language === "sql"; +} + +function blockComments(language) { + return slashComments(language); +} + +function hashComments(language) { + return language === "python"; +} + +function sqlLineComments(language) { + return language === "sql"; +} + +function htmlComments(language) { + return language === "html"; +} + +function sourceRoles(text, language) { + const lang = normalizeLanguage(language); + const roles = new Array(text.length).fill("code"); + let index = 0; + let inString = ""; + let escape = false; + while (index < text.length) { + const ch = text[index]; + if (inString) { + roles[index] = "string"; + if (escape) { + escape = false; + index += 1; + continue; + } + if (ch === "\\" && index + 1 < text.length) { + escape = true; + index += 1; + continue; + } + if (ch === inString) inString = ""; + index += 1; + continue; + } + const pair = text.slice(index, index + 2); + if (slashComments(lang) && pair === "//" && !(index > 0 && text[index - 1] === ":")) { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (sqlLineComments(lang) && pair === "--") { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (blockComments(lang) && pair === "/*") { + const close = text.indexOf("*/", index + 2); + const stop = close >= 0 ? close + 2 : text.length; + for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment"; + index = stop; + continue; + } + if (hashComments(lang) && ch === "#") { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (htmlComments(lang) && text.slice(index, index + 4) === "", index + 4); + const stop = close >= 0 ? close + 3 : text.length; + for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment"; + index = stop; + continue; + } + if (ch === '"' || ch === "'" || ch === "`") { + inString = ch; + roles[index] = "string"; + index += 1; + continue; + } + index += 1; + } + return roles; +} + +function classifyContext(text, offset, role, category) { + const length = text.codePointAt(offset) > 0xffff ? 2 : 1; + const prev = neighborBefore(text, offset); + const next = neighborAfter(text, offset, length); + if ((role === "comment" || role === "string") && category === CATEGORY_BIDI_CONTROL) { + return role; + } + if ( + isEmojiish(prev.cp) || + isEmojiish(next.cp) || + (prev.cp >= 0xfe00 && prev.cp <= 0xfe0f) || + (next.cp >= 0xfe00 && next.cp <= 0xfe0f) + ) { + return "emoji"; + } + if (role === "comment") return "comment"; + if (role === "string") return "string"; + if (isIdentChar(prev.ch) || isIdentChar(next.ch)) return "identifier"; + return "prose"; +} + +function scoreSeverity(category, context) { + if (category === CATEGORY_TAG) return "critical"; + if (category === CATEGORY_BIDI_CONTROL) { + if (context === "identifier" || context === "comment" || context === "string") return "critical"; + return "high"; + } + if (category === CATEGORY_ZERO_WIDTH) { + if (context === "emoji") return "info"; + if (context === "identifier") return "high"; + return "medium"; + } + if (category === CATEGORY_VARIATION_SELECTOR) return context === "emoji" ? "info" : "medium"; + if (category === CATEGORY_CONTROL || category === CATEGORY_NONCHARACTER || category === CATEGORY_SURROGATE) { + return "high"; + } + return "medium"; +} + +function explainFinding(category, context, severity) { + if (category === CATEGORY_BIDI_CONTROL && context === "identifier") { + return { + why: "Bidirectional override sits inside an identifier, so the glyphs can read differently than the bytes (Trojan Source).", + remedy: "Strip the bidi control and keep the identifier left-to-right.", + }; + } + if (category === CATEGORY_BIDI_CONTROL && context === "comment") { + return { + why: "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).", + remedy: "Strip the bidi control from the comment.", + }; + } + if (category === CATEGORY_BIDI_CONTROL && context === "string") { + return { + why: "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).", + remedy: "Strip the bidi control from the string.", + }; + } + if (category === CATEGORY_BIDI_CONTROL) { + return { + why: "Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).", + remedy: "Strip U+202A-U+202E / U+2066-U+2069 and rewrite the text left-to-right.", + }; + } + if (category === CATEGORY_TAG) { + return { + why: "Unicode tag characters encode a second ASCII string that models read and humans do not.", + remedy: "Strip U+E0020-U+E007F; inspect tag_payload for the smuggled text.", + }; + } + if (category === CATEGORY_ZERO_WIDTH && context === "emoji") { + return { + why: "Zero-width joiner or invisible mark inside an emoji cluster; usually a legitimate emoji sequence.", + remedy: "Leave emoji ZWJ sequences unless you are sanitizing for a security boundary.", + }; + } + if (category === CATEGORY_ZERO_WIDTH && context === "identifier") { + return { + why: "Zero-width character splits an identifier, breaking search and some compilers while looking unchanged.", + remedy: "Strip the zero-width character from the identifier.", + }; + } + if (category === CATEGORY_ZERO_WIDTH) { + return { + why: "Invisible spacing or joining character that changes the byte stream without changing the glyphs.", + remedy: "Strip the zero-width character.", + }; + } + if (category === CATEGORY_VARIATION_SELECTOR && context === "emoji") { + return { + why: "Variation selector tunes an emoji glyph; usually benign.", + remedy: "Keep emoji variation selectors unless you are stripping all hidden marks.", + }; + } + if (severity === "high") { + return { + why: "Hidden or non-text codepoint that should not appear in ordinary source or prompts.", + remedy: "Strip the character.", + }; + } + return { + why: CATEGORY_DESCRIPTIONS[category] || "Hidden or format codepoint that is invisible or renderer-defined.", + remedy: "Strip the character if this text crosses a trust boundary.", + }; +} + +function scanText(text, categories, language) { + const selected = categories ? new Set(categories) : null; + const roles = sourceRoles(text, language); + const counts = {}; + const findings = []; + let total = 0; + let offset = 0; + while (offset < text.length) { + const cp = text.codePointAt(offset); + const length = cp > 0xffff ? 2 : 1; + const category = classify(cp); + if (category !== null && (selected === null || selected.has(category))) { + total += 1; + counts[category] = (counts[category] || 0) + 1; + const context = classifyContext(text, offset, roles[offset] || "code", category); + const severity = scoreSeverity(category, context); + const explained = explainFinding(category, context, severity); + findings.push({ + offset, + length, + codepoint: cp, + category, + context, + severity, + why: explained.why, + remedy: explained.remedy, + }); + } + offset += length; + } + return { total, counts, findings }; +} + +function cleanText(text, categories) { + const selected = categories ? new Set(categories) : null; + let out = ""; + let removed = 0; + let offset = 0; + while (offset < text.length) { + const cp = text.codePointAt(offset); + const length = cp > 0xffff ? 2 : 1; + const category = classify(cp); + if (category !== null && (selected === null || selected.has(category))) { + removed += 1; + } else { + out += text.slice(offset, offset + length); + } + offset += length; + } + return { cleaned: out, removed }; +} + +function autofixTrojanSource(text) { + return cleanText(text, TROJAN_SOURCE_CATEGORIES); +} + +function languageFromDocumentId(languageId) { + return normalizeLanguage(languageId); +} + +const LANGUAGE_BY_SUFFIX = { + ".js": "javascript", + ".mjs": "javascript", + ".cjs": "javascript", + ".ts": "javascript", + ".jsx": "javascript", + ".tsx": "javascript", + ".c": "c", + ".h": "c", + ".hh": "c", + ".cc": "c", + ".cpp": "c", + ".cxx": "c", + ".java": "c", + ".go": "c", + ".rs": "c", + ".cs": "c", + ".css": "c", + ".py": "python", + ".pyi": "python", + ".sh": "python", + ".bash": "python", + ".zsh": "python", + ".yaml": "python", + ".yml": "python", + ".rb": "python", + ".toml": "python", + ".html": "html", + ".htm": "html", + ".xml": "html", + ".sql": "sql", +}; + +function languageFromPath(path) { + const name = String(path || ""); + const slash = Math.max(name.lastIndexOf("/"), name.lastIndexOf("\\")); + const base = slash >= 0 ? name.slice(slash + 1) : name; + const dot = base.lastIndexOf("."); + if (dot <= 0) return "auto"; + const suffix = base.slice(dot).toLowerCase(); + return LANGUAGE_BY_SUFFIX[suffix] || "auto"; +} + +const FuckMarkScan = { + SCAN_ALGORITHM_VERSION, + SCAN_CATEGORIES, + CATEGORY_DESCRIPTIONS, + DEFAULT_SECURITY_CATEGORIES, + TROJAN_SOURCE_CATEGORIES, + classify, + classifyContext, + scoreSeverity, + normalizeLanguage, + languageFromDocumentId, + languageFromPath, + sourceRoles, + codepointToken, + scanText, + cleanText, + autofixTrojanSource, +}; + +if (typeof module !== "undefined" && module.exports) { + module.exports = FuckMarkScan; +} +if (typeof globalThis !== "undefined") { + globalThis.FuckMarkScan = FuckMarkScan; +} diff --git a/docs/website.md b/docs/website.md index 21a15aa6..51f4560c 100644 --- a/docs/website.md +++ b/docs/website.md @@ -11,6 +11,7 @@ Deploy these static files (repository CI cannot push them; copy after each relea | Repo file | Public URL | | --- | --- | | [`mark.html`](mark.html) | `https://mark.q1z.org/mark.html` (browser tool) | +| [`scan.html`](scan.html) + [`scan.js`](scan.js) | `https://mark.q1z.org/scan.html` (hidden-Unicode reveal; deploy both files) | | [`rec.html`](rec.html) | gate that waits for `go.txt`, then opens `mark.html?demo=1` | | [`demo.html`](demo.html) | `https://mark.q1z.org/demo.html` (research / CLI samples) | @@ -27,10 +28,16 @@ Local beginners can run `fuckmark web` after install. That serves the same `mark - `GET /api/health` reports `backend: "python"` - `POST /api/remove-marks` with `{ "text": "..." }` calls `detect_fuckmark_insertions` and `project_visible_v1` -- Local-only extras on `fuckmark web`: `POST /api/scan`, `POST /api/guard`, `POST /api/normalize` +- Local-only extras on `fuckmark web`: `POST /api/scan`, `POST /api/guard`, `POST /api/normalize`, and `/scan.html` The page uses that API when the local server is up. Static `https://mark.q1z.org/mark.html` and `file://` have no Python process, so those deploys keep the in-browser scan. +### `scan.html` (hidden-Unicode reveal) + +Static, JS-first. Paste text or open a file; the page classifies hidden characters with `fuckmark-hidden-scan-v1` (`scan.js`, the same port as the editor extension). Language (`auto`, JavaScript/C-like, Python, SQL, HTML) selects comment syntax so Trojan Source commenting-out and stretched-string encodings rank `critical`. Copy uses the security category set. **Fix Trojan Source** strips only bidirectional controls. + +Deploy `scan.html` and `scan.js` next to each other. `file://` works. Under `fuckmark web`, an optional checkbox uses `POST /api/scan` on the local Python engine; text still does not leave the machine. Do not embed raw hidden characters in the HTML file; examples are built with `String.fromCodePoint`. + ### `demo.html` (research walkthrough) Keep it a static file that works from `file://` with baked CLI samples. The live page must show: @@ -43,7 +50,7 @@ Keep it a static file that works from `file://` with baked CLI samples. The live If none are found, the demo offers contact at `Fhelp@q1z.org`. Do not present demo numbers as a guarantee for arbitrary user text. -Homepage copy should link `mark.html` (tool) and `demo.html` (evidence) before install instructions. +Homepage copy should link `mark.html` (tool), `scan.html` (hidden Unicode), and `demo.html` (evidence) before install instructions. ## What the product does diff --git a/editors/vscode/README.md b/editors/vscode/README.md index bc50774c..f59878dd 100644 --- a/editors/vscode/README.md +++ b/editors/vscode/README.md @@ -17,8 +17,9 @@ It flags the classes that read as ordinary text but are not: - **Inline reveal:** every hidden character gets a red box and a visible `‹U+202E›` badge, even when the character itself paints nothing. - **Problems panel:** each hit is a diagnostic. `critical` (Trojan Source in - an identifier, tag smuggling) is an Error; `high` and other security classes - are warnings; the rest are info. Hover text states why the character matters. + an identifier, comment, or string; tag smuggling) is an Error; `high` and + other security classes are warnings; the rest are info. Hover text states why + the character matters. Comment syntax follows the editor language id. - **Status bar:** a live hidden-character count. Click it to clean the file. - **Commands** (Command Palette): - `FuckMark: Clean hidden Unicode in file` diff --git a/editors/vscode/extension.js b/editors/vscode/extension.js index 83f4bd61..48d57c45 100644 --- a/editors/vscode/extension.js +++ b/editors/vscode/extension.js @@ -8,6 +8,7 @@ const { codepointToken, scanText, cleanText, + languageFromDocumentId, } = require("./scan.js"); let decorationType; @@ -113,7 +114,7 @@ function refresh(editor) { editor.setDecorations(decorationType, []); return { total: 0 }; } - const result = scanText(document.getText()); + const result = scanText(document.getText(), null, languageFromDocumentId(document.languageId)); diagnostics.set(document.uri, buildDiagnostics(document, result.findings)); editor.setDecorations(decorationType, revealEnabled ? buildDecorations(document, result.findings) : []); return result; diff --git a/editors/vscode/scan.js b/editors/vscode/scan.js index c2351b96..f28c001f 100644 --- a/editors/vscode/scan.js +++ b/editors/vscode/scan.js @@ -57,6 +57,8 @@ const DEFAULT_SECURITY_CATEGORIES = [ CATEGORY_SURROGATE, ]; +const TROJAN_SOURCE_CATEGORIES = [CATEGORY_BIDI_CONTROL]; + const ALLOWED_WHITESPACE = new Set([0x09, 0x0a, 0x0d, 0x20]); function inRange(cp, lo, hi) { @@ -188,10 +190,156 @@ function neighborAfter(text, offset, length) { return { cp, ch: String.fromCodePoint(cp) }; } -function classifyContext(text, offset, inString) { +const LANGUAGE_ALIASES = { + auto: "auto", + javascript: "javascript", + js: "javascript", + ts: "javascript", + typescript: "javascript", + jsx: "javascript", + tsx: "javascript", + c: "c", + h: "c", + cc: "c", + cpp: "c", + cxx: "c", + java: "c", + go: "c", + rs: "c", + rust: "c", + cs: "c", + css: "c", + jsonc: "c", + python: "python", + py: "python", + pyi: "python", + hash: "python", + sh: "python", + bash: "python", + zsh: "python", + shell: "python", + shellscript: "python", + yaml: "python", + yml: "python", + rb: "python", + ruby: "python", + toml: "python", + html: "html", + htm: "html", + xml: "html", + sql: "sql", +}; + +function normalizeLanguage(language) { + if (!language) return "auto"; + const key = String(language).trim().toLowerCase(); + return LANGUAGE_ALIASES[key] || "auto"; +} + +function slashComments(language) { + return language === "auto" || language === "javascript" || language === "c" || language === "sql"; +} + +function blockComments(language) { + return slashComments(language); +} + +function hashComments(language) { + return language === "python"; +} + +function sqlLineComments(language) { + return language === "sql"; +} + +function htmlComments(language) { + return language === "html"; +} + +function sourceRoles(text, language) { + const lang = normalizeLanguage(language); + const roles = new Array(text.length).fill("code"); + let index = 0; + let inString = ""; + let escape = false; + while (index < text.length) { + const ch = text[index]; + if (inString) { + roles[index] = "string"; + if (escape) { + escape = false; + index += 1; + continue; + } + if (ch === "\\" && index + 1 < text.length) { + escape = true; + index += 1; + continue; + } + if (ch === inString) inString = ""; + index += 1; + continue; + } + const pair = text.slice(index, index + 2); + if (slashComments(lang) && pair === "//" && !(index > 0 && text[index - 1] === ":")) { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (sqlLineComments(lang) && pair === "--") { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (blockComments(lang) && pair === "/*") { + const close = text.indexOf("*/", index + 2); + const stop = close >= 0 ? close + 2 : text.length; + for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment"; + index = stop; + continue; + } + if (hashComments(lang) && ch === "#") { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (htmlComments(lang) && text.slice(index, index + 4) === "", index + 4); + const stop = close >= 0 ? close + 3 : text.length; + for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment"; + index = stop; + continue; + } + if (ch === '"' || ch === "'" || ch === "`") { + inString = ch; + roles[index] = "string"; + index += 1; + continue; + } + index += 1; + } + return roles; +} + +function classifyContext(text, offset, role, category) { const length = text.codePointAt(offset) > 0xffff ? 2 : 1; const prev = neighborBefore(text, offset); const next = neighborAfter(text, offset, length); + if ((role === "comment" || role === "string") && category === CATEGORY_BIDI_CONTROL) { + return role; + } if ( isEmojiish(prev.cp) || isEmojiish(next.cp) || @@ -200,14 +348,18 @@ function classifyContext(text, offset, inString) { ) { return "emoji"; } + if (role === "comment") return "comment"; + if (role === "string") return "string"; if (isIdentChar(prev.ch) || isIdentChar(next.ch)) return "identifier"; - if (inString) return "string"; return "prose"; } function scoreSeverity(category, context) { if (category === CATEGORY_TAG) return "critical"; - if (category === CATEGORY_BIDI_CONTROL) return context === "identifier" ? "critical" : "high"; + if (category === CATEGORY_BIDI_CONTROL) { + if (context === "identifier" || context === "comment" || context === "string") return "critical"; + return "high"; + } if (category === CATEGORY_ZERO_WIDTH) { if (context === "emoji") return "info"; if (context === "identifier") return "high"; @@ -227,6 +379,18 @@ function explainFinding(category, context, severity) { remedy: "Strip the bidi control and keep the identifier left-to-right.", }; } + if (category === CATEGORY_BIDI_CONTROL && context === "comment") { + return { + why: "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).", + remedy: "Strip the bidi control from the comment.", + }; + } + if (category === CATEGORY_BIDI_CONTROL && context === "string") { + return { + why: "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).", + remedy: "Strip the bidi control from the string.", + }; + } if (category === CATEGORY_BIDI_CONTROL) { return { why: "Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).", @@ -275,27 +439,21 @@ function explainFinding(category, context, severity) { }; } -function scanText(text, categories) { +function scanText(text, categories, language) { const selected = categories ? new Set(categories) : null; + const roles = sourceRoles(text, language); const counts = {}; const findings = []; let total = 0; let offset = 0; - let inString = ""; while (offset < text.length) { const cp = text.codePointAt(offset); const length = cp > 0xffff ? 2 : 1; - const ch = text.slice(offset, offset + length); - if (inString) { - if (ch === inString) inString = ""; - } else if (ch === '"' || ch === "'" || ch === "`") { - inString = ch; - } const category = classify(cp); if (category !== null && (selected === null || selected.has(category))) { total += 1; counts[category] = (counts[category] || 0) + 1; - const context = classifyContext(text, offset, Boolean(inString)); + const context = classifyContext(text, offset, roles[offset] || "code", category); const severity = scoreSeverity(category, context); const explained = explainFinding(category, context, severity); findings.push({ @@ -333,15 +491,79 @@ function cleanText(text, categories) { return { cleaned: out, removed }; } -module.exports = { +function autofixTrojanSource(text) { + return cleanText(text, TROJAN_SOURCE_CATEGORIES); +} + +function languageFromDocumentId(languageId) { + return normalizeLanguage(languageId); +} + +const LANGUAGE_BY_SUFFIX = { + ".js": "javascript", + ".mjs": "javascript", + ".cjs": "javascript", + ".ts": "javascript", + ".jsx": "javascript", + ".tsx": "javascript", + ".c": "c", + ".h": "c", + ".hh": "c", + ".cc": "c", + ".cpp": "c", + ".cxx": "c", + ".java": "c", + ".go": "c", + ".rs": "c", + ".cs": "c", + ".css": "c", + ".py": "python", + ".pyi": "python", + ".sh": "python", + ".bash": "python", + ".zsh": "python", + ".yaml": "python", + ".yml": "python", + ".rb": "python", + ".toml": "python", + ".html": "html", + ".htm": "html", + ".xml": "html", + ".sql": "sql", +}; + +function languageFromPath(path) { + const name = String(path || ""); + const slash = Math.max(name.lastIndexOf("/"), name.lastIndexOf("\\")); + const base = slash >= 0 ? name.slice(slash + 1) : name; + const dot = base.lastIndexOf("."); + if (dot <= 0) return "auto"; + const suffix = base.slice(dot).toLowerCase(); + return LANGUAGE_BY_SUFFIX[suffix] || "auto"; +} + +const FuckMarkScan = { SCAN_ALGORITHM_VERSION, SCAN_CATEGORIES, CATEGORY_DESCRIPTIONS, DEFAULT_SECURITY_CATEGORIES, + TROJAN_SOURCE_CATEGORIES, classify, classifyContext, scoreSeverity, + normalizeLanguage, + languageFromDocumentId, + languageFromPath, + sourceRoles, codepointToken, scanText, cleanText, + autofixTrojanSource, }; + +if (typeof module !== "undefined" && module.exports) { + module.exports = FuckMarkScan; +} +if (typeof globalThis !== "undefined") { + globalThis.FuckMarkScan = FuckMarkScan; +} diff --git a/fuckmark/__init__.py b/fuckmark/__init__.py index 62590022..6da7ab00 100644 --- a/fuckmark/__init__.py +++ b/fuckmark/__init__.py @@ -95,6 +95,7 @@ classify_hidden_codepoint, clean_hidden_characters, extract_tag_payload, + autofix_trojan_source, scan_dict, scan_hidden_characters, scan_human_report, @@ -383,6 +384,7 @@ "ScanResult", "classify_hidden_codepoint", "clean_hidden_characters", + "autofix_trojan_source", "extract_tag_payload", "scan_dict", "scan_hidden_characters", diff --git a/fuckmark/lint.py b/fuckmark/lint.py index 094a41af..20f5a603 100644 --- a/fuckmark/lint.py +++ b/fuckmark/lint.py @@ -18,6 +18,7 @@ normalize_scan_categories, scan_hidden_characters, ) +from .product.severity import language_from_path LINT_ALGORITHM_VERSION = "fuckmark-lint-v1" @@ -203,7 +204,7 @@ def _write_text_atomic(path: Path, text: str) -> bool: def _lint_file(path: Path, categories: frozenset[str], text: str, fix: bool) -> FileLintResult: - scan = scan_hidden_characters(text) + scan = scan_hidden_characters(text, language=language_from_path(path)) counts = {name: scan.counts.get(name, 0) for name in scan.active_categories() if name in categories} total = sum(counts.values()) first_locations = tuple(finding for finding in scan.findings if finding.category in categories)[:10] diff --git a/fuckmark/product/__init__.py b/fuckmark/product/__init__.py index d7dacdac..4f17ba5f 100644 --- a/fuckmark/product/__init__.py +++ b/fuckmark/product/__init__.py @@ -47,8 +47,10 @@ SCAN_ALGORITHM_VERSION, SCAN_CATEGORIES, SECURITY_SCAN_CATEGORIES, + TROJAN_SOURCE_CATEGORIES, HiddenFinding, ScanResult, + autofix_trojan_source, classify_hidden_codepoint, clean_hidden_characters, extract_tag_payload, @@ -58,6 +60,7 @@ scan_machine_line, ) from .severity import ( + CONTEXT_COMMENT, CONTEXT_EMOJI, CONTEXT_IDENTIFIER, CONTEXT_PROSE, @@ -68,7 +71,10 @@ SEVERITY_MEDIUM, annotate_finding, classify_context, + language_from_path, + normalize_language, score_severity, + source_roles, ) from .search import PRODUCT_SEARCH_ALGORITHM_VERSION, raw_codepoint_contains, visible_contains from .visible_projection import ( @@ -95,12 +101,14 @@ "SCAN_ALGORITHM_VERSION", "SCAN_CATEGORIES", "SECURITY_SCAN_CATEGORIES", + "TROJAN_SOURCE_CATEGORIES", "NORMALIZE_ALGORITHM_VERSION", "NormalizeReceipt", "normalize_payload", "normalize_receipt_dict", "normalize_text", "skeleton_fold", + "CONTEXT_COMMENT", "CONTEXT_EMOJI", "CONTEXT_IDENTIFIER", "CONTEXT_PROSE", @@ -111,7 +119,11 @@ "SEVERITY_MEDIUM", "annotate_finding", "classify_context", + "language_from_path", + "normalize_language", "score_severity", + "source_roles", + "autofix_trojan_source", "HiddenFinding", "ScanResult", "classify_hidden_codepoint", diff --git a/fuckmark/product/scan.py b/fuckmark/product/scan.py index ad2d8136..a7462e45 100644 --- a/fuckmark/product/scan.py +++ b/fuckmark/product/scan.py @@ -5,7 +5,7 @@ from dataclasses import dataclass from .._validation import require_int -from .severity import annotate_finding, advance_string_state +from .severity import annotate_finding, normalize_language, source_roles from .visible_projection import product_approved_carriers_v1 @@ -48,6 +48,7 @@ CATEGORY_NONCHARACTER, CATEGORY_SURROGATE, ) +TROJAN_SOURCE_CATEGORIES = (CATEGORY_BIDI_CONTROL,) CATEGORY_DESCRIPTIONS = { CATEGORY_BIDI_CONTROL: "bidirectional override/isolate (Trojan Source reordering)", @@ -213,6 +214,7 @@ class ScanResult: findings: tuple[HiddenFinding, ...] truncated: bool fuckmark_carriers: int + highest_severity: str = "" @property def detected(self) -> bool: @@ -229,31 +231,31 @@ def first_hit(self) -> str: first = self.findings[0] return f"U+{first.codepoint:04X}@{first.index}({first.category})" - @property - def highest_severity(self) -> str: - if not self.findings: - return "" - return max(self.findings, key=lambda item: _SEVERITY_RANK.get(item.severity, 0)).severity - def active_categories(self) -> tuple[str, ...]: return tuple(name for name in SCAN_CATEGORIES if self.counts.get(name, 0) > 0) -def scan_hidden_characters(text: str, *, max_findings: int = DEFAULT_MAX_FINDINGS) -> ScanResult: +def scan_hidden_characters( + text: str, + *, + max_findings: int = DEFAULT_MAX_FINDINGS, + language: str | None = None, +) -> ScanResult: if not isinstance(text, str): raise TypeError("text must be a string") require_int("max_findings", max_findings) if max_findings < 0: raise ValueError("max_findings must not be negative") + lang = normalize_language(language) + roles = source_roles(text, lang) approved = product_approved_carriers_v1() counts = {name: 0 for name in SCAN_CATEGORIES} findings: list[HiddenFinding] = [] total = 0 truncated = False carriers = 0 - in_string = "" + peak = "" for index, character in enumerate(text): - in_string = advance_string_state(in_string, character) code = ord(character) if code in approved: carriers += 1 @@ -262,8 +264,10 @@ def scan_hidden_characters(text: str, *, max_findings: int = DEFAULT_MAX_FINDING continue total += 1 counts[category] += 1 + context, severity, why, remedy = annotate_finding(text, index, category, roles[index]) + if _SEVERITY_RANK.get(severity, -1) > _SEVERITY_RANK.get(peak, -1): + peak = severity if len(findings) < max_findings: - context, severity, why, remedy = annotate_finding(text, index, category, bool(in_string)) findings.append( HiddenFinding( index=index, @@ -284,6 +288,7 @@ def scan_hidden_characters(text: str, *, max_findings: int = DEFAULT_MAX_FINDING findings=tuple(findings), truncated=truncated, fuckmark_carriers=carriers, + highest_severity=peak, ) @@ -317,6 +322,10 @@ def clean_hidden_characters( return "".join(kept), removed +def autofix_trojan_source(text: str) -> tuple[str, int]: + return clean_hidden_characters(text, categories=TROJAN_SOURCE_CATEGORIES) + + def scan_machine_line(result: ScanResult) -> str: found = "yes" if result.detected else "no" categories = ",".join(f"{name}={result.counts[name]}" for name in result.active_categories()) diff --git a/fuckmark/product/severity.py b/fuckmark/product/severity.py index 395d4ec4..936dfccc 100644 --- a/fuckmark/product/severity.py +++ b/fuckmark/product/severity.py @@ -1,9 +1,12 @@ from __future__ import annotations +from pathlib import Path + CONTEXT_IDENTIFIER = "identifier" CONTEXT_EMOJI = "emoji" CONTEXT_STRING = "string" +CONTEXT_COMMENT = "comment" CONTEXT_PROSE = "prose" SEVERITY_CRITICAL = "critical" @@ -11,9 +14,200 @@ SEVERITY_MEDIUM = "medium" SEVERITY_INFO = "info" +ROLE_CODE = "code" +ROLE_STRING = "string" +ROLE_COMMENT = "comment" + _ZWJ = 0x200D _VS = frozenset({*range(0xFE00, 0xFE10)}) +_LANGUAGE_ALIASES = { + "auto": "auto", + "javascript": "javascript", + "js": "javascript", + "ts": "javascript", + "typescript": "javascript", + "jsx": "javascript", + "tsx": "javascript", + "c": "c", + "h": "c", + "cc": "c", + "cpp": "c", + "cxx": "c", + "java": "c", + "go": "c", + "rs": "c", + "rust": "c", + "cs": "c", + "css": "c", + "jsonc": "c", + "python": "python", + "py": "python", + "pyi": "python", + "hash": "python", + "sh": "python", + "bash": "python", + "zsh": "python", + "shell": "python", + "shellscript": "python", + "yaml": "python", + "yml": "python", + "rb": "python", + "ruby": "python", + "toml": "python", + "html": "html", + "htm": "html", + "xml": "html", + "sql": "sql", +} + +_LANGUAGE_BY_SUFFIX = { + ".js": "javascript", + ".mjs": "javascript", + ".cjs": "javascript", + ".ts": "javascript", + ".jsx": "javascript", + ".tsx": "javascript", + ".c": "c", + ".h": "c", + ".hh": "c", + ".cc": "c", + ".cpp": "c", + ".cxx": "c", + ".java": "c", + ".go": "c", + ".rs": "c", + ".cs": "c", + ".css": "c", + ".py": "python", + ".pyi": "python", + ".sh": "python", + ".bash": "python", + ".zsh": "python", + ".yaml": "python", + ".yml": "python", + ".rb": "python", + ".toml": "python", + ".html": "html", + ".htm": "html", + ".xml": "html", + ".sql": "sql", +} + + +def normalize_language(language: str | None) -> str: + if language is None: + return "auto" + if not isinstance(language, str): + raise TypeError("language must be a string") + key = language.strip().lower() + if not key: + return "auto" + return _LANGUAGE_ALIASES.get(key, "auto") + + +def language_from_path(path: str | Path) -> str: + suffix = Path(path).suffix.lower() + return _LANGUAGE_BY_SUFFIX.get(suffix, "auto") + + +def _slash_comments(language: str) -> bool: + return language in {"auto", "javascript", "c", "sql"} + + +def _block_comments(language: str) -> bool: + return language in {"auto", "javascript", "c", "sql"} + + +def _hash_comments(language: str) -> bool: + return language == "python" + + +def _sql_line_comments(language: str) -> bool: + return language == "sql" + + +def _html_comments(language: str) -> bool: + return language == "html" + + +def _is_url_slash_slash(text: str, index: int) -> bool: + return index > 0 and text[index - 1] == ":" + + +def source_roles(text: str, language: str | None = None) -> tuple[str, ...]: + if not isinstance(text, str): + raise TypeError("text must be a string") + lang = normalize_language(language) + roles = [ROLE_CODE] * len(text) + length = len(text) + index = 0 + in_string = "" + escape = False + while index < length: + character = text[index] + if in_string: + roles[index] = ROLE_STRING + if escape: + escape = False + index += 1 + continue + if character == "\\" and index + 1 < length: + escape = True + index += 1 + continue + if character == in_string: + in_string = "" + index += 1 + continue + pair = text[index : index + 2] + if _slash_comments(lang) and pair == "//" and not _is_url_slash_slash(text, index): + cursor = index + while cursor < length and text[cursor] not in "\n\r": + roles[cursor] = ROLE_COMMENT + cursor += 1 + index = cursor + continue + if _sql_line_comments(lang) and pair == "--": + cursor = index + while cursor < length and text[cursor] not in "\n\r": + roles[cursor] = ROLE_COMMENT + cursor += 1 + index = cursor + continue + if _block_comments(lang) and pair == "/*": + close = text.find("*/", index + 2) + stop = close + 2 if close >= 0 else length + cursor = index + while cursor < stop: + roles[cursor] = ROLE_COMMENT + cursor += 1 + index = stop + continue + if _hash_comments(lang) and character == "#": + cursor = index + while cursor < length and text[cursor] not in "\n\r": + roles[cursor] = ROLE_COMMENT + cursor += 1 + index = cursor + continue + if _html_comments(lang) and text[index : index + 4] == "", index + 4) + stop = close + 3 if close >= 0 else length + cursor = index + while cursor < stop: + roles[cursor] = ROLE_COMMENT + cursor += 1 + index = stop + continue + if character in {'"', "'", "`"}: + in_string = character + roles[index] = ROLE_STRING + index += 1 + continue + index += 1 + return tuple(roles) + def _is_emojiish(codepoint: int) -> bool: if codepoint < 0: @@ -33,19 +227,23 @@ def _is_ident_char(character: str) -> bool: return character == "_" or character.isalnum() -def classify_context(text: str, index: int, in_string: bool) -> str: +def classify_context(text: str, index: int, role: str = ROLE_CODE, category: str = "") -> str: prev = text[index - 1] if index > 0 else "" nxt = text[index + 1] if index + 1 < len(text) else "" prev_cp = ord(prev) if prev else -1 nxt_cp = ord(nxt) if nxt else -1 + if role in {ROLE_COMMENT, ROLE_STRING} and category == "bidi_control": + return CONTEXT_COMMENT if role == ROLE_COMMENT else CONTEXT_STRING if _is_emojiish(prev_cp) or _is_emojiish(nxt_cp) or prev_cp in _VS or nxt_cp in _VS: return CONTEXT_EMOJI + if role == ROLE_COMMENT: + return CONTEXT_COMMENT + if role == ROLE_STRING: + return CONTEXT_STRING if _is_ident_char(prev) and _is_ident_char(nxt): return CONTEXT_IDENTIFIER if _is_ident_char(nxt) or _is_ident_char(prev): return CONTEXT_IDENTIFIER - if in_string: - return CONTEXT_STRING return CONTEXT_PROSE @@ -53,7 +251,7 @@ def score_severity(category: str, context: str) -> str: if category == "tag": return SEVERITY_CRITICAL if category == "bidi_control": - if context == CONTEXT_IDENTIFIER: + if context in {CONTEXT_IDENTIFIER, CONTEXT_COMMENT, CONTEXT_STRING}: return SEVERITY_CRITICAL return SEVERITY_HIGH if category == "zero_width": @@ -79,6 +277,16 @@ def explain_finding(category: str, context: str, severity: str) -> tuple[str, st "Bidirectional override sits inside an identifier, so the glyphs can read differently than the bytes (Trojan Source).", "Strip the bidi control and keep the identifier left-to-right.", ) + if category == "bidi_control" and context == CONTEXT_COMMENT: + return ( + "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).", + "Strip the bidi control from the comment.", + ) + if category == "bidi_control" and context == CONTEXT_STRING: + return ( + "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).", + "Strip the bidi control from the string.", + ) if category == "bidi_control": return ( "Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).", @@ -120,8 +328,13 @@ def explain_finding(category: str, context: str, severity: str) -> tuple[str, st ) -def annotate_finding(text: str, index: int, category: str, in_string: bool) -> tuple[str, str, str, str]: - context = classify_context(text, index, in_string) +def annotate_finding( + text: str, + index: int, + category: str, + role: str = ROLE_CODE, +) -> tuple[str, str, str, str]: + context = classify_context(text, index, role, category) severity = score_severity(category, context) why, remedy = explain_finding(category, context, severity) return context, severity, why, remedy diff --git a/fuckmark/web.py b/fuckmark/web.py index d30a0084..865b90df 100644 --- a/fuckmark/web.py +++ b/fuckmark/web.py @@ -19,6 +19,7 @@ from .product.domain import PRODUCT_MAX_INPUT_CHARS from .product.normalize import normalize_payload from .product.scan import clean_hidden_characters, scan_dict, scan_hidden_characters +from .product.severity import normalize_language from .product.visible_projection import product_approved_carriers_v1, project_visible_v1 @@ -103,7 +104,7 @@ def remove_marks_payload(text: str) -> dict[str, object]: } -def scan_payload(text: str) -> dict[str, object]: +def scan_payload(text: str, *, language: str | None = None) -> dict[str, object]: if not isinstance(text, str): raise TypeError("text must be a string") if len(text) > PRODUCT_MAX_INPUT_CHARS: @@ -114,12 +115,13 @@ def scan_payload(text: str) -> dict[str, object]: "max": PRODUCT_MAX_INPUT_CHARS, "contact": DETECT_CONTACT_EMAIL, } - result = scan_hidden_characters(text) + result = scan_hidden_characters(text, language=language) cleaned, removed = clean_hidden_characters(text) return { "ok": True, "reason": "found" if result.detected else "clean", "backend": "python", + "language": normalize_language(language), "scan": scan_dict(result), "cleaned": cleaned, "removed": removed, @@ -144,7 +146,8 @@ def _web_parser() -> argparse.ArgumentParser: prog="fuckmark web", description=( "Open the FuckMark browser tool locally. " - "Serves mark.html and a Python API for detect/strip. " + "Serves mark.html, the hidden-Unicode scan page at /scan.html, " + "and a Python API for detect/strip/scan/guard/normalize. " "For beginners who prefer a page over the CLI." ), ) @@ -265,7 +268,10 @@ def do_POST(self) -> None: text = payload.get("text", "") if not isinstance(text, str): raise ValueError("text must be a string") - result = scan_payload(text) + language = payload.get("language", None) + if language is not None and not isinstance(language, str): + raise ValueError("language must be a string") + result = scan_payload(text, language=language) status = 200 if result.get("reason") != "too-large" else 413 self._json_response(status, result) except ValueError as error: @@ -340,6 +346,7 @@ class _Server(socketserver.TCPServer): errors.write( "FuckMark web: Python API at /api/health, /api/remove-marks, /api/scan, /api/guard, /api/normalize\n" ) + errors.write("FuckMark web: hidden-Unicode scan UI at /scan.html\n") errors.write("FuckMark web: press Ctrl+C to stop.\n") errors.flush() if on_ready is not None: diff --git a/fuckmark/webui/scan.html b/fuckmark/webui/scan.html new file mode 100644 index 00000000..0b007977 --- /dev/null +++ b/fuckmark/webui/scan.html @@ -0,0 +1,606 @@ + + + + + +FuckMark — Hidden Unicode scan + + + + + + + + + + +
+
+ +

See the bytes.

+

Paste source or a prompt. This page reveals Trojan Source bidi overrides, zero-width characters, and Unicode tag smuggling without sending the text anywhere.

+ + +
+ + + + Engine: browser +
+ + + +
+
+ + +
+ +

Paste text to scan.

+
+ +

+ Scanning is fuckmark-hidden-scan-v1. Copy uses the security category set + (bidi_control, zero_width, tag, control, + noncharacter, surrogate). Fix Trojan Source strips only + bidirectional controls. Nothing is uploaded from a static page or file://. + Under fuckmark web an optional local Python API can confirm the same verdicts. + Contact Fhelp@q1z.org. +

+
+
+ + + + diff --git a/fuckmark/webui/scan.js b/fuckmark/webui/scan.js new file mode 100644 index 00000000..f28c001f --- /dev/null +++ b/fuckmark/webui/scan.js @@ -0,0 +1,569 @@ +// Faithful JavaScript port of fuckmark-hidden-scan-v1 (fuckmark/product/scan.py). +// Kept in lockstep with the Python engine by tests/test_vscode_scanner_parity.py. + +"use strict"; + +const SCAN_ALGORITHM_VERSION = "fuckmark-hidden-scan-v1"; + +const CATEGORY_BIDI_CONTROL = "bidi_control"; +const CATEGORY_ZERO_WIDTH = "zero_width"; +const CATEGORY_VARIATION_SELECTOR = "variation_selector"; +const CATEGORY_TAG = "tag"; +const CATEGORY_ENCLOSING_MARK = "enclosing_mark"; +const CATEGORY_LINE_SEPARATOR = "line_separator"; +const CATEGORY_DEPRECATED = "deprecated"; +const CATEGORY_FORMAT = "format"; +const CATEGORY_CONTROL = "control"; +const CATEGORY_PRIVATE_USE = "private_use"; +const CATEGORY_NONCHARACTER = "noncharacter"; +const CATEGORY_SURROGATE = "surrogate"; + +const SCAN_CATEGORIES = [ + CATEGORY_BIDI_CONTROL, + CATEGORY_ZERO_WIDTH, + CATEGORY_VARIATION_SELECTOR, + CATEGORY_TAG, + CATEGORY_ENCLOSING_MARK, + CATEGORY_LINE_SEPARATOR, + CATEGORY_DEPRECATED, + CATEGORY_FORMAT, + CATEGORY_CONTROL, + CATEGORY_PRIVATE_USE, + CATEGORY_NONCHARACTER, + CATEGORY_SURROGATE, +]; + +const CATEGORY_DESCRIPTIONS = { + [CATEGORY_BIDI_CONTROL]: "bidirectional override/isolate (Trojan Source reordering)", + [CATEGORY_ZERO_WIDTH]: "zero-width or invisible spacing character", + [CATEGORY_VARIATION_SELECTOR]: "variation selector (glyph/steganography carrier)", + [CATEGORY_TAG]: "Unicode tag character (hidden text / prompt-injection smuggling)", + [CATEGORY_ENCLOSING_MARK]: "enclosing combining mark (alters surrounding glyph)", + [CATEGORY_LINE_SEPARATOR]: "line/paragraph separator (breaks parsers, invisible)", + [CATEGORY_DEPRECATED]: "deprecated format control or interlinear annotation", + [CATEGORY_FORMAT]: "general Unicode format control (Cf)", + [CATEGORY_CONTROL]: "C0/C1 control character", + [CATEGORY_PRIVATE_USE]: "private-use codepoint (renderer-defined)", + [CATEGORY_NONCHARACTER]: "Unicode noncharacter or unassigned codepoint", + [CATEGORY_SURROGATE]: "lone surrogate codepoint", +}; + +const DEFAULT_SECURITY_CATEGORIES = [ + CATEGORY_BIDI_CONTROL, + CATEGORY_ZERO_WIDTH, + CATEGORY_TAG, + CATEGORY_CONTROL, + CATEGORY_NONCHARACTER, + CATEGORY_SURROGATE, +]; + +const TROJAN_SOURCE_CATEGORIES = [CATEGORY_BIDI_CONTROL]; + +const ALLOWED_WHITESPACE = new Set([0x09, 0x0a, 0x0d, 0x20]); + +function inRange(cp, lo, hi) { + return cp >= lo && cp <= hi; +} + +const BIDI_CONTROL = new Set([0x061c, 0x200e, 0x200f]); +for (let cp = 0x202a; cp <= 0x202e; cp += 1) BIDI_CONTROL.add(cp); +for (let cp = 0x2066; cp <= 0x2069; cp += 1) BIDI_CONTROL.add(cp); + +const DEPRECATED = new Set([0xfff9, 0xfffa, 0xfffb]); +for (let cp = 0x206a; cp <= 0x206f; cp += 1) DEPRECATED.add(cp); + +const ZERO_WIDTH = new Set([ + 0x00ad, 0x034f, 0x115f, 0x1160, 0x17b4, 0x17b5, 0x180e, 0x200b, 0x200c, + 0x200d, 0x2060, 0x2061, 0x2062, 0x2063, 0x2064, 0x3164, 0xfeff, 0xffa0, +]); + +function isVariationSelector(cp) { + return inRange(cp, 0xfe00, 0xfe0f) || inRange(cp, 0xe0100, 0xe01ef); +} + +function isTag(cp) { + return inRange(cp, 0xe0000, 0xe007f); +} + +function isNoncharacter(cp) { + if (inRange(cp, 0xfdd0, 0xfdef)) return true; + const low = cp & 0xffff; + return low === 0xfffe || low === 0xffff; +} + +function isEnclosingMark(cp) { + // Unicode general category Me. Enumerated to avoid a runtime Unicode DB. + return ( + inRange(cp, 0x0488, 0x0489) || + cp === 0x1abe || + inRange(cp, 0x20dd, 0x20e0) || + inRange(cp, 0x20e2, 0x20e4) || + inRange(cp, 0xa670, 0xa672) + ); +} + +function isControl(cp) { + return inRange(cp, 0x00, 0x1f) || inRange(cp, 0x7f, 0x9f); +} + +function isLineSeparator(cp) { + return cp === 0x2028 || cp === 0x2029; +} + +// General category Cf ranges (Unicode 15-era), excluding codepoints already +// classified above. Mirrors what unicodedata.category returns as "Cf". +const FORMAT_RANGES = [ + [0x0600, 0x0605], + [0x06dd, 0x06dd], + [0x070f, 0x070f], + [0x0890, 0x0891], + [0x08e2, 0x08e2], + [0x110bd, 0x110bd], + [0x110cd, 0x110cd], + [0x13430, 0x1343f], + [0x1bca0, 0x1bca3], + [0x1d173, 0x1d17a], +]; + +function isFormat(cp) { + for (const [lo, hi] of FORMAT_RANGES) { + if (inRange(cp, lo, hi)) return true; + } + return false; +} + +function isPrivateUse(cp) { + return ( + inRange(cp, 0xe000, 0xf8ff) || + inRange(cp, 0xf0000, 0xffffd) || + inRange(cp, 0x100000, 0x10fffd) + ); +} + +function classify(cp) { + if (ALLOWED_WHITESPACE.has(cp)) return null; + if (BIDI_CONTROL.has(cp)) return CATEGORY_BIDI_CONTROL; + if (DEPRECATED.has(cp)) return CATEGORY_DEPRECATED; + if (ZERO_WIDTH.has(cp)) return CATEGORY_ZERO_WIDTH; + if (isVariationSelector(cp)) return CATEGORY_VARIATION_SELECTOR; + if (isTag(cp)) return CATEGORY_TAG; + if (isNoncharacter(cp)) return CATEGORY_NONCHARACTER; + if (cp >= 0xd800 && cp <= 0xdfff) return CATEGORY_SURROGATE; + if (isEnclosingMark(cp)) return CATEGORY_ENCLOSING_MARK; + if (isLineSeparator(cp)) return CATEGORY_LINE_SEPARATOR; + if (isControl(cp)) return CATEGORY_CONTROL; + if (isFormat(cp)) return CATEGORY_FORMAT; + if (isPrivateUse(cp)) return CATEGORY_PRIVATE_USE; + return null; +} + +function codepointToken(cp) { + return "U+" + cp.toString(16).toUpperCase().padStart(4, "0"); +} + +function isEmojiish(cp) { + if (cp < 0) return false; + if (cp >= 0x1f1e6 && cp <= 0x1f1ff) return true; + if (cp >= 0x1f000 && cp <= 0x1faff) return true; + if (cp >= 0x2600 && cp <= 0x27bf) return true; + return cp === 0x200d || [0x00a9, 0x00ae, 0x203c, 0x2049, 0x2122, 0x2139, 0x3030, 0x303d, 0x3297, 0x3299].includes(cp); +} + +function isIdentChar(ch) { + return ch === "_" || /\p{L}|\p{N}/u.test(ch); +} + +function neighborBefore(text, offset) { + if (offset <= 0) return { cp: -1, ch: "" }; + if (offset >= 2) { + const cp = text.codePointAt(offset - 2); + if (cp > 0xffff) return { cp, ch: String.fromCodePoint(cp) }; + } + const cp = text.codePointAt(offset - 1); + return { cp, ch: String.fromCodePoint(cp) }; +} + +function neighborAfter(text, offset, length) { + const nextIndex = offset + length; + if (nextIndex >= text.length) return { cp: -1, ch: "" }; + const cp = text.codePointAt(nextIndex); + return { cp, ch: String.fromCodePoint(cp) }; +} + +const LANGUAGE_ALIASES = { + auto: "auto", + javascript: "javascript", + js: "javascript", + ts: "javascript", + typescript: "javascript", + jsx: "javascript", + tsx: "javascript", + c: "c", + h: "c", + cc: "c", + cpp: "c", + cxx: "c", + java: "c", + go: "c", + rs: "c", + rust: "c", + cs: "c", + css: "c", + jsonc: "c", + python: "python", + py: "python", + pyi: "python", + hash: "python", + sh: "python", + bash: "python", + zsh: "python", + shell: "python", + shellscript: "python", + yaml: "python", + yml: "python", + rb: "python", + ruby: "python", + toml: "python", + html: "html", + htm: "html", + xml: "html", + sql: "sql", +}; + +function normalizeLanguage(language) { + if (!language) return "auto"; + const key = String(language).trim().toLowerCase(); + return LANGUAGE_ALIASES[key] || "auto"; +} + +function slashComments(language) { + return language === "auto" || language === "javascript" || language === "c" || language === "sql"; +} + +function blockComments(language) { + return slashComments(language); +} + +function hashComments(language) { + return language === "python"; +} + +function sqlLineComments(language) { + return language === "sql"; +} + +function htmlComments(language) { + return language === "html"; +} + +function sourceRoles(text, language) { + const lang = normalizeLanguage(language); + const roles = new Array(text.length).fill("code"); + let index = 0; + let inString = ""; + let escape = false; + while (index < text.length) { + const ch = text[index]; + if (inString) { + roles[index] = "string"; + if (escape) { + escape = false; + index += 1; + continue; + } + if (ch === "\\" && index + 1 < text.length) { + escape = true; + index += 1; + continue; + } + if (ch === inString) inString = ""; + index += 1; + continue; + } + const pair = text.slice(index, index + 2); + if (slashComments(lang) && pair === "//" && !(index > 0 && text[index - 1] === ":")) { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (sqlLineComments(lang) && pair === "--") { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (blockComments(lang) && pair === "/*") { + const close = text.indexOf("*/", index + 2); + const stop = close >= 0 ? close + 2 : text.length; + for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment"; + index = stop; + continue; + } + if (hashComments(lang) && ch === "#") { + let cursor = index; + while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") { + roles[cursor] = "comment"; + cursor += 1; + } + index = cursor; + continue; + } + if (htmlComments(lang) && text.slice(index, index + 4) === "", index + 4); + const stop = close >= 0 ? close + 3 : text.length; + for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment"; + index = stop; + continue; + } + if (ch === '"' || ch === "'" || ch === "`") { + inString = ch; + roles[index] = "string"; + index += 1; + continue; + } + index += 1; + } + return roles; +} + +function classifyContext(text, offset, role, category) { + const length = text.codePointAt(offset) > 0xffff ? 2 : 1; + const prev = neighborBefore(text, offset); + const next = neighborAfter(text, offset, length); + if ((role === "comment" || role === "string") && category === CATEGORY_BIDI_CONTROL) { + return role; + } + if ( + isEmojiish(prev.cp) || + isEmojiish(next.cp) || + (prev.cp >= 0xfe00 && prev.cp <= 0xfe0f) || + (next.cp >= 0xfe00 && next.cp <= 0xfe0f) + ) { + return "emoji"; + } + if (role === "comment") return "comment"; + if (role === "string") return "string"; + if (isIdentChar(prev.ch) || isIdentChar(next.ch)) return "identifier"; + return "prose"; +} + +function scoreSeverity(category, context) { + if (category === CATEGORY_TAG) return "critical"; + if (category === CATEGORY_BIDI_CONTROL) { + if (context === "identifier" || context === "comment" || context === "string") return "critical"; + return "high"; + } + if (category === CATEGORY_ZERO_WIDTH) { + if (context === "emoji") return "info"; + if (context === "identifier") return "high"; + return "medium"; + } + if (category === CATEGORY_VARIATION_SELECTOR) return context === "emoji" ? "info" : "medium"; + if (category === CATEGORY_CONTROL || category === CATEGORY_NONCHARACTER || category === CATEGORY_SURROGATE) { + return "high"; + } + return "medium"; +} + +function explainFinding(category, context, severity) { + if (category === CATEGORY_BIDI_CONTROL && context === "identifier") { + return { + why: "Bidirectional override sits inside an identifier, so the glyphs can read differently than the bytes (Trojan Source).", + remedy: "Strip the bidi control and keep the identifier left-to-right.", + }; + } + if (category === CATEGORY_BIDI_CONTROL && context === "comment") { + return { + why: "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).", + remedy: "Strip the bidi control from the comment.", + }; + } + if (category === CATEGORY_BIDI_CONTROL && context === "string") { + return { + why: "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).", + remedy: "Strip the bidi control from the string.", + }; + } + if (category === CATEGORY_BIDI_CONTROL) { + return { + why: "Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).", + remedy: "Strip U+202A-U+202E / U+2066-U+2069 and rewrite the text left-to-right.", + }; + } + if (category === CATEGORY_TAG) { + return { + why: "Unicode tag characters encode a second ASCII string that models read and humans do not.", + remedy: "Strip U+E0020-U+E007F; inspect tag_payload for the smuggled text.", + }; + } + if (category === CATEGORY_ZERO_WIDTH && context === "emoji") { + return { + why: "Zero-width joiner or invisible mark inside an emoji cluster; usually a legitimate emoji sequence.", + remedy: "Leave emoji ZWJ sequences unless you are sanitizing for a security boundary.", + }; + } + if (category === CATEGORY_ZERO_WIDTH && context === "identifier") { + return { + why: "Zero-width character splits an identifier, breaking search and some compilers while looking unchanged.", + remedy: "Strip the zero-width character from the identifier.", + }; + } + if (category === CATEGORY_ZERO_WIDTH) { + return { + why: "Invisible spacing or joining character that changes the byte stream without changing the glyphs.", + remedy: "Strip the zero-width character.", + }; + } + if (category === CATEGORY_VARIATION_SELECTOR && context === "emoji") { + return { + why: "Variation selector tunes an emoji glyph; usually benign.", + remedy: "Keep emoji variation selectors unless you are stripping all hidden marks.", + }; + } + if (severity === "high") { + return { + why: "Hidden or non-text codepoint that should not appear in ordinary source or prompts.", + remedy: "Strip the character.", + }; + } + return { + why: CATEGORY_DESCRIPTIONS[category] || "Hidden or format codepoint that is invisible or renderer-defined.", + remedy: "Strip the character if this text crosses a trust boundary.", + }; +} + +function scanText(text, categories, language) { + const selected = categories ? new Set(categories) : null; + const roles = sourceRoles(text, language); + const counts = {}; + const findings = []; + let total = 0; + let offset = 0; + while (offset < text.length) { + const cp = text.codePointAt(offset); + const length = cp > 0xffff ? 2 : 1; + const category = classify(cp); + if (category !== null && (selected === null || selected.has(category))) { + total += 1; + counts[category] = (counts[category] || 0) + 1; + const context = classifyContext(text, offset, roles[offset] || "code", category); + const severity = scoreSeverity(category, context); + const explained = explainFinding(category, context, severity); + findings.push({ + offset, + length, + codepoint: cp, + category, + context, + severity, + why: explained.why, + remedy: explained.remedy, + }); + } + offset += length; + } + return { total, counts, findings }; +} + +function cleanText(text, categories) { + const selected = categories ? new Set(categories) : null; + let out = ""; + let removed = 0; + let offset = 0; + while (offset < text.length) { + const cp = text.codePointAt(offset); + const length = cp > 0xffff ? 2 : 1; + const category = classify(cp); + if (category !== null && (selected === null || selected.has(category))) { + removed += 1; + } else { + out += text.slice(offset, offset + length); + } + offset += length; + } + return { cleaned: out, removed }; +} + +function autofixTrojanSource(text) { + return cleanText(text, TROJAN_SOURCE_CATEGORIES); +} + +function languageFromDocumentId(languageId) { + return normalizeLanguage(languageId); +} + +const LANGUAGE_BY_SUFFIX = { + ".js": "javascript", + ".mjs": "javascript", + ".cjs": "javascript", + ".ts": "javascript", + ".jsx": "javascript", + ".tsx": "javascript", + ".c": "c", + ".h": "c", + ".hh": "c", + ".cc": "c", + ".cpp": "c", + ".cxx": "c", + ".java": "c", + ".go": "c", + ".rs": "c", + ".cs": "c", + ".css": "c", + ".py": "python", + ".pyi": "python", + ".sh": "python", + ".bash": "python", + ".zsh": "python", + ".yaml": "python", + ".yml": "python", + ".rb": "python", + ".toml": "python", + ".html": "html", + ".htm": "html", + ".xml": "html", + ".sql": "sql", +}; + +function languageFromPath(path) { + const name = String(path || ""); + const slash = Math.max(name.lastIndexOf("/"), name.lastIndexOf("\\")); + const base = slash >= 0 ? name.slice(slash + 1) : name; + const dot = base.lastIndexOf("."); + if (dot <= 0) return "auto"; + const suffix = base.slice(dot).toLowerCase(); + return LANGUAGE_BY_SUFFIX[suffix] || "auto"; +} + +const FuckMarkScan = { + SCAN_ALGORITHM_VERSION, + SCAN_CATEGORIES, + CATEGORY_DESCRIPTIONS, + DEFAULT_SECURITY_CATEGORIES, + TROJAN_SOURCE_CATEGORIES, + classify, + classifyContext, + scoreSeverity, + normalizeLanguage, + languageFromDocumentId, + languageFromPath, + sourceRoles, + codepointToken, + scanText, + cleanText, + autofixTrojanSource, +}; + +if (typeof module !== "undefined" && module.exports) { + module.exports = FuckMarkScan; +} +if (typeof globalThis !== "undefined") { + globalThis.FuckMarkScan = FuckMarkScan; +} diff --git a/pyproject.toml b/pyproject.toml index 21bc71b3..7f78157b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -50,4 +50,4 @@ where = ["."] include = ["fuckmark*"] [tool.setuptools.package-data] -fuckmark = ["webui/*.html"] +fuckmark = ["webui/*.html", "webui/*.js"] diff --git a/specs/fuckmark-hidden-scan-v1.freeze.json b/specs/fuckmark-hidden-scan-v1.freeze.json index 68084540..f3e4237d 100644 --- a/specs/fuckmark-hidden-scan-v1.freeze.json +++ b/specs/fuckmark-hidden-scan-v1.freeze.json @@ -1,9 +1,9 @@ { "algorithm_version": "fuckmark-hidden-scan-v1", - "freeze_record_version": "fuckmark-hidden-scan-v1-freeze-v1", - "frozen_at_utc": "2026-09-01T23:30:00+00:00", + "freeze_record_version": "fuckmark-hidden-scan-v1-freeze-v3", + "frozen_at_utc": "2026-09-02T01:50:00+00:00", "protocol_id": "fuckmark-hidden-scan-v1", - "protocol_sha256": "859b1b138d8de3795cb09228ea78775b560aeb63520a309f9f9e59a7acc31046", - "vectors_file_sha256": "a9fbdb7fc2bf30786d9d9199099e740da1e1b33b52518a2d61b038fd85d80610", - "vectors_canonical_sha256": "2602ca87e5480813caf4452f8314e1ff2dbb43e94fa3cd81a6937b1595287915" + "protocol_sha256": "4e594b87ba2f68b929e62fc420b4543186ecc7f71f41f501f1bb646c33c23045", + "vectors_file_sha256": "acb8e4ddbd8849979b8f493fabb136fa63e8d4364ce60aedef3ce853006802a3", + "vectors_canonical_sha256": "763f2967f68882ec037b6245ad89e166f6b7482292b0e13d774475acb81c5e4a" } diff --git a/specs/fuckmark-hidden-scan-v1.protocol.md b/specs/fuckmark-hidden-scan-v1.protocol.md index 64bacc47..38d5a71e 100644 --- a/specs/fuckmark-hidden-scan-v1.protocol.md +++ b/specs/fuckmark-hidden-scan-v1.protocol.md @@ -60,30 +60,47 @@ controls and related Cf assignments. ## 4. Context (heuristic) -Context is assigned to each finding from immediate neighbors. It is not a -language parser. +Context is assigned to each finding from a lightweight source mask plus +immediate neighbors. It is not a full language parser. +Roles (optional `language` argument to `scan_hidden_characters`; default +`auto`): + +- `comment`: `//` line comments and `/* */` block comments (`auto`, + `javascript`, `c`, `sql`). `#` line comments (`python`). `--` line comments + (`sql`). `` (`html`). `//` after `:` is not a comment (`http://`). +- `string`: `"`, `'`, or backtick literals, with backslash escapes. + +Then: + +- `comment` / `string` from the role mask when the category is `bidi_control` + (Trojan Source encodings stay on that role even next to emoji). - `emoji` if the previous or next scalar is emoji-ish (regional indicators U+1F1E6-U+1F1FF, U+1F000-U+1FAFF, U+2600-U+27BF, a small BMP emoji set, ZWJ, or a BMP variation selector U+FE00-U+FE0F). +- else `comment` / `string` from the role mask. - else `identifier` if the previous or next scalar is `_` or alphanumeric. -- else `string` if a naive quote tracker currently has an open `"`, `'`, or - backtick. The tracker flips on the current scalar before classification, so - a hidden character inside quotes is `string` unless a stronger context - applies. It does not understand escapes or nested languages. - else `prose`. +`language` may be passed on the vector as `"language"`. Omitted means `auto`. +Lint infers language from the file suffix. This lexer does not nest block +comments, does not parse raw strings, and does not understand every dialect. + ## 5. Severity -| Category | identifier | emoji | other | +| Category | identifier / comment / string | emoji | other | | --- | --- | --- | --- | | `tag` | critical | critical | critical | | `bidi_control` | critical | high | high | -| `zero_width` | high | info | medium | +| `zero_width` | high (identifier) / medium (comment, string) | info | medium | | `variation_selector` | medium | info | medium | | `control`, `noncharacter`, `surrogate` | high | high | high | | remaining flagged categories | medium | medium | medium | +`bidi_control` in identifier, comment, or string is the three Trojan Source +encodings (CVE-2021-42574): identifier reordering, commenting-out, and +stretched-string. `autofix_trojan_source` strips only `bidi_control`. + Each finding also carries a one-line `why` and `remedy`. Those strings are informative; conformance is category, context, and severity. @@ -106,8 +123,9 @@ Default security categories (lint, guard, normalize strip): `bidi_control`, ## 7. Conformance Reconstruct each vector as `"".join(chr(cp) for cp in codepoints)` (Python -scalars). Run `scan_hidden_characters`. For every expected finding, require -matching `index`, `codepoint`, `category`, `context`, and `severity`, in -order. Vectors that expect no findings must produce `detected is False`. +scalars). Run `scan_hidden_characters` with `language` from the vector or +`auto`. For every expected finding, require matching `index`, `codepoint`, +`category`, `context`, and `severity`, in order. Vectors that expect no +findings must produce `detected is False`. Do not store the reconstructed strings in the vector file. diff --git a/specs/fuckmark-hidden-scan-v1.vectors.json b/specs/fuckmark-hidden-scan-v1.vectors.json index e8bb44bf..a3d6a78b 100644 --- a/specs/fuckmark-hidden-scan-v1.vectors.json +++ b/specs/fuckmark-hidden-scan-v1.vectors.json @@ -140,7 +140,7 @@ "codepoint": 8238, "category": "bidi_control", "context": "string", - "severity": "high" + "severity": "critical" } ] }, @@ -185,6 +185,199 @@ 99 ], "expect": [] + }, + { + "id": "bidi-stretched-string", + "codepoints": [ + 34, + 8238, + 97, + 34 + ], + "expect": [ + { + "index": 1, + "codepoint": 8238, + "category": "bidi_control", + "context": "string", + "severity": "critical" + } + ] + }, + { + "id": "bidi-slash-comment", + "codepoints": [ + 47, + 47, + 32, + 8238 + ], + "expect": [ + { + "index": 3, + "codepoint": 8238, + "category": "bidi_control", + "context": "comment", + "severity": "critical" + } + ] + }, + { + "id": "bidi-block-comment", + "codepoints": [ + 47, + 42, + 32, + 8238, + 32, + 42, + 47 + ], + "expect": [ + { + "index": 3, + "codepoint": 8238, + "category": "bidi_control", + "context": "comment", + "severity": "critical" + } + ] + }, + { + "id": "bidi-hash-comment", + "language": "python", + "codepoints": [ + 35, + 32, + 8238 + ], + "expect": [ + { + "index": 2, + "codepoint": 8238, + "category": "bidi_control", + "context": "comment", + "severity": "critical" + } + ] + }, + { + "id": "hash-not-comment-in-auto", + "codepoints": [ + 35, + 32, + 8238 + ], + "expect": [ + { + "index": 2, + "codepoint": 8238, + "category": "bidi_control", + "context": "prose", + "severity": "high" + } + ] + }, + { + "id": "bidi-after-url-not-comment", + "codepoints": [ + 104, + 116, + 116, + 112, + 58, + 47, + 47, + 8238 + ], + "expect": [ + { + "index": 7, + "codepoint": 8238, + "category": "bidi_control", + "context": "prose", + "severity": "high" + } + ] + }, + { + "id": "bidi-sql-line-comment", + "language": "sql", + "codepoints": [ + 45, + 45, + 32, + 8238 + ], + "expect": [ + { + "index": 3, + "codepoint": 8238, + "category": "bidi_control", + "context": "comment", + "severity": "critical" + } + ] + }, + { + "id": "bidi-html-comment", + "language": "html", + "codepoints": [ + 60, + 33, + 45, + 45, + 32, + 8238, + 45, + 45, + 62 + ], + "expect": [ + { + "index": 5, + "codepoint": 8238, + "category": "bidi_control", + "context": "comment", + "severity": "critical" + } + ] + }, + { + "id": "bidi-slash-comment-beside-emoji", + "codepoints": [ + 47, + 47, + 128512, + 8238 + ], + "expect": [ + { + "index": 3, + "codepoint": 8238, + "category": "bidi_control", + "context": "comment", + "severity": "critical" + } + ] + }, + { + "id": "bidi-string-beside-emoji", + "codepoints": [ + 34, + 128512, + 8238, + 34 + ], + "expect": [ + { + "index": 2, + "codepoint": 8238, + "category": "bidi_control", + "context": "string", + "severity": "critical" + } + ] } ] } diff --git a/tests/test_hidden_scan_spec.py b/tests/test_hidden_scan_spec.py index e8e8ec62..dba8f604 100644 --- a/tests/test_hidden_scan_spec.py +++ b/tests/test_hidden_scan_spec.py @@ -38,7 +38,7 @@ def test_hidden_scan_vectors_replay_on_python_engine() -> None: payload = _load_vectors() for vector in payload["vectors"]: text = "".join(chr(code) for code in vector["codepoints"]) - result = scan_hidden_characters(text) + result = scan_hidden_characters(text, language=vector.get("language")) expected = vector["expect"] assert len(result.findings) == len(expected), vector["id"] for finding, want in zip(result.findings, expected, strict=True): @@ -62,7 +62,7 @@ def test_hidden_scan_js_replays_vector_verdicts() -> None: const payload = JSON.parse(require("fs").readFileSync(0, "utf8")); const out = payload.vectors.map((vector) => { const text = String.fromCodePoint(...vector.codepoints); - const result = scan.scanText(text); + const result = scan.scanText(text, null, vector.language || "auto"); return result.findings.map((finding) => ({ codepoint: finding.codepoint, category: finding.category, @@ -107,3 +107,11 @@ def test_scan_js_syntax() -> None: subprocess.run([node, "--check", str(SCAN_JS)], check=True, timeout=15) extension = ROOT / "editors" / "vscode" / "extension.js" subprocess.run([node, "--check", str(extension)], check=True, timeout=15) + copies = ( + ROOT / "docs" / "scan.js", + ROOT / "fuckmark" / "webui" / "scan.js", + ) + canonical = SCAN_JS.read_text(encoding="utf-8") + for path in copies: + assert path.read_text(encoding="utf-8") == canonical + subprocess.run([node, "--check", str(path)], check=True, timeout=15) diff --git a/tests/test_lint.py b/tests/test_lint.py index 697f1dc8..a27f75af 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -66,6 +66,18 @@ def test_json_report_structure(tmp_path: Path) -> None: assert result["locations"][0]["remedy"] +def test_javascript_comment_context_from_suffix(tmp_path: Path) -> None: + (tmp_path / "note.js").write_text("// \u202e\n", encoding="utf-8") + code, out, err = _run(["--json", str(tmp_path)]) + assert code == LINT_EXIT_FINDINGS + assert err == "" + report = json.loads(out) + location = report["results"][0]["locations"][0] + assert location["context"] == "comment" + assert location["severity"] == "critical" + assert location["codepoint"] == "U+202E" + + def test_fix_rewrites_files_and_then_passes(tmp_path: Path) -> None: target = tmp_path / "evil.js" target.write_text(TROJAN, encoding="utf-8") diff --git a/tests/test_scan.py b/tests/test_scan.py index e1e0908e..bed0b7ba 100644 --- a/tests/test_scan.py +++ b/tests/test_scan.py @@ -13,6 +13,7 @@ CATEGORY_VARIATION_SELECTOR, CATEGORY_ZERO_WIDTH, SCAN_CATEGORIES, + autofix_trojan_source, classify_hidden_codepoint, clean_hidden_characters, scan_dict, @@ -20,6 +21,7 @@ scan_human_report, scan_machine_line, ) +from fuckmark.product.severity import language_from_path, source_roles from fuckmark.web import scan_payload, serve_mark_web @@ -209,6 +211,7 @@ def test_web_scan_payload_and_endpoint() -> None: assert payload["cleaned"] == "Hello from the scanner." assert int(payload["removed"]) > 0 assert payload["scan"]["found"] is True + assert payload["language"] == "auto" seen: dict[str, object] = {} @@ -233,6 +236,32 @@ def on_ready(url: str, port: int) -> None: assert clean_data["reason"] == "clean" assert clean_data["scan"]["found"] is False + def on_ready_language(url: str, port: int) -> None: + py_status, py_data = _post_json( + f"http://127.0.0.1:{port}/api/scan", + {"text": "# \u202e", "language": "python"}, + ) + seen["py_status"] = py_status + seen["py_data"] = py_data + auto_status, auto_data = _post_json( + f"http://127.0.0.1:{port}/api/scan", + {"text": "# \u202e"}, + ) + seen["auto_status"] = auto_status + seen["auto_data"] = auto_data + + serve_mark_web(host="127.0.0.1", port=0, open_browser=False, serve_seconds=0.05, on_ready=on_ready_language) + assert seen["py_status"] == 200 + py_data = seen["py_data"] + assert isinstance(py_data, dict) + assert py_data["language"] == "python" + assert py_data["scan"]["findings"][0]["context"] == "comment" + assert py_data["scan"]["findings"][0]["severity"] == "critical" + auto_data = seen["auto_data"] + assert isinstance(auto_data, dict) + assert auto_data["language"] == "auto" + assert auto_data["scan"]["findings"][0]["context"] == "prose" + def test_severity_is_context_aware() -> None: ident = scan_hidden_characters("a\u202eb") @@ -252,4 +281,65 @@ def test_severity_is_context_aware() -> None: quoted = scan_hidden_characters('"\u202e"') assert quoted.findings[0].context == "string" - assert quoted.findings[0].severity == "high" + assert quoted.findings[0].severity == "critical" + + comment = scan_hidden_characters("// \u202e") + assert comment.findings[0].context == "comment" + assert comment.findings[0].severity == "critical" + + +def test_language_from_path_maps_suffixes() -> None: + assert language_from_path("app.js") == "javascript" + assert language_from_path("mod.py") == "python" + assert language_from_path("q.sql") == "sql" + assert language_from_path("page.html") == "html" + assert language_from_path("notes.txt") == "auto" + + +def test_hash_comment_is_language_aware() -> None: + python = scan_hidden_characters("# \u202e", language="python") + assert python.findings[0].context == "comment" + assert python.findings[0].severity == "critical" + auto = scan_hidden_characters("# \u202e") + assert auto.findings[0].context == "prose" + assert auto.findings[0].severity == "high" + + +def test_url_double_slash_is_not_a_comment() -> None: + roles = source_roles("http://x") + assert all(role == "code" for role in roles) + result = scan_hidden_characters("http://\u202e") + assert result.findings[0].context == "prose" + assert result.findings[0].severity == "high" + + +def test_autofix_trojan_source_strips_only_bidi() -> None: + text = "a\u202e\u200bb" + cleaned, removed = autofix_trojan_source(text) + assert removed == 1 + assert cleaned == "a\u200bb" + assert "\u202e" not in cleaned + + +def test_bidi_in_comment_or_string_stays_critical_beside_emoji() -> None: + comment = scan_hidden_characters("//\U0001F600\u202e") + assert comment.findings[0].category == CATEGORY_BIDI_CONTROL + assert comment.findings[0].context == "comment" + assert comment.findings[0].severity == "critical" + quoted = scan_hidden_characters('"\U0001F600\u202e"') + assert quoted.findings[0].context == "string" + assert quoted.findings[0].severity == "critical" + emoji = scan_hidden_characters("// \U0001F468\u200d\U0001F469") + zwj = next(item for item in emoji.findings if item.category == CATEGORY_ZERO_WIDTH) + assert zwj.context == "emoji" + assert zwj.severity == "info" + + +def test_highest_severity_includes_hits_past_findings_cap() -> None: + text = "\u200b" * 300 + "a\u202eb" + result = scan_hidden_characters(text, max_findings=256) + assert result.truncated is True + assert result.total == 301 + assert len(result.findings) == 256 + assert result.counts[CATEGORY_BIDI_CONTROL] == 1 + assert result.highest_severity == "critical" diff --git a/tests/test_web.py b/tests/test_web.py index 8b18183d..fc4f8553 100644 --- a/tests/test_web.py +++ b/tests/test_web.py @@ -47,6 +47,26 @@ def test_packaged_mark_html_matches_docs_source() -> None: assert web_root() == mark_html_path().parent +def test_packaged_scan_page_matches_docs_and_editor_engine() -> None: + docs_html = (ROOT / "docs" / "scan.html").read_text(encoding="utf-8") + packaged_html = (ROOT / "fuckmark" / "webui" / "scan.html").read_text(encoding="utf-8") + assert packaged_html == docs_html + canonical_js = (ROOT / "editors" / "vscode" / "scan.js").read_text(encoding="utf-8") + assert (ROOT / "docs" / "scan.js").read_text(encoding="utf-8") == canonical_js + assert (ROOT / "fuckmark" / "webui" / "scan.js").read_text(encoding="utf-8") == canonical_js + assert 'script src="scan.js"' in docs_html + assert "FuckMarkScan" in docs_html + assert "autofixTrojanSource" in docs_html + assert "DEFAULT_SECURITY_CATEGORIES" in docs_html + assert "fromCodePoint(0x202E)" in docs_html + assert "truncated" in docs_html + assert "bidi_control" in docs_html + from fuckmark.product.scan import scan_hidden_characters + + assert scan_hidden_characters(docs_html, language="html").detected is False + assert scan_hidden_characters(canonical_js, language="javascript").detected is False + + def test_mark_html_miss_copy_is_english() -> None: html = mark_html_path().read_text(encoding="utf-8") assert "We did not detect a watermark in this text." in html @@ -112,6 +132,11 @@ def on_ready(url: str, port: int) -> None: seen["cache"] = response.headers.get("Cache-Control") with urlopen(f"http://127.0.0.1:{port}/", timeout=2) as response: seen["index"] = response.read().decode("utf-8") + with urlopen(f"http://127.0.0.1:{port}/scan.html", timeout=2) as response: + seen["scan_html"] = response.read().decode("utf-8") + seen["scan_cache"] = response.headers.get("Cache-Control") + with urlopen(f"http://127.0.0.1:{port}/scan.js", timeout=2) as response: + seen["scan_js"] = response.read().decode("utf-8") health_status, health = _get_json(f"http://127.0.0.1:{port}/api/health") seen["health_status"] = health_status seen["health"] = health @@ -145,6 +170,7 @@ def on_ready(url: str, port: int) -> None: assert "FuckMark web:" in log assert "Python API" in log assert "/api/remove-marks" in log + assert "/scan.html" in log body = str(seen["body"]) assert "FuckMark" in body assert "detectFuckMark" in body @@ -152,6 +178,11 @@ def on_ready(url: str, port: int) -> None: assert "/api/remove-marks" in body assert "We did not detect a watermark in this text." in body assert "detectFuckMark" in str(seen["index"]) + scan_html = str(seen["scan_html"]) + assert "See the bytes." in scan_html + assert 'script src="scan.js"' in scan_html + assert "FuckMarkScan" in str(seen["scan_js"]) + assert "no-store" in str(seen.get("scan_cache", "")).casefold() assert "no-store" in str(seen.get("cache", "")).casefold() assert seen["health_status"] == 200 health = seen["health"] @@ -191,4 +222,5 @@ def test_cli_web_help_documents_browser_tool() -> None: help_text = out.getvalue() + errors.getvalue() assert "fuckmark web" in help_text assert "Python API" in help_text + assert "scan.html" in help_text assert "browser" in help_text.casefold() or "beginner" in help_text.casefold()