diff --git a/CHANGELOG.md b/CHANGELOG.md
index f0687e97..f485fdce 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,7 +11,9 @@
- LLM input guard (`fuckmark-guard-v1`). `protect()` / `inspect()` / `Guard` / `@Guard().wrap` sanitize strings, lists, and dicts (OpenAI-style `messages` included) before a model call. Default action is strip; `on_findings="refuse"` raises `HiddenTextRefused` without calling the inner function; `report` scans only. Unicode tag characters are decoded to ASCII as `tag_payload` on the receipt (`extract_tag_payload`). CLI: `fuckmark guard` (`--json`, `--refuse`, `--report`, `--receipt`, `--select`). HTTP: `POST /api/guard`. JS: `editors/vscode/guard.js`. This does not detect semantic prompt injection. Default categories are the shared `SECURITY_SCAN_CATEGORIES` also used by `fuckmark lint`. Reference: `docs/guard.md`.
- Hidden-scan `format` classification uses the same explicit Cf ranges as `editors/vscode/scan.js` (including U+13430-U+1343F and U+0890-U+0891) instead of only `unicodedata.category`, so Python 3.11 (Unicode 14) agrees with the JS port and with Python 3.12+ (Unicode 15).
- Hidden-scan findings now carry **context**, **severity**, and a one-line **why/remedy**. A bidi override inside an identifier is `critical` (Trojan Source); a ZWJ inside an emoji cluster is `info`. Lint JSON, `--scan` reports, and the VS Code hover/Problems panel all show those fields. Diagnostic rank: `critical` is Error, `high` is Warning.
-- Frozen classification spec `fuckmark-hidden-scan-v1` under `specs/` (`protocol.md`, `vectors.json`, `freeze.json`). Conformance vectors are codepoint arrays only; `tests/test_hidden_scan_spec.py` replays them on the Python engine and on `editors/vscode/scan.js`.
+- Frozen classification spec `fuckmark-hidden-scan-v1` under `specs/` (`protocol.md`, `vectors.json`, `freeze.json`). Conformance vectors are codepoint arrays only; `tests/test_hidden_scan_spec.py` replays them on the Python engine and on `editors/vscode/scan.js`. Freeze record **v3** adds language-aware comment/string roles: a bidi override in a comment or string is `critical` even next to emoji, `#` is a comment only for `python`, and `//` after `:` is not a comment.
+- Language-aware Trojan Source analysis and autofix. `scan_hidden_characters(..., language=)` (and lint, from the file suffix) classifies source as `comment` / `string` / code. `autofix_trojan_source` / `autofixTrojanSource` strip only `bidi_control`.
+- Zero-install hidden-Unicode scan page: `docs/scan.html` (packaged as `fuckmark/webui/scan.html`, engine copy `scan.js`). `fuckmark web` serves it at `/scan.html`. After deploy: `mark.q1z.org/scan.html`. Scanning runs in the browser; a local Python `/api/scan` confirm is optional when the CLI server is up.
- `fuckmark normalize` (`fuckmark-normalize-v1`): NFC, optional identifier lookalike fold (UTS #39-inspired subset, not a full confusable map), then strip of the security category set, plus a JSON receipt (`input_sha256` / `output_sha256` / `steps` / `report_hash`). `--clean` remains the blunt strip with no receipt. HTTP: `POST /api/normalize`. Python: `normalize_text`, `skeleton_fold`. Reference: `docs/normalize.md`.
- VS Code / Cursor: `fuckmark.cleanOnSave` (default off) strips hidden Unicode on save.
- Frozen confirmation files, hashes, mix mechanisms, and the published wheel are unchanged. Package version stays 0.4.1.
diff --git a/README.md b/README.md
index a487f75f..69d417bc 100644
--- a/README.md
+++ b/README.md
@@ -26,6 +26,7 @@ Opens a local page at `http://127.0.0.1:8765/mark.html`. Paste text in the brows
`fuckmark web` also serves a local Python API. Detect and strip POST to `/api/remove-marks` and run the same engine as `fuckmark --detect`. Opening `mark.html` with `file://` or the static website has no Python process, so those paths keep the in-browser fallback.
- Product paste UI source: [`docs/mark.html`](docs/mark.html) (packaged as `fuckmark/webui/mark.html`) — after deploy: [mark.q1z.org/mark.html](https://mark.q1z.org/mark.html)
+- Hidden-Unicode scan (no install): [`docs/scan.html`](docs/scan.html) — `file://` works; `fuckmark web` serves `/scan.html`; after deploy: [mark.q1z.org/scan.html](https://mark.q1z.org/scan.html)
- Research demo: [`docs/demo.html`](docs/demo.html) — `file://` works; after deploy: [mark.q1z.org/demo.html](https://mark.q1z.org/demo.html)
`mark.html` runs a local closed-set FuckMark scan, strips approved insertions when found, and on a miss shows an English no-watermark card with contact to `Fhelp@q1z.org`. Under `fuckmark web` that scan is the Python detector; the JS copy is the fallback.
@@ -132,7 +133,7 @@ printf 'if (x != \u202eadmin\u202c) {\n' | .venv/bin/fuckmark --scan
.venv/bin/fuckmark --clean --file suspect.txt -o clean.txt
```
-`--scan` prints a human report by default, a machine line with `-q`, and a `fuckmark-scan ...` status line to stderr with `--status`. Findings include context (`identifier` / `emoji` / `string` / `prose`) and severity (`critical` / `high` / `medium` / `info`): a bidi override inside an identifier is critical; a ZWJ inside an emoji cluster is info. `--clean` removes every flagged category (including FuckMark's own carriers, so `--clean` reverses a mix back to the visible text) and reports how many characters it removed. The same engine is available in the browser tool via `POST /api/scan`, and in Python through `scan_hidden_characters`, `clean_hidden_characters`, and `classify_hidden_codepoint`. Frozen spec: [`specs/fuckmark-hidden-scan-v1.protocol.md`](specs/fuckmark-hidden-scan-v1.protocol.md).
+`--scan` prints a human report by default, a machine line with `-q`, and a `fuckmark-scan ...` status line to stderr with `--status`. Findings include context (`identifier` / `comment` / `string` / `emoji` / `prose`) and severity (`critical` / `high` / `medium` / `info`): a bidi override inside an identifier, comment, or string is critical (Trojan Source); a ZWJ inside an emoji cluster is info. Lint infers comment syntax from the file suffix (`javascript`, `python`, `sql`, `html`). `--clean` removes every flagged category (including FuckMark's own carriers, so `--clean` reverses a mix back to the visible text) and reports how many characters it removed. `autofix_trojan_source` strips only bidirectional controls. The same engine is in the browser at [`docs/scan.html`](docs/scan.html) (`fuckmark web` → `/scan.html`), via `POST /api/scan`, and in Python through `scan_hidden_characters`, `clean_hidden_characters`, and `classify_hidden_codepoint`. Frozen spec: [`specs/fuckmark-hidden-scan-v1.protocol.md`](specs/fuckmark-hidden-scan-v1.protocol.md).
Whitespace tab, newline, carriage return, and space are never flagged, and ordinary combining accents (`Mn`) are left alone. `--clean` strips emoji zero-width joiners and variation selectors as well, so pass a category subset to `clean_hidden_characters(...)` in Python if you need to keep emoji sequences intact.
@@ -176,7 +177,7 @@ Binary, non-UTF-8, oversized, and vendored/VCS paths are skipped; `--exclude GLO
## See it in your editor (VS Code / Cursor)
-The [`editors/vscode`](editors/vscode) extension reveals hidden Unicode inline as you read code: a red box and a visible `‹U+202E›` badge on every hidden character, hover text with why it matters, diagnostics in the Problems panel (`critical` as Error), a status-bar count, one-click clean, and optional clean-on-save (`fuckmark.cleanOnSave`). It is zero-dependency plain JavaScript with no build step, and its scanner is a faithful port of `fuckmark-hidden-scan-v1` — pinned to the Python engine across every Unicode codepoint by [`tests/test_vscode_scanner_parity.py`](tests/test_vscode_scanner_parity.py), so the editor and the CLI agree exactly. Load it with `code --extensionDevelopmentPath=editors/vscode` or open the folder and press `F5`.
+The [`editors/vscode`](editors/vscode) extension reveals hidden Unicode inline as you read code: a red box and a visible `‹U+202E›` badge on every hidden character, hover text with why it matters, diagnostics in the Problems panel (`critical` as Error, including Trojan Source in comments and strings), a status-bar count, one-click clean, and optional clean-on-save (`fuckmark.cleanOnSave`). It is zero-dependency plain JavaScript with no build step, and its scanner is a faithful port of `fuckmark-hidden-scan-v1` — pinned to the Python engine across every Unicode codepoint by [`tests/test_vscode_scanner_parity.py`](tests/test_vscode_scanner_parity.py), so the editor and the CLI agree exactly. Load it with `code --extensionDevelopmentPath=editors/vscode` or open the folder and press `F5`.
## Guard model input (LLM prompt-injection smuggling)
diff --git a/docs/cli.md b/docs/cli.md
index 98f3add5..d7919a20 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -116,9 +116,9 @@ Existing files are read as UTF-8 bytes with no newline conversion. LF, CRLF, CR,
`--scan` and `--clean` are the defensive inverse of the mix. The scan is general, not FuckMark-only. Flagged categories: `bidi_control` (Trojan Source, CVE-2021-42574), `zero_width`, `variation_selector`, `tag` (hidden-text / prompt-injection smuggling), `enclosing_mark`, `line_separator`, `deprecated` (interlinear annotation and deprecated format controls), `format` (other `Cf`), `control` (C0/C1), `private_use`, `noncharacter`, and `surrogate`. Tab, newline, carriage return, and space are never flagged; ordinary combining accents (`Mn`) are left alone.
-Findings include context (`identifier`, `emoji`, `string`, `prose`) and severity (`critical`, `high`, `medium`, `info`). A bidi override next to an identifier is `critical`; a ZWJ inside an emoji cluster is `info`. The frozen table is [`specs/fuckmark-hidden-scan-v1.protocol.md`](../specs/fuckmark-hidden-scan-v1.protocol.md).
+Findings include context (`identifier`, `comment`, `string`, `emoji`, `prose`) and severity (`critical`, `high`, `medium`, `info`). A bidi override in an identifier, comment, or string is `critical` (Trojan Source); a ZWJ inside an emoji cluster is `info`. Comment syntax follows `language` (`auto` treats `//` and `/* */` as comments; `python` also treats `#`; `sql` treats `--`; `html` treats ``). The frozen table is [`specs/fuckmark-hidden-scan-v1.protocol.md`](../specs/fuckmark-hidden-scan-v1.protocol.md).
-`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. The browser tool exposes the same engine at `POST /api/scan`.
+`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. `autofix_trojan_source` strips only `bidi_control`. The browser scan page is `docs/scan.html` (`fuckmark web` serves `/scan.html`); the API is `POST /api/scan` with optional `"language"`.
### `fuckmark normalize`
@@ -143,7 +143,7 @@ fuckmark guard --refuse --receipt --json < messages.json
### `fuckmark web`
-Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`.
+Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`. Hidden-Unicode reveal is `/scan.html` (same engine as `--scan`; zero-install copy at `docs/scan.html`).
```text
fuckmark web
diff --git a/docs/lint.md b/docs/lint.md
index dc46f825..04a32c32 100644
--- a/docs/lint.md
+++ b/docs/lint.md
@@ -34,7 +34,9 @@ Pass `--select all` to include `variation_selector`, `enclosing_mark`,
`line_separator`, `deprecated`, `format`, and `private_use`, or pass an explicit
comma-separated subset. See [`cli.md`](cli.md) for the full category list.
JSON locations include `context`, `severity`, `why`, and `remedy` from
-`fuckmark-hidden-scan-v1`.
+`fuckmark-hidden-scan-v1`. Context uses the file suffix: `.js`/`.ts` get
+JavaScript comments (`//`, `/* */`), `.py` gets `#` comments, `.sql` gets `--`,
+`.html` gets ``. A bidi override in a comment or string is `critical`.
### What is skipped
diff --git a/docs/scan.html b/docs/scan.html
new file mode 100644
index 00000000..0b007977
--- /dev/null
+++ b/docs/scan.html
@@ -0,0 +1,606 @@
+
+
+
Paste source or a prompt. This page reveals Trojan Source bidi overrides, zero-width characters, and Unicode tag smuggling without sending the text anywhere.
+
+
+
+
+
+
+ Engine: browser
+
+
+
+
+
+
+
+
+
+
+
Paste text to scan.
+
+
+
+ Scanning is fuckmark-hidden-scan-v1. Copy uses the security category set
+ (bidi_control, zero_width, tag, control,
+ noncharacter, surrogate). Fix Trojan Source strips only
+ bidirectional controls. Nothing is uploaded from a static page or file://.
+ Under fuckmark web an optional local Python API can confirm the same verdicts.
+ Contact Fhelp@q1z.org.
+
+
+
+
+
+
+
diff --git a/docs/scan.js b/docs/scan.js
new file mode 100644
index 00000000..f28c001f
--- /dev/null
+++ b/docs/scan.js
@@ -0,0 +1,569 @@
+// Faithful JavaScript port of fuckmark-hidden-scan-v1 (fuckmark/product/scan.py).
+// Kept in lockstep with the Python engine by tests/test_vscode_scanner_parity.py.
+
+"use strict";
+
+const SCAN_ALGORITHM_VERSION = "fuckmark-hidden-scan-v1";
+
+const CATEGORY_BIDI_CONTROL = "bidi_control";
+const CATEGORY_ZERO_WIDTH = "zero_width";
+const CATEGORY_VARIATION_SELECTOR = "variation_selector";
+const CATEGORY_TAG = "tag";
+const CATEGORY_ENCLOSING_MARK = "enclosing_mark";
+const CATEGORY_LINE_SEPARATOR = "line_separator";
+const CATEGORY_DEPRECATED = "deprecated";
+const CATEGORY_FORMAT = "format";
+const CATEGORY_CONTROL = "control";
+const CATEGORY_PRIVATE_USE = "private_use";
+const CATEGORY_NONCHARACTER = "noncharacter";
+const CATEGORY_SURROGATE = "surrogate";
+
+const SCAN_CATEGORIES = [
+ CATEGORY_BIDI_CONTROL,
+ CATEGORY_ZERO_WIDTH,
+ CATEGORY_VARIATION_SELECTOR,
+ CATEGORY_TAG,
+ CATEGORY_ENCLOSING_MARK,
+ CATEGORY_LINE_SEPARATOR,
+ CATEGORY_DEPRECATED,
+ CATEGORY_FORMAT,
+ CATEGORY_CONTROL,
+ CATEGORY_PRIVATE_USE,
+ CATEGORY_NONCHARACTER,
+ CATEGORY_SURROGATE,
+];
+
+const CATEGORY_DESCRIPTIONS = {
+ [CATEGORY_BIDI_CONTROL]: "bidirectional override/isolate (Trojan Source reordering)",
+ [CATEGORY_ZERO_WIDTH]: "zero-width or invisible spacing character",
+ [CATEGORY_VARIATION_SELECTOR]: "variation selector (glyph/steganography carrier)",
+ [CATEGORY_TAG]: "Unicode tag character (hidden text / prompt-injection smuggling)",
+ [CATEGORY_ENCLOSING_MARK]: "enclosing combining mark (alters surrounding glyph)",
+ [CATEGORY_LINE_SEPARATOR]: "line/paragraph separator (breaks parsers, invisible)",
+ [CATEGORY_DEPRECATED]: "deprecated format control or interlinear annotation",
+ [CATEGORY_FORMAT]: "general Unicode format control (Cf)",
+ [CATEGORY_CONTROL]: "C0/C1 control character",
+ [CATEGORY_PRIVATE_USE]: "private-use codepoint (renderer-defined)",
+ [CATEGORY_NONCHARACTER]: "Unicode noncharacter or unassigned codepoint",
+ [CATEGORY_SURROGATE]: "lone surrogate codepoint",
+};
+
+const DEFAULT_SECURITY_CATEGORIES = [
+ CATEGORY_BIDI_CONTROL,
+ CATEGORY_ZERO_WIDTH,
+ CATEGORY_TAG,
+ CATEGORY_CONTROL,
+ CATEGORY_NONCHARACTER,
+ CATEGORY_SURROGATE,
+];
+
+const TROJAN_SOURCE_CATEGORIES = [CATEGORY_BIDI_CONTROL];
+
+const ALLOWED_WHITESPACE = new Set([0x09, 0x0a, 0x0d, 0x20]);
+
+function inRange(cp, lo, hi) {
+ return cp >= lo && cp <= hi;
+}
+
+const BIDI_CONTROL = new Set([0x061c, 0x200e, 0x200f]);
+for (let cp = 0x202a; cp <= 0x202e; cp += 1) BIDI_CONTROL.add(cp);
+for (let cp = 0x2066; cp <= 0x2069; cp += 1) BIDI_CONTROL.add(cp);
+
+const DEPRECATED = new Set([0xfff9, 0xfffa, 0xfffb]);
+for (let cp = 0x206a; cp <= 0x206f; cp += 1) DEPRECATED.add(cp);
+
+const ZERO_WIDTH = new Set([
+ 0x00ad, 0x034f, 0x115f, 0x1160, 0x17b4, 0x17b5, 0x180e, 0x200b, 0x200c,
+ 0x200d, 0x2060, 0x2061, 0x2062, 0x2063, 0x2064, 0x3164, 0xfeff, 0xffa0,
+]);
+
+function isVariationSelector(cp) {
+ return inRange(cp, 0xfe00, 0xfe0f) || inRange(cp, 0xe0100, 0xe01ef);
+}
+
+function isTag(cp) {
+ return inRange(cp, 0xe0000, 0xe007f);
+}
+
+function isNoncharacter(cp) {
+ if (inRange(cp, 0xfdd0, 0xfdef)) return true;
+ const low = cp & 0xffff;
+ return low === 0xfffe || low === 0xffff;
+}
+
+function isEnclosingMark(cp) {
+ // Unicode general category Me. Enumerated to avoid a runtime Unicode DB.
+ return (
+ inRange(cp, 0x0488, 0x0489) ||
+ cp === 0x1abe ||
+ inRange(cp, 0x20dd, 0x20e0) ||
+ inRange(cp, 0x20e2, 0x20e4) ||
+ inRange(cp, 0xa670, 0xa672)
+ );
+}
+
+function isControl(cp) {
+ return inRange(cp, 0x00, 0x1f) || inRange(cp, 0x7f, 0x9f);
+}
+
+function isLineSeparator(cp) {
+ return cp === 0x2028 || cp === 0x2029;
+}
+
+// General category Cf ranges (Unicode 15-era), excluding codepoints already
+// classified above. Mirrors what unicodedata.category returns as "Cf".
+const FORMAT_RANGES = [
+ [0x0600, 0x0605],
+ [0x06dd, 0x06dd],
+ [0x070f, 0x070f],
+ [0x0890, 0x0891],
+ [0x08e2, 0x08e2],
+ [0x110bd, 0x110bd],
+ [0x110cd, 0x110cd],
+ [0x13430, 0x1343f],
+ [0x1bca0, 0x1bca3],
+ [0x1d173, 0x1d17a],
+];
+
+function isFormat(cp) {
+ for (const [lo, hi] of FORMAT_RANGES) {
+ if (inRange(cp, lo, hi)) return true;
+ }
+ return false;
+}
+
+function isPrivateUse(cp) {
+ return (
+ inRange(cp, 0xe000, 0xf8ff) ||
+ inRange(cp, 0xf0000, 0xffffd) ||
+ inRange(cp, 0x100000, 0x10fffd)
+ );
+}
+
+function classify(cp) {
+ if (ALLOWED_WHITESPACE.has(cp)) return null;
+ if (BIDI_CONTROL.has(cp)) return CATEGORY_BIDI_CONTROL;
+ if (DEPRECATED.has(cp)) return CATEGORY_DEPRECATED;
+ if (ZERO_WIDTH.has(cp)) return CATEGORY_ZERO_WIDTH;
+ if (isVariationSelector(cp)) return CATEGORY_VARIATION_SELECTOR;
+ if (isTag(cp)) return CATEGORY_TAG;
+ if (isNoncharacter(cp)) return CATEGORY_NONCHARACTER;
+ if (cp >= 0xd800 && cp <= 0xdfff) return CATEGORY_SURROGATE;
+ if (isEnclosingMark(cp)) return CATEGORY_ENCLOSING_MARK;
+ if (isLineSeparator(cp)) return CATEGORY_LINE_SEPARATOR;
+ if (isControl(cp)) return CATEGORY_CONTROL;
+ if (isFormat(cp)) return CATEGORY_FORMAT;
+ if (isPrivateUse(cp)) return CATEGORY_PRIVATE_USE;
+ return null;
+}
+
+function codepointToken(cp) {
+ return "U+" + cp.toString(16).toUpperCase().padStart(4, "0");
+}
+
+function isEmojiish(cp) {
+ if (cp < 0) return false;
+ if (cp >= 0x1f1e6 && cp <= 0x1f1ff) return true;
+ if (cp >= 0x1f000 && cp <= 0x1faff) return true;
+ if (cp >= 0x2600 && cp <= 0x27bf) return true;
+ return cp === 0x200d || [0x00a9, 0x00ae, 0x203c, 0x2049, 0x2122, 0x2139, 0x3030, 0x303d, 0x3297, 0x3299].includes(cp);
+}
+
+function isIdentChar(ch) {
+ return ch === "_" || /\p{L}|\p{N}/u.test(ch);
+}
+
+function neighborBefore(text, offset) {
+ if (offset <= 0) return { cp: -1, ch: "" };
+ if (offset >= 2) {
+ const cp = text.codePointAt(offset - 2);
+ if (cp > 0xffff) return { cp, ch: String.fromCodePoint(cp) };
+ }
+ const cp = text.codePointAt(offset - 1);
+ return { cp, ch: String.fromCodePoint(cp) };
+}
+
+function neighborAfter(text, offset, length) {
+ const nextIndex = offset + length;
+ if (nextIndex >= text.length) return { cp: -1, ch: "" };
+ const cp = text.codePointAt(nextIndex);
+ return { cp, ch: String.fromCodePoint(cp) };
+}
+
+const LANGUAGE_ALIASES = {
+ auto: "auto",
+ javascript: "javascript",
+ js: "javascript",
+ ts: "javascript",
+ typescript: "javascript",
+ jsx: "javascript",
+ tsx: "javascript",
+ c: "c",
+ h: "c",
+ cc: "c",
+ cpp: "c",
+ cxx: "c",
+ java: "c",
+ go: "c",
+ rs: "c",
+ rust: "c",
+ cs: "c",
+ css: "c",
+ jsonc: "c",
+ python: "python",
+ py: "python",
+ pyi: "python",
+ hash: "python",
+ sh: "python",
+ bash: "python",
+ zsh: "python",
+ shell: "python",
+ shellscript: "python",
+ yaml: "python",
+ yml: "python",
+ rb: "python",
+ ruby: "python",
+ toml: "python",
+ html: "html",
+ htm: "html",
+ xml: "html",
+ sql: "sql",
+};
+
+function normalizeLanguage(language) {
+ if (!language) return "auto";
+ const key = String(language).trim().toLowerCase();
+ return LANGUAGE_ALIASES[key] || "auto";
+}
+
+function slashComments(language) {
+ return language === "auto" || language === "javascript" || language === "c" || language === "sql";
+}
+
+function blockComments(language) {
+ return slashComments(language);
+}
+
+function hashComments(language) {
+ return language === "python";
+}
+
+function sqlLineComments(language) {
+ return language === "sql";
+}
+
+function htmlComments(language) {
+ return language === "html";
+}
+
+function sourceRoles(text, language) {
+ const lang = normalizeLanguage(language);
+ const roles = new Array(text.length).fill("code");
+ let index = 0;
+ let inString = "";
+ let escape = false;
+ while (index < text.length) {
+ const ch = text[index];
+ if (inString) {
+ roles[index] = "string";
+ if (escape) {
+ escape = false;
+ index += 1;
+ continue;
+ }
+ if (ch === "\\" && index + 1 < text.length) {
+ escape = true;
+ index += 1;
+ continue;
+ }
+ if (ch === inString) inString = "";
+ index += 1;
+ continue;
+ }
+ const pair = text.slice(index, index + 2);
+ if (slashComments(lang) && pair === "//" && !(index > 0 && text[index - 1] === ":")) {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (sqlLineComments(lang) && pair === "--") {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (blockComments(lang) && pair === "/*") {
+ const close = text.indexOf("*/", index + 2);
+ const stop = close >= 0 ? close + 2 : text.length;
+ for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment";
+ index = stop;
+ continue;
+ }
+ if (hashComments(lang) && ch === "#") {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (htmlComments(lang) && text.slice(index, index + 4) === "", index + 4);
+ const stop = close >= 0 ? close + 3 : text.length;
+ for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment";
+ index = stop;
+ continue;
+ }
+ if (ch === '"' || ch === "'" || ch === "`") {
+ inString = ch;
+ roles[index] = "string";
+ index += 1;
+ continue;
+ }
+ index += 1;
+ }
+ return roles;
+}
+
+function classifyContext(text, offset, role, category) {
+ const length = text.codePointAt(offset) > 0xffff ? 2 : 1;
+ const prev = neighborBefore(text, offset);
+ const next = neighborAfter(text, offset, length);
+ if ((role === "comment" || role === "string") && category === CATEGORY_BIDI_CONTROL) {
+ return role;
+ }
+ if (
+ isEmojiish(prev.cp) ||
+ isEmojiish(next.cp) ||
+ (prev.cp >= 0xfe00 && prev.cp <= 0xfe0f) ||
+ (next.cp >= 0xfe00 && next.cp <= 0xfe0f)
+ ) {
+ return "emoji";
+ }
+ if (role === "comment") return "comment";
+ if (role === "string") return "string";
+ if (isIdentChar(prev.ch) || isIdentChar(next.ch)) return "identifier";
+ return "prose";
+}
+
+function scoreSeverity(category, context) {
+ if (category === CATEGORY_TAG) return "critical";
+ if (category === CATEGORY_BIDI_CONTROL) {
+ if (context === "identifier" || context === "comment" || context === "string") return "critical";
+ return "high";
+ }
+ if (category === CATEGORY_ZERO_WIDTH) {
+ if (context === "emoji") return "info";
+ if (context === "identifier") return "high";
+ return "medium";
+ }
+ if (category === CATEGORY_VARIATION_SELECTOR) return context === "emoji" ? "info" : "medium";
+ if (category === CATEGORY_CONTROL || category === CATEGORY_NONCHARACTER || category === CATEGORY_SURROGATE) {
+ return "high";
+ }
+ return "medium";
+}
+
+function explainFinding(category, context, severity) {
+ if (category === CATEGORY_BIDI_CONTROL && context === "identifier") {
+ return {
+ why: "Bidirectional override sits inside an identifier, so the glyphs can read differently than the bytes (Trojan Source).",
+ remedy: "Strip the bidi control and keep the identifier left-to-right.",
+ };
+ }
+ if (category === CATEGORY_BIDI_CONTROL && context === "comment") {
+ return {
+ why: "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).",
+ remedy: "Strip the bidi control from the comment.",
+ };
+ }
+ if (category === CATEGORY_BIDI_CONTROL && context === "string") {
+ return {
+ why: "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).",
+ remedy: "Strip the bidi control from the string.",
+ };
+ }
+ if (category === CATEGORY_BIDI_CONTROL) {
+ return {
+ why: "Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).",
+ remedy: "Strip U+202A-U+202E / U+2066-U+2069 and rewrite the text left-to-right.",
+ };
+ }
+ if (category === CATEGORY_TAG) {
+ return {
+ why: "Unicode tag characters encode a second ASCII string that models read and humans do not.",
+ remedy: "Strip U+E0020-U+E007F; inspect tag_payload for the smuggled text.",
+ };
+ }
+ if (category === CATEGORY_ZERO_WIDTH && context === "emoji") {
+ return {
+ why: "Zero-width joiner or invisible mark inside an emoji cluster; usually a legitimate emoji sequence.",
+ remedy: "Leave emoji ZWJ sequences unless you are sanitizing for a security boundary.",
+ };
+ }
+ if (category === CATEGORY_ZERO_WIDTH && context === "identifier") {
+ return {
+ why: "Zero-width character splits an identifier, breaking search and some compilers while looking unchanged.",
+ remedy: "Strip the zero-width character from the identifier.",
+ };
+ }
+ if (category === CATEGORY_ZERO_WIDTH) {
+ return {
+ why: "Invisible spacing or joining character that changes the byte stream without changing the glyphs.",
+ remedy: "Strip the zero-width character.",
+ };
+ }
+ if (category === CATEGORY_VARIATION_SELECTOR && context === "emoji") {
+ return {
+ why: "Variation selector tunes an emoji glyph; usually benign.",
+ remedy: "Keep emoji variation selectors unless you are stripping all hidden marks.",
+ };
+ }
+ if (severity === "high") {
+ return {
+ why: "Hidden or non-text codepoint that should not appear in ordinary source or prompts.",
+ remedy: "Strip the character.",
+ };
+ }
+ return {
+ why: CATEGORY_DESCRIPTIONS[category] || "Hidden or format codepoint that is invisible or renderer-defined.",
+ remedy: "Strip the character if this text crosses a trust boundary.",
+ };
+}
+
+function scanText(text, categories, language) {
+ const selected = categories ? new Set(categories) : null;
+ const roles = sourceRoles(text, language);
+ const counts = {};
+ const findings = [];
+ let total = 0;
+ let offset = 0;
+ while (offset < text.length) {
+ const cp = text.codePointAt(offset);
+ const length = cp > 0xffff ? 2 : 1;
+ const category = classify(cp);
+ if (category !== null && (selected === null || selected.has(category))) {
+ total += 1;
+ counts[category] = (counts[category] || 0) + 1;
+ const context = classifyContext(text, offset, roles[offset] || "code", category);
+ const severity = scoreSeverity(category, context);
+ const explained = explainFinding(category, context, severity);
+ findings.push({
+ offset,
+ length,
+ codepoint: cp,
+ category,
+ context,
+ severity,
+ why: explained.why,
+ remedy: explained.remedy,
+ });
+ }
+ offset += length;
+ }
+ return { total, counts, findings };
+}
+
+function cleanText(text, categories) {
+ const selected = categories ? new Set(categories) : null;
+ let out = "";
+ let removed = 0;
+ let offset = 0;
+ while (offset < text.length) {
+ const cp = text.codePointAt(offset);
+ const length = cp > 0xffff ? 2 : 1;
+ const category = classify(cp);
+ if (category !== null && (selected === null || selected.has(category))) {
+ removed += 1;
+ } else {
+ out += text.slice(offset, offset + length);
+ }
+ offset += length;
+ }
+ return { cleaned: out, removed };
+}
+
+function autofixTrojanSource(text) {
+ return cleanText(text, TROJAN_SOURCE_CATEGORIES);
+}
+
+function languageFromDocumentId(languageId) {
+ return normalizeLanguage(languageId);
+}
+
+const LANGUAGE_BY_SUFFIX = {
+ ".js": "javascript",
+ ".mjs": "javascript",
+ ".cjs": "javascript",
+ ".ts": "javascript",
+ ".jsx": "javascript",
+ ".tsx": "javascript",
+ ".c": "c",
+ ".h": "c",
+ ".hh": "c",
+ ".cc": "c",
+ ".cpp": "c",
+ ".cxx": "c",
+ ".java": "c",
+ ".go": "c",
+ ".rs": "c",
+ ".cs": "c",
+ ".css": "c",
+ ".py": "python",
+ ".pyi": "python",
+ ".sh": "python",
+ ".bash": "python",
+ ".zsh": "python",
+ ".yaml": "python",
+ ".yml": "python",
+ ".rb": "python",
+ ".toml": "python",
+ ".html": "html",
+ ".htm": "html",
+ ".xml": "html",
+ ".sql": "sql",
+};
+
+function languageFromPath(path) {
+ const name = String(path || "");
+ const slash = Math.max(name.lastIndexOf("/"), name.lastIndexOf("\\"));
+ const base = slash >= 0 ? name.slice(slash + 1) : name;
+ const dot = base.lastIndexOf(".");
+ if (dot <= 0) return "auto";
+ const suffix = base.slice(dot).toLowerCase();
+ return LANGUAGE_BY_SUFFIX[suffix] || "auto";
+}
+
+const FuckMarkScan = {
+ SCAN_ALGORITHM_VERSION,
+ SCAN_CATEGORIES,
+ CATEGORY_DESCRIPTIONS,
+ DEFAULT_SECURITY_CATEGORIES,
+ TROJAN_SOURCE_CATEGORIES,
+ classify,
+ classifyContext,
+ scoreSeverity,
+ normalizeLanguage,
+ languageFromDocumentId,
+ languageFromPath,
+ sourceRoles,
+ codepointToken,
+ scanText,
+ cleanText,
+ autofixTrojanSource,
+};
+
+if (typeof module !== "undefined" && module.exports) {
+ module.exports = FuckMarkScan;
+}
+if (typeof globalThis !== "undefined") {
+ globalThis.FuckMarkScan = FuckMarkScan;
+}
diff --git a/docs/website.md b/docs/website.md
index 21a15aa6..51f4560c 100644
--- a/docs/website.md
+++ b/docs/website.md
@@ -11,6 +11,7 @@ Deploy these static files (repository CI cannot push them; copy after each relea
| Repo file | Public URL |
| --- | --- |
| [`mark.html`](mark.html) | `https://mark.q1z.org/mark.html` (browser tool) |
+| [`scan.html`](scan.html) + [`scan.js`](scan.js) | `https://mark.q1z.org/scan.html` (hidden-Unicode reveal; deploy both files) |
| [`rec.html`](rec.html) | gate that waits for `go.txt`, then opens `mark.html?demo=1` |
| [`demo.html`](demo.html) | `https://mark.q1z.org/demo.html` (research / CLI samples) |
@@ -27,10 +28,16 @@ Local beginners can run `fuckmark web` after install. That serves the same `mark
- `GET /api/health` reports `backend: "python"`
- `POST /api/remove-marks` with `{ "text": "..." }` calls `detect_fuckmark_insertions` and `project_visible_v1`
-- Local-only extras on `fuckmark web`: `POST /api/scan`, `POST /api/guard`, `POST /api/normalize`
+- Local-only extras on `fuckmark web`: `POST /api/scan`, `POST /api/guard`, `POST /api/normalize`, and `/scan.html`
The page uses that API when the local server is up. Static `https://mark.q1z.org/mark.html` and `file://` have no Python process, so those deploys keep the in-browser scan.
+### `scan.html` (hidden-Unicode reveal)
+
+Static, JS-first. Paste text or open a file; the page classifies hidden characters with `fuckmark-hidden-scan-v1` (`scan.js`, the same port as the editor extension). Language (`auto`, JavaScript/C-like, Python, SQL, HTML) selects comment syntax so Trojan Source commenting-out and stretched-string encodings rank `critical`. Copy uses the security category set. **Fix Trojan Source** strips only bidirectional controls.
+
+Deploy `scan.html` and `scan.js` next to each other. `file://` works. Under `fuckmark web`, an optional checkbox uses `POST /api/scan` on the local Python engine; text still does not leave the machine. Do not embed raw hidden characters in the HTML file; examples are built with `String.fromCodePoint`.
+
### `demo.html` (research walkthrough)
Keep it a static file that works from `file://` with baked CLI samples. The live page must show:
@@ -43,7 +50,7 @@ Keep it a static file that works from `file://` with baked CLI samples. The live
If none are found, the demo offers contact at `Fhelp@q1z.org`. Do not present demo numbers as a guarantee for arbitrary user text.
-Homepage copy should link `mark.html` (tool) and `demo.html` (evidence) before install instructions.
+Homepage copy should link `mark.html` (tool), `scan.html` (hidden Unicode), and `demo.html` (evidence) before install instructions.
## What the product does
diff --git a/editors/vscode/README.md b/editors/vscode/README.md
index bc50774c..f59878dd 100644
--- a/editors/vscode/README.md
+++ b/editors/vscode/README.md
@@ -17,8 +17,9 @@ It flags the classes that read as ordinary text but are not:
- **Inline reveal:** every hidden character gets a red box and a visible
`‹U+202E›` badge, even when the character itself paints nothing.
- **Problems panel:** each hit is a diagnostic. `critical` (Trojan Source in
- an identifier, tag smuggling) is an Error; `high` and other security classes
- are warnings; the rest are info. Hover text states why the character matters.
+ an identifier, comment, or string; tag smuggling) is an Error; `high` and
+ other security classes are warnings; the rest are info. Hover text states why
+ the character matters. Comment syntax follows the editor language id.
- **Status bar:** a live hidden-character count. Click it to clean the file.
- **Commands** (Command Palette):
- `FuckMark: Clean hidden Unicode in file`
diff --git a/editors/vscode/extension.js b/editors/vscode/extension.js
index 83f4bd61..48d57c45 100644
--- a/editors/vscode/extension.js
+++ b/editors/vscode/extension.js
@@ -8,6 +8,7 @@ const {
codepointToken,
scanText,
cleanText,
+ languageFromDocumentId,
} = require("./scan.js");
let decorationType;
@@ -113,7 +114,7 @@ function refresh(editor) {
editor.setDecorations(decorationType, []);
return { total: 0 };
}
- const result = scanText(document.getText());
+ const result = scanText(document.getText(), null, languageFromDocumentId(document.languageId));
diagnostics.set(document.uri, buildDiagnostics(document, result.findings));
editor.setDecorations(decorationType, revealEnabled ? buildDecorations(document, result.findings) : []);
return result;
diff --git a/editors/vscode/scan.js b/editors/vscode/scan.js
index c2351b96..f28c001f 100644
--- a/editors/vscode/scan.js
+++ b/editors/vscode/scan.js
@@ -57,6 +57,8 @@ const DEFAULT_SECURITY_CATEGORIES = [
CATEGORY_SURROGATE,
];
+const TROJAN_SOURCE_CATEGORIES = [CATEGORY_BIDI_CONTROL];
+
const ALLOWED_WHITESPACE = new Set([0x09, 0x0a, 0x0d, 0x20]);
function inRange(cp, lo, hi) {
@@ -188,10 +190,156 @@ function neighborAfter(text, offset, length) {
return { cp, ch: String.fromCodePoint(cp) };
}
-function classifyContext(text, offset, inString) {
+const LANGUAGE_ALIASES = {
+ auto: "auto",
+ javascript: "javascript",
+ js: "javascript",
+ ts: "javascript",
+ typescript: "javascript",
+ jsx: "javascript",
+ tsx: "javascript",
+ c: "c",
+ h: "c",
+ cc: "c",
+ cpp: "c",
+ cxx: "c",
+ java: "c",
+ go: "c",
+ rs: "c",
+ rust: "c",
+ cs: "c",
+ css: "c",
+ jsonc: "c",
+ python: "python",
+ py: "python",
+ pyi: "python",
+ hash: "python",
+ sh: "python",
+ bash: "python",
+ zsh: "python",
+ shell: "python",
+ shellscript: "python",
+ yaml: "python",
+ yml: "python",
+ rb: "python",
+ ruby: "python",
+ toml: "python",
+ html: "html",
+ htm: "html",
+ xml: "html",
+ sql: "sql",
+};
+
+function normalizeLanguage(language) {
+ if (!language) return "auto";
+ const key = String(language).trim().toLowerCase();
+ return LANGUAGE_ALIASES[key] || "auto";
+}
+
+function slashComments(language) {
+ return language === "auto" || language === "javascript" || language === "c" || language === "sql";
+}
+
+function blockComments(language) {
+ return slashComments(language);
+}
+
+function hashComments(language) {
+ return language === "python";
+}
+
+function sqlLineComments(language) {
+ return language === "sql";
+}
+
+function htmlComments(language) {
+ return language === "html";
+}
+
+function sourceRoles(text, language) {
+ const lang = normalizeLanguage(language);
+ const roles = new Array(text.length).fill("code");
+ let index = 0;
+ let inString = "";
+ let escape = false;
+ while (index < text.length) {
+ const ch = text[index];
+ if (inString) {
+ roles[index] = "string";
+ if (escape) {
+ escape = false;
+ index += 1;
+ continue;
+ }
+ if (ch === "\\" && index + 1 < text.length) {
+ escape = true;
+ index += 1;
+ continue;
+ }
+ if (ch === inString) inString = "";
+ index += 1;
+ continue;
+ }
+ const pair = text.slice(index, index + 2);
+ if (slashComments(lang) && pair === "//" && !(index > 0 && text[index - 1] === ":")) {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (sqlLineComments(lang) && pair === "--") {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (blockComments(lang) && pair === "/*") {
+ const close = text.indexOf("*/", index + 2);
+ const stop = close >= 0 ? close + 2 : text.length;
+ for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment";
+ index = stop;
+ continue;
+ }
+ if (hashComments(lang) && ch === "#") {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (htmlComments(lang) && text.slice(index, index + 4) === "", index + 4);
+ const stop = close >= 0 ? close + 3 : text.length;
+ for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment";
+ index = stop;
+ continue;
+ }
+ if (ch === '"' || ch === "'" || ch === "`") {
+ inString = ch;
+ roles[index] = "string";
+ index += 1;
+ continue;
+ }
+ index += 1;
+ }
+ return roles;
+}
+
+function classifyContext(text, offset, role, category) {
const length = text.codePointAt(offset) > 0xffff ? 2 : 1;
const prev = neighborBefore(text, offset);
const next = neighborAfter(text, offset, length);
+ if ((role === "comment" || role === "string") && category === CATEGORY_BIDI_CONTROL) {
+ return role;
+ }
if (
isEmojiish(prev.cp) ||
isEmojiish(next.cp) ||
@@ -200,14 +348,18 @@ function classifyContext(text, offset, inString) {
) {
return "emoji";
}
+ if (role === "comment") return "comment";
+ if (role === "string") return "string";
if (isIdentChar(prev.ch) || isIdentChar(next.ch)) return "identifier";
- if (inString) return "string";
return "prose";
}
function scoreSeverity(category, context) {
if (category === CATEGORY_TAG) return "critical";
- if (category === CATEGORY_BIDI_CONTROL) return context === "identifier" ? "critical" : "high";
+ if (category === CATEGORY_BIDI_CONTROL) {
+ if (context === "identifier" || context === "comment" || context === "string") return "critical";
+ return "high";
+ }
if (category === CATEGORY_ZERO_WIDTH) {
if (context === "emoji") return "info";
if (context === "identifier") return "high";
@@ -227,6 +379,18 @@ function explainFinding(category, context, severity) {
remedy: "Strip the bidi control and keep the identifier left-to-right.",
};
}
+ if (category === CATEGORY_BIDI_CONTROL && context === "comment") {
+ return {
+ why: "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).",
+ remedy: "Strip the bidi control from the comment.",
+ };
+ }
+ if (category === CATEGORY_BIDI_CONTROL && context === "string") {
+ return {
+ why: "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).",
+ remedy: "Strip the bidi control from the string.",
+ };
+ }
if (category === CATEGORY_BIDI_CONTROL) {
return {
why: "Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).",
@@ -275,27 +439,21 @@ function explainFinding(category, context, severity) {
};
}
-function scanText(text, categories) {
+function scanText(text, categories, language) {
const selected = categories ? new Set(categories) : null;
+ const roles = sourceRoles(text, language);
const counts = {};
const findings = [];
let total = 0;
let offset = 0;
- let inString = "";
while (offset < text.length) {
const cp = text.codePointAt(offset);
const length = cp > 0xffff ? 2 : 1;
- const ch = text.slice(offset, offset + length);
- if (inString) {
- if (ch === inString) inString = "";
- } else if (ch === '"' || ch === "'" || ch === "`") {
- inString = ch;
- }
const category = classify(cp);
if (category !== null && (selected === null || selected.has(category))) {
total += 1;
counts[category] = (counts[category] || 0) + 1;
- const context = classifyContext(text, offset, Boolean(inString));
+ const context = classifyContext(text, offset, roles[offset] || "code", category);
const severity = scoreSeverity(category, context);
const explained = explainFinding(category, context, severity);
findings.push({
@@ -333,15 +491,79 @@ function cleanText(text, categories) {
return { cleaned: out, removed };
}
-module.exports = {
+function autofixTrojanSource(text) {
+ return cleanText(text, TROJAN_SOURCE_CATEGORIES);
+}
+
+function languageFromDocumentId(languageId) {
+ return normalizeLanguage(languageId);
+}
+
+const LANGUAGE_BY_SUFFIX = {
+ ".js": "javascript",
+ ".mjs": "javascript",
+ ".cjs": "javascript",
+ ".ts": "javascript",
+ ".jsx": "javascript",
+ ".tsx": "javascript",
+ ".c": "c",
+ ".h": "c",
+ ".hh": "c",
+ ".cc": "c",
+ ".cpp": "c",
+ ".cxx": "c",
+ ".java": "c",
+ ".go": "c",
+ ".rs": "c",
+ ".cs": "c",
+ ".css": "c",
+ ".py": "python",
+ ".pyi": "python",
+ ".sh": "python",
+ ".bash": "python",
+ ".zsh": "python",
+ ".yaml": "python",
+ ".yml": "python",
+ ".rb": "python",
+ ".toml": "python",
+ ".html": "html",
+ ".htm": "html",
+ ".xml": "html",
+ ".sql": "sql",
+};
+
+function languageFromPath(path) {
+ const name = String(path || "");
+ const slash = Math.max(name.lastIndexOf("/"), name.lastIndexOf("\\"));
+ const base = slash >= 0 ? name.slice(slash + 1) : name;
+ const dot = base.lastIndexOf(".");
+ if (dot <= 0) return "auto";
+ const suffix = base.slice(dot).toLowerCase();
+ return LANGUAGE_BY_SUFFIX[suffix] || "auto";
+}
+
+const FuckMarkScan = {
SCAN_ALGORITHM_VERSION,
SCAN_CATEGORIES,
CATEGORY_DESCRIPTIONS,
DEFAULT_SECURITY_CATEGORIES,
+ TROJAN_SOURCE_CATEGORIES,
classify,
classifyContext,
scoreSeverity,
+ normalizeLanguage,
+ languageFromDocumentId,
+ languageFromPath,
+ sourceRoles,
codepointToken,
scanText,
cleanText,
+ autofixTrojanSource,
};
+
+if (typeof module !== "undefined" && module.exports) {
+ module.exports = FuckMarkScan;
+}
+if (typeof globalThis !== "undefined") {
+ globalThis.FuckMarkScan = FuckMarkScan;
+}
diff --git a/fuckmark/__init__.py b/fuckmark/__init__.py
index 62590022..6da7ab00 100644
--- a/fuckmark/__init__.py
+++ b/fuckmark/__init__.py
@@ -95,6 +95,7 @@
classify_hidden_codepoint,
clean_hidden_characters,
extract_tag_payload,
+ autofix_trojan_source,
scan_dict,
scan_hidden_characters,
scan_human_report,
@@ -383,6 +384,7 @@
"ScanResult",
"classify_hidden_codepoint",
"clean_hidden_characters",
+ "autofix_trojan_source",
"extract_tag_payload",
"scan_dict",
"scan_hidden_characters",
diff --git a/fuckmark/lint.py b/fuckmark/lint.py
index 094a41af..20f5a603 100644
--- a/fuckmark/lint.py
+++ b/fuckmark/lint.py
@@ -18,6 +18,7 @@
normalize_scan_categories,
scan_hidden_characters,
)
+from .product.severity import language_from_path
LINT_ALGORITHM_VERSION = "fuckmark-lint-v1"
@@ -203,7 +204,7 @@ def _write_text_atomic(path: Path, text: str) -> bool:
def _lint_file(path: Path, categories: frozenset[str], text: str, fix: bool) -> FileLintResult:
- scan = scan_hidden_characters(text)
+ scan = scan_hidden_characters(text, language=language_from_path(path))
counts = {name: scan.counts.get(name, 0) for name in scan.active_categories() if name in categories}
total = sum(counts.values())
first_locations = tuple(finding for finding in scan.findings if finding.category in categories)[:10]
diff --git a/fuckmark/product/__init__.py b/fuckmark/product/__init__.py
index d7dacdac..4f17ba5f 100644
--- a/fuckmark/product/__init__.py
+++ b/fuckmark/product/__init__.py
@@ -47,8 +47,10 @@
SCAN_ALGORITHM_VERSION,
SCAN_CATEGORIES,
SECURITY_SCAN_CATEGORIES,
+ TROJAN_SOURCE_CATEGORIES,
HiddenFinding,
ScanResult,
+ autofix_trojan_source,
classify_hidden_codepoint,
clean_hidden_characters,
extract_tag_payload,
@@ -58,6 +60,7 @@
scan_machine_line,
)
from .severity import (
+ CONTEXT_COMMENT,
CONTEXT_EMOJI,
CONTEXT_IDENTIFIER,
CONTEXT_PROSE,
@@ -68,7 +71,10 @@
SEVERITY_MEDIUM,
annotate_finding,
classify_context,
+ language_from_path,
+ normalize_language,
score_severity,
+ source_roles,
)
from .search import PRODUCT_SEARCH_ALGORITHM_VERSION, raw_codepoint_contains, visible_contains
from .visible_projection import (
@@ -95,12 +101,14 @@
"SCAN_ALGORITHM_VERSION",
"SCAN_CATEGORIES",
"SECURITY_SCAN_CATEGORIES",
+ "TROJAN_SOURCE_CATEGORIES",
"NORMALIZE_ALGORITHM_VERSION",
"NormalizeReceipt",
"normalize_payload",
"normalize_receipt_dict",
"normalize_text",
"skeleton_fold",
+ "CONTEXT_COMMENT",
"CONTEXT_EMOJI",
"CONTEXT_IDENTIFIER",
"CONTEXT_PROSE",
@@ -111,7 +119,11 @@
"SEVERITY_MEDIUM",
"annotate_finding",
"classify_context",
+ "language_from_path",
+ "normalize_language",
"score_severity",
+ "source_roles",
+ "autofix_trojan_source",
"HiddenFinding",
"ScanResult",
"classify_hidden_codepoint",
diff --git a/fuckmark/product/scan.py b/fuckmark/product/scan.py
index ad2d8136..a7462e45 100644
--- a/fuckmark/product/scan.py
+++ b/fuckmark/product/scan.py
@@ -5,7 +5,7 @@
from dataclasses import dataclass
from .._validation import require_int
-from .severity import annotate_finding, advance_string_state
+from .severity import annotate_finding, normalize_language, source_roles
from .visible_projection import product_approved_carriers_v1
@@ -48,6 +48,7 @@
CATEGORY_NONCHARACTER,
CATEGORY_SURROGATE,
)
+TROJAN_SOURCE_CATEGORIES = (CATEGORY_BIDI_CONTROL,)
CATEGORY_DESCRIPTIONS = {
CATEGORY_BIDI_CONTROL: "bidirectional override/isolate (Trojan Source reordering)",
@@ -213,6 +214,7 @@ class ScanResult:
findings: tuple[HiddenFinding, ...]
truncated: bool
fuckmark_carriers: int
+ highest_severity: str = ""
@property
def detected(self) -> bool:
@@ -229,31 +231,31 @@ def first_hit(self) -> str:
first = self.findings[0]
return f"U+{first.codepoint:04X}@{first.index}({first.category})"
- @property
- def highest_severity(self) -> str:
- if not self.findings:
- return ""
- return max(self.findings, key=lambda item: _SEVERITY_RANK.get(item.severity, 0)).severity
-
def active_categories(self) -> tuple[str, ...]:
return tuple(name for name in SCAN_CATEGORIES if self.counts.get(name, 0) > 0)
-def scan_hidden_characters(text: str, *, max_findings: int = DEFAULT_MAX_FINDINGS) -> ScanResult:
+def scan_hidden_characters(
+ text: str,
+ *,
+ max_findings: int = DEFAULT_MAX_FINDINGS,
+ language: str | None = None,
+) -> ScanResult:
if not isinstance(text, str):
raise TypeError("text must be a string")
require_int("max_findings", max_findings)
if max_findings < 0:
raise ValueError("max_findings must not be negative")
+ lang = normalize_language(language)
+ roles = source_roles(text, lang)
approved = product_approved_carriers_v1()
counts = {name: 0 for name in SCAN_CATEGORIES}
findings: list[HiddenFinding] = []
total = 0
truncated = False
carriers = 0
- in_string = ""
+ peak = ""
for index, character in enumerate(text):
- in_string = advance_string_state(in_string, character)
code = ord(character)
if code in approved:
carriers += 1
@@ -262,8 +264,10 @@ def scan_hidden_characters(text: str, *, max_findings: int = DEFAULT_MAX_FINDING
continue
total += 1
counts[category] += 1
+ context, severity, why, remedy = annotate_finding(text, index, category, roles[index])
+ if _SEVERITY_RANK.get(severity, -1) > _SEVERITY_RANK.get(peak, -1):
+ peak = severity
if len(findings) < max_findings:
- context, severity, why, remedy = annotate_finding(text, index, category, bool(in_string))
findings.append(
HiddenFinding(
index=index,
@@ -284,6 +288,7 @@ def scan_hidden_characters(text: str, *, max_findings: int = DEFAULT_MAX_FINDING
findings=tuple(findings),
truncated=truncated,
fuckmark_carriers=carriers,
+ highest_severity=peak,
)
@@ -317,6 +322,10 @@ def clean_hidden_characters(
return "".join(kept), removed
+def autofix_trojan_source(text: str) -> tuple[str, int]:
+ return clean_hidden_characters(text, categories=TROJAN_SOURCE_CATEGORIES)
+
+
def scan_machine_line(result: ScanResult) -> str:
found = "yes" if result.detected else "no"
categories = ",".join(f"{name}={result.counts[name]}" for name in result.active_categories())
diff --git a/fuckmark/product/severity.py b/fuckmark/product/severity.py
index 395d4ec4..936dfccc 100644
--- a/fuckmark/product/severity.py
+++ b/fuckmark/product/severity.py
@@ -1,9 +1,12 @@
from __future__ import annotations
+from pathlib import Path
+
CONTEXT_IDENTIFIER = "identifier"
CONTEXT_EMOJI = "emoji"
CONTEXT_STRING = "string"
+CONTEXT_COMMENT = "comment"
CONTEXT_PROSE = "prose"
SEVERITY_CRITICAL = "critical"
@@ -11,9 +14,200 @@
SEVERITY_MEDIUM = "medium"
SEVERITY_INFO = "info"
+ROLE_CODE = "code"
+ROLE_STRING = "string"
+ROLE_COMMENT = "comment"
+
_ZWJ = 0x200D
_VS = frozenset({*range(0xFE00, 0xFE10)})
+_LANGUAGE_ALIASES = {
+ "auto": "auto",
+ "javascript": "javascript",
+ "js": "javascript",
+ "ts": "javascript",
+ "typescript": "javascript",
+ "jsx": "javascript",
+ "tsx": "javascript",
+ "c": "c",
+ "h": "c",
+ "cc": "c",
+ "cpp": "c",
+ "cxx": "c",
+ "java": "c",
+ "go": "c",
+ "rs": "c",
+ "rust": "c",
+ "cs": "c",
+ "css": "c",
+ "jsonc": "c",
+ "python": "python",
+ "py": "python",
+ "pyi": "python",
+ "hash": "python",
+ "sh": "python",
+ "bash": "python",
+ "zsh": "python",
+ "shell": "python",
+ "shellscript": "python",
+ "yaml": "python",
+ "yml": "python",
+ "rb": "python",
+ "ruby": "python",
+ "toml": "python",
+ "html": "html",
+ "htm": "html",
+ "xml": "html",
+ "sql": "sql",
+}
+
+_LANGUAGE_BY_SUFFIX = {
+ ".js": "javascript",
+ ".mjs": "javascript",
+ ".cjs": "javascript",
+ ".ts": "javascript",
+ ".jsx": "javascript",
+ ".tsx": "javascript",
+ ".c": "c",
+ ".h": "c",
+ ".hh": "c",
+ ".cc": "c",
+ ".cpp": "c",
+ ".cxx": "c",
+ ".java": "c",
+ ".go": "c",
+ ".rs": "c",
+ ".cs": "c",
+ ".css": "c",
+ ".py": "python",
+ ".pyi": "python",
+ ".sh": "python",
+ ".bash": "python",
+ ".zsh": "python",
+ ".yaml": "python",
+ ".yml": "python",
+ ".rb": "python",
+ ".toml": "python",
+ ".html": "html",
+ ".htm": "html",
+ ".xml": "html",
+ ".sql": "sql",
+}
+
+
+def normalize_language(language: str | None) -> str:
+ if language is None:
+ return "auto"
+ if not isinstance(language, str):
+ raise TypeError("language must be a string")
+ key = language.strip().lower()
+ if not key:
+ return "auto"
+ return _LANGUAGE_ALIASES.get(key, "auto")
+
+
+def language_from_path(path: str | Path) -> str:
+ suffix = Path(path).suffix.lower()
+ return _LANGUAGE_BY_SUFFIX.get(suffix, "auto")
+
+
+def _slash_comments(language: str) -> bool:
+ return language in {"auto", "javascript", "c", "sql"}
+
+
+def _block_comments(language: str) -> bool:
+ return language in {"auto", "javascript", "c", "sql"}
+
+
+def _hash_comments(language: str) -> bool:
+ return language == "python"
+
+
+def _sql_line_comments(language: str) -> bool:
+ return language == "sql"
+
+
+def _html_comments(language: str) -> bool:
+ return language == "html"
+
+
+def _is_url_slash_slash(text: str, index: int) -> bool:
+ return index > 0 and text[index - 1] == ":"
+
+
+def source_roles(text: str, language: str | None = None) -> tuple[str, ...]:
+ if not isinstance(text, str):
+ raise TypeError("text must be a string")
+ lang = normalize_language(language)
+ roles = [ROLE_CODE] * len(text)
+ length = len(text)
+ index = 0
+ in_string = ""
+ escape = False
+ while index < length:
+ character = text[index]
+ if in_string:
+ roles[index] = ROLE_STRING
+ if escape:
+ escape = False
+ index += 1
+ continue
+ if character == "\\" and index + 1 < length:
+ escape = True
+ index += 1
+ continue
+ if character == in_string:
+ in_string = ""
+ index += 1
+ continue
+ pair = text[index : index + 2]
+ if _slash_comments(lang) and pair == "//" and not _is_url_slash_slash(text, index):
+ cursor = index
+ while cursor < length and text[cursor] not in "\n\r":
+ roles[cursor] = ROLE_COMMENT
+ cursor += 1
+ index = cursor
+ continue
+ if _sql_line_comments(lang) and pair == "--":
+ cursor = index
+ while cursor < length and text[cursor] not in "\n\r":
+ roles[cursor] = ROLE_COMMENT
+ cursor += 1
+ index = cursor
+ continue
+ if _block_comments(lang) and pair == "/*":
+ close = text.find("*/", index + 2)
+ stop = close + 2 if close >= 0 else length
+ cursor = index
+ while cursor < stop:
+ roles[cursor] = ROLE_COMMENT
+ cursor += 1
+ index = stop
+ continue
+ if _hash_comments(lang) and character == "#":
+ cursor = index
+ while cursor < length and text[cursor] not in "\n\r":
+ roles[cursor] = ROLE_COMMENT
+ cursor += 1
+ index = cursor
+ continue
+ if _html_comments(lang) and text[index : index + 4] == "", index + 4)
+ stop = close + 3 if close >= 0 else length
+ cursor = index
+ while cursor < stop:
+ roles[cursor] = ROLE_COMMENT
+ cursor += 1
+ index = stop
+ continue
+ if character in {'"', "'", "`"}:
+ in_string = character
+ roles[index] = ROLE_STRING
+ index += 1
+ continue
+ index += 1
+ return tuple(roles)
+
def _is_emojiish(codepoint: int) -> bool:
if codepoint < 0:
@@ -33,19 +227,23 @@ def _is_ident_char(character: str) -> bool:
return character == "_" or character.isalnum()
-def classify_context(text: str, index: int, in_string: bool) -> str:
+def classify_context(text: str, index: int, role: str = ROLE_CODE, category: str = "") -> str:
prev = text[index - 1] if index > 0 else ""
nxt = text[index + 1] if index + 1 < len(text) else ""
prev_cp = ord(prev) if prev else -1
nxt_cp = ord(nxt) if nxt else -1
+ if role in {ROLE_COMMENT, ROLE_STRING} and category == "bidi_control":
+ return CONTEXT_COMMENT if role == ROLE_COMMENT else CONTEXT_STRING
if _is_emojiish(prev_cp) or _is_emojiish(nxt_cp) or prev_cp in _VS or nxt_cp in _VS:
return CONTEXT_EMOJI
+ if role == ROLE_COMMENT:
+ return CONTEXT_COMMENT
+ if role == ROLE_STRING:
+ return CONTEXT_STRING
if _is_ident_char(prev) and _is_ident_char(nxt):
return CONTEXT_IDENTIFIER
if _is_ident_char(nxt) or _is_ident_char(prev):
return CONTEXT_IDENTIFIER
- if in_string:
- return CONTEXT_STRING
return CONTEXT_PROSE
@@ -53,7 +251,7 @@ def score_severity(category: str, context: str) -> str:
if category == "tag":
return SEVERITY_CRITICAL
if category == "bidi_control":
- if context == CONTEXT_IDENTIFIER:
+ if context in {CONTEXT_IDENTIFIER, CONTEXT_COMMENT, CONTEXT_STRING}:
return SEVERITY_CRITICAL
return SEVERITY_HIGH
if category == "zero_width":
@@ -79,6 +277,16 @@ def explain_finding(category: str, context: str, severity: str) -> tuple[str, st
"Bidirectional override sits inside an identifier, so the glyphs can read differently than the bytes (Trojan Source).",
"Strip the bidi control and keep the identifier left-to-right.",
)
+ if category == "bidi_control" and context == CONTEXT_COMMENT:
+ return (
+ "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).",
+ "Strip the bidi control from the comment.",
+ )
+ if category == "bidi_control" and context == CONTEXT_STRING:
+ return (
+ "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).",
+ "Strip the bidi control from the string.",
+ )
if category == "bidi_control":
return (
"Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).",
@@ -120,8 +328,13 @@ def explain_finding(category: str, context: str, severity: str) -> tuple[str, st
)
-def annotate_finding(text: str, index: int, category: str, in_string: bool) -> tuple[str, str, str, str]:
- context = classify_context(text, index, in_string)
+def annotate_finding(
+ text: str,
+ index: int,
+ category: str,
+ role: str = ROLE_CODE,
+) -> tuple[str, str, str, str]:
+ context = classify_context(text, index, role, category)
severity = score_severity(category, context)
why, remedy = explain_finding(category, context, severity)
return context, severity, why, remedy
diff --git a/fuckmark/web.py b/fuckmark/web.py
index d30a0084..865b90df 100644
--- a/fuckmark/web.py
+++ b/fuckmark/web.py
@@ -19,6 +19,7 @@
from .product.domain import PRODUCT_MAX_INPUT_CHARS
from .product.normalize import normalize_payload
from .product.scan import clean_hidden_characters, scan_dict, scan_hidden_characters
+from .product.severity import normalize_language
from .product.visible_projection import product_approved_carriers_v1, project_visible_v1
@@ -103,7 +104,7 @@ def remove_marks_payload(text: str) -> dict[str, object]:
}
-def scan_payload(text: str) -> dict[str, object]:
+def scan_payload(text: str, *, language: str | None = None) -> dict[str, object]:
if not isinstance(text, str):
raise TypeError("text must be a string")
if len(text) > PRODUCT_MAX_INPUT_CHARS:
@@ -114,12 +115,13 @@ def scan_payload(text: str) -> dict[str, object]:
"max": PRODUCT_MAX_INPUT_CHARS,
"contact": DETECT_CONTACT_EMAIL,
}
- result = scan_hidden_characters(text)
+ result = scan_hidden_characters(text, language=language)
cleaned, removed = clean_hidden_characters(text)
return {
"ok": True,
"reason": "found" if result.detected else "clean",
"backend": "python",
+ "language": normalize_language(language),
"scan": scan_dict(result),
"cleaned": cleaned,
"removed": removed,
@@ -144,7 +146,8 @@ def _web_parser() -> argparse.ArgumentParser:
prog="fuckmark web",
description=(
"Open the FuckMark browser tool locally. "
- "Serves mark.html and a Python API for detect/strip. "
+ "Serves mark.html, the hidden-Unicode scan page at /scan.html, "
+ "and a Python API for detect/strip/scan/guard/normalize. "
"For beginners who prefer a page over the CLI."
),
)
@@ -265,7 +268,10 @@ def do_POST(self) -> None:
text = payload.get("text", "")
if not isinstance(text, str):
raise ValueError("text must be a string")
- result = scan_payload(text)
+ language = payload.get("language", None)
+ if language is not None and not isinstance(language, str):
+ raise ValueError("language must be a string")
+ result = scan_payload(text, language=language)
status = 200 if result.get("reason") != "too-large" else 413
self._json_response(status, result)
except ValueError as error:
@@ -340,6 +346,7 @@ class _Server(socketserver.TCPServer):
errors.write(
"FuckMark web: Python API at /api/health, /api/remove-marks, /api/scan, /api/guard, /api/normalize\n"
)
+ errors.write("FuckMark web: hidden-Unicode scan UI at /scan.html\n")
errors.write("FuckMark web: press Ctrl+C to stop.\n")
errors.flush()
if on_ready is not None:
diff --git a/fuckmark/webui/scan.html b/fuckmark/webui/scan.html
new file mode 100644
index 00000000..0b007977
--- /dev/null
+++ b/fuckmark/webui/scan.html
@@ -0,0 +1,606 @@
+
+
+
+
+
+FuckMark — Hidden Unicode scan
+
+
+
+
+
+
+
+
+
+
+
+
+
+
See the bytes.
+
Paste source or a prompt. This page reveals Trojan Source bidi overrides, zero-width characters, and Unicode tag smuggling without sending the text anywhere.
+
+
+
+
+
+
+ Engine: browser
+
+
+
+
+
+
+
+
+
+
+
Paste text to scan.
+
+
+
+ Scanning is fuckmark-hidden-scan-v1. Copy uses the security category set
+ (bidi_control, zero_width, tag, control,
+ noncharacter, surrogate). Fix Trojan Source strips only
+ bidirectional controls. Nothing is uploaded from a static page or file://.
+ Under fuckmark web an optional local Python API can confirm the same verdicts.
+ Contact Fhelp@q1z.org.
+
+
+
+
+
+
+
diff --git a/fuckmark/webui/scan.js b/fuckmark/webui/scan.js
new file mode 100644
index 00000000..f28c001f
--- /dev/null
+++ b/fuckmark/webui/scan.js
@@ -0,0 +1,569 @@
+// Faithful JavaScript port of fuckmark-hidden-scan-v1 (fuckmark/product/scan.py).
+// Kept in lockstep with the Python engine by tests/test_vscode_scanner_parity.py.
+
+"use strict";
+
+const SCAN_ALGORITHM_VERSION = "fuckmark-hidden-scan-v1";
+
+const CATEGORY_BIDI_CONTROL = "bidi_control";
+const CATEGORY_ZERO_WIDTH = "zero_width";
+const CATEGORY_VARIATION_SELECTOR = "variation_selector";
+const CATEGORY_TAG = "tag";
+const CATEGORY_ENCLOSING_MARK = "enclosing_mark";
+const CATEGORY_LINE_SEPARATOR = "line_separator";
+const CATEGORY_DEPRECATED = "deprecated";
+const CATEGORY_FORMAT = "format";
+const CATEGORY_CONTROL = "control";
+const CATEGORY_PRIVATE_USE = "private_use";
+const CATEGORY_NONCHARACTER = "noncharacter";
+const CATEGORY_SURROGATE = "surrogate";
+
+const SCAN_CATEGORIES = [
+ CATEGORY_BIDI_CONTROL,
+ CATEGORY_ZERO_WIDTH,
+ CATEGORY_VARIATION_SELECTOR,
+ CATEGORY_TAG,
+ CATEGORY_ENCLOSING_MARK,
+ CATEGORY_LINE_SEPARATOR,
+ CATEGORY_DEPRECATED,
+ CATEGORY_FORMAT,
+ CATEGORY_CONTROL,
+ CATEGORY_PRIVATE_USE,
+ CATEGORY_NONCHARACTER,
+ CATEGORY_SURROGATE,
+];
+
+const CATEGORY_DESCRIPTIONS = {
+ [CATEGORY_BIDI_CONTROL]: "bidirectional override/isolate (Trojan Source reordering)",
+ [CATEGORY_ZERO_WIDTH]: "zero-width or invisible spacing character",
+ [CATEGORY_VARIATION_SELECTOR]: "variation selector (glyph/steganography carrier)",
+ [CATEGORY_TAG]: "Unicode tag character (hidden text / prompt-injection smuggling)",
+ [CATEGORY_ENCLOSING_MARK]: "enclosing combining mark (alters surrounding glyph)",
+ [CATEGORY_LINE_SEPARATOR]: "line/paragraph separator (breaks parsers, invisible)",
+ [CATEGORY_DEPRECATED]: "deprecated format control or interlinear annotation",
+ [CATEGORY_FORMAT]: "general Unicode format control (Cf)",
+ [CATEGORY_CONTROL]: "C0/C1 control character",
+ [CATEGORY_PRIVATE_USE]: "private-use codepoint (renderer-defined)",
+ [CATEGORY_NONCHARACTER]: "Unicode noncharacter or unassigned codepoint",
+ [CATEGORY_SURROGATE]: "lone surrogate codepoint",
+};
+
+const DEFAULT_SECURITY_CATEGORIES = [
+ CATEGORY_BIDI_CONTROL,
+ CATEGORY_ZERO_WIDTH,
+ CATEGORY_TAG,
+ CATEGORY_CONTROL,
+ CATEGORY_NONCHARACTER,
+ CATEGORY_SURROGATE,
+];
+
+const TROJAN_SOURCE_CATEGORIES = [CATEGORY_BIDI_CONTROL];
+
+const ALLOWED_WHITESPACE = new Set([0x09, 0x0a, 0x0d, 0x20]);
+
+function inRange(cp, lo, hi) {
+ return cp >= lo && cp <= hi;
+}
+
+const BIDI_CONTROL = new Set([0x061c, 0x200e, 0x200f]);
+for (let cp = 0x202a; cp <= 0x202e; cp += 1) BIDI_CONTROL.add(cp);
+for (let cp = 0x2066; cp <= 0x2069; cp += 1) BIDI_CONTROL.add(cp);
+
+const DEPRECATED = new Set([0xfff9, 0xfffa, 0xfffb]);
+for (let cp = 0x206a; cp <= 0x206f; cp += 1) DEPRECATED.add(cp);
+
+const ZERO_WIDTH = new Set([
+ 0x00ad, 0x034f, 0x115f, 0x1160, 0x17b4, 0x17b5, 0x180e, 0x200b, 0x200c,
+ 0x200d, 0x2060, 0x2061, 0x2062, 0x2063, 0x2064, 0x3164, 0xfeff, 0xffa0,
+]);
+
+function isVariationSelector(cp) {
+ return inRange(cp, 0xfe00, 0xfe0f) || inRange(cp, 0xe0100, 0xe01ef);
+}
+
+function isTag(cp) {
+ return inRange(cp, 0xe0000, 0xe007f);
+}
+
+function isNoncharacter(cp) {
+ if (inRange(cp, 0xfdd0, 0xfdef)) return true;
+ const low = cp & 0xffff;
+ return low === 0xfffe || low === 0xffff;
+}
+
+function isEnclosingMark(cp) {
+ // Unicode general category Me. Enumerated to avoid a runtime Unicode DB.
+ return (
+ inRange(cp, 0x0488, 0x0489) ||
+ cp === 0x1abe ||
+ inRange(cp, 0x20dd, 0x20e0) ||
+ inRange(cp, 0x20e2, 0x20e4) ||
+ inRange(cp, 0xa670, 0xa672)
+ );
+}
+
+function isControl(cp) {
+ return inRange(cp, 0x00, 0x1f) || inRange(cp, 0x7f, 0x9f);
+}
+
+function isLineSeparator(cp) {
+ return cp === 0x2028 || cp === 0x2029;
+}
+
+// General category Cf ranges (Unicode 15-era), excluding codepoints already
+// classified above. Mirrors what unicodedata.category returns as "Cf".
+const FORMAT_RANGES = [
+ [0x0600, 0x0605],
+ [0x06dd, 0x06dd],
+ [0x070f, 0x070f],
+ [0x0890, 0x0891],
+ [0x08e2, 0x08e2],
+ [0x110bd, 0x110bd],
+ [0x110cd, 0x110cd],
+ [0x13430, 0x1343f],
+ [0x1bca0, 0x1bca3],
+ [0x1d173, 0x1d17a],
+];
+
+function isFormat(cp) {
+ for (const [lo, hi] of FORMAT_RANGES) {
+ if (inRange(cp, lo, hi)) return true;
+ }
+ return false;
+}
+
+function isPrivateUse(cp) {
+ return (
+ inRange(cp, 0xe000, 0xf8ff) ||
+ inRange(cp, 0xf0000, 0xffffd) ||
+ inRange(cp, 0x100000, 0x10fffd)
+ );
+}
+
+function classify(cp) {
+ if (ALLOWED_WHITESPACE.has(cp)) return null;
+ if (BIDI_CONTROL.has(cp)) return CATEGORY_BIDI_CONTROL;
+ if (DEPRECATED.has(cp)) return CATEGORY_DEPRECATED;
+ if (ZERO_WIDTH.has(cp)) return CATEGORY_ZERO_WIDTH;
+ if (isVariationSelector(cp)) return CATEGORY_VARIATION_SELECTOR;
+ if (isTag(cp)) return CATEGORY_TAG;
+ if (isNoncharacter(cp)) return CATEGORY_NONCHARACTER;
+ if (cp >= 0xd800 && cp <= 0xdfff) return CATEGORY_SURROGATE;
+ if (isEnclosingMark(cp)) return CATEGORY_ENCLOSING_MARK;
+ if (isLineSeparator(cp)) return CATEGORY_LINE_SEPARATOR;
+ if (isControl(cp)) return CATEGORY_CONTROL;
+ if (isFormat(cp)) return CATEGORY_FORMAT;
+ if (isPrivateUse(cp)) return CATEGORY_PRIVATE_USE;
+ return null;
+}
+
+function codepointToken(cp) {
+ return "U+" + cp.toString(16).toUpperCase().padStart(4, "0");
+}
+
+function isEmojiish(cp) {
+ if (cp < 0) return false;
+ if (cp >= 0x1f1e6 && cp <= 0x1f1ff) return true;
+ if (cp >= 0x1f000 && cp <= 0x1faff) return true;
+ if (cp >= 0x2600 && cp <= 0x27bf) return true;
+ return cp === 0x200d || [0x00a9, 0x00ae, 0x203c, 0x2049, 0x2122, 0x2139, 0x3030, 0x303d, 0x3297, 0x3299].includes(cp);
+}
+
+function isIdentChar(ch) {
+ return ch === "_" || /\p{L}|\p{N}/u.test(ch);
+}
+
+function neighborBefore(text, offset) {
+ if (offset <= 0) return { cp: -1, ch: "" };
+ if (offset >= 2) {
+ const cp = text.codePointAt(offset - 2);
+ if (cp > 0xffff) return { cp, ch: String.fromCodePoint(cp) };
+ }
+ const cp = text.codePointAt(offset - 1);
+ return { cp, ch: String.fromCodePoint(cp) };
+}
+
+function neighborAfter(text, offset, length) {
+ const nextIndex = offset + length;
+ if (nextIndex >= text.length) return { cp: -1, ch: "" };
+ const cp = text.codePointAt(nextIndex);
+ return { cp, ch: String.fromCodePoint(cp) };
+}
+
+const LANGUAGE_ALIASES = {
+ auto: "auto",
+ javascript: "javascript",
+ js: "javascript",
+ ts: "javascript",
+ typescript: "javascript",
+ jsx: "javascript",
+ tsx: "javascript",
+ c: "c",
+ h: "c",
+ cc: "c",
+ cpp: "c",
+ cxx: "c",
+ java: "c",
+ go: "c",
+ rs: "c",
+ rust: "c",
+ cs: "c",
+ css: "c",
+ jsonc: "c",
+ python: "python",
+ py: "python",
+ pyi: "python",
+ hash: "python",
+ sh: "python",
+ bash: "python",
+ zsh: "python",
+ shell: "python",
+ shellscript: "python",
+ yaml: "python",
+ yml: "python",
+ rb: "python",
+ ruby: "python",
+ toml: "python",
+ html: "html",
+ htm: "html",
+ xml: "html",
+ sql: "sql",
+};
+
+function normalizeLanguage(language) {
+ if (!language) return "auto";
+ const key = String(language).trim().toLowerCase();
+ return LANGUAGE_ALIASES[key] || "auto";
+}
+
+function slashComments(language) {
+ return language === "auto" || language === "javascript" || language === "c" || language === "sql";
+}
+
+function blockComments(language) {
+ return slashComments(language);
+}
+
+function hashComments(language) {
+ return language === "python";
+}
+
+function sqlLineComments(language) {
+ return language === "sql";
+}
+
+function htmlComments(language) {
+ return language === "html";
+}
+
+function sourceRoles(text, language) {
+ const lang = normalizeLanguage(language);
+ const roles = new Array(text.length).fill("code");
+ let index = 0;
+ let inString = "";
+ let escape = false;
+ while (index < text.length) {
+ const ch = text[index];
+ if (inString) {
+ roles[index] = "string";
+ if (escape) {
+ escape = false;
+ index += 1;
+ continue;
+ }
+ if (ch === "\\" && index + 1 < text.length) {
+ escape = true;
+ index += 1;
+ continue;
+ }
+ if (ch === inString) inString = "";
+ index += 1;
+ continue;
+ }
+ const pair = text.slice(index, index + 2);
+ if (slashComments(lang) && pair === "//" && !(index > 0 && text[index - 1] === ":")) {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (sqlLineComments(lang) && pair === "--") {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (blockComments(lang) && pair === "/*") {
+ const close = text.indexOf("*/", index + 2);
+ const stop = close >= 0 ? close + 2 : text.length;
+ for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment";
+ index = stop;
+ continue;
+ }
+ if (hashComments(lang) && ch === "#") {
+ let cursor = index;
+ while (cursor < text.length && text[cursor] !== "\n" && text[cursor] !== "\r") {
+ roles[cursor] = "comment";
+ cursor += 1;
+ }
+ index = cursor;
+ continue;
+ }
+ if (htmlComments(lang) && text.slice(index, index + 4) === "", index + 4);
+ const stop = close >= 0 ? close + 3 : text.length;
+ for (let cursor = index; cursor < stop; cursor += 1) roles[cursor] = "comment";
+ index = stop;
+ continue;
+ }
+ if (ch === '"' || ch === "'" || ch === "`") {
+ inString = ch;
+ roles[index] = "string";
+ index += 1;
+ continue;
+ }
+ index += 1;
+ }
+ return roles;
+}
+
+function classifyContext(text, offset, role, category) {
+ const length = text.codePointAt(offset) > 0xffff ? 2 : 1;
+ const prev = neighborBefore(text, offset);
+ const next = neighborAfter(text, offset, length);
+ if ((role === "comment" || role === "string") && category === CATEGORY_BIDI_CONTROL) {
+ return role;
+ }
+ if (
+ isEmojiish(prev.cp) ||
+ isEmojiish(next.cp) ||
+ (prev.cp >= 0xfe00 && prev.cp <= 0xfe0f) ||
+ (next.cp >= 0xfe00 && next.cp <= 0xfe0f)
+ ) {
+ return "emoji";
+ }
+ if (role === "comment") return "comment";
+ if (role === "string") return "string";
+ if (isIdentChar(prev.ch) || isIdentChar(next.ch)) return "identifier";
+ return "prose";
+}
+
+function scoreSeverity(category, context) {
+ if (category === CATEGORY_TAG) return "critical";
+ if (category === CATEGORY_BIDI_CONTROL) {
+ if (context === "identifier" || context === "comment" || context === "string") return "critical";
+ return "high";
+ }
+ if (category === CATEGORY_ZERO_WIDTH) {
+ if (context === "emoji") return "info";
+ if (context === "identifier") return "high";
+ return "medium";
+ }
+ if (category === CATEGORY_VARIATION_SELECTOR) return context === "emoji" ? "info" : "medium";
+ if (category === CATEGORY_CONTROL || category === CATEGORY_NONCHARACTER || category === CATEGORY_SURROGATE) {
+ return "high";
+ }
+ return "medium";
+}
+
+function explainFinding(category, context, severity) {
+ if (category === CATEGORY_BIDI_CONTROL && context === "identifier") {
+ return {
+ why: "Bidirectional override sits inside an identifier, so the glyphs can read differently than the bytes (Trojan Source).",
+ remedy: "Strip the bidi control and keep the identifier left-to-right.",
+ };
+ }
+ if (category === CATEGORY_BIDI_CONTROL && context === "comment") {
+ return {
+ why: "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).",
+ remedy: "Strip the bidi control from the comment.",
+ };
+ }
+ if (category === CATEGORY_BIDI_CONTROL && context === "string") {
+ return {
+ why: "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).",
+ remedy: "Strip the bidi control from the string.",
+ };
+ }
+ if (category === CATEGORY_BIDI_CONTROL) {
+ return {
+ why: "Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).",
+ remedy: "Strip U+202A-U+202E / U+2066-U+2069 and rewrite the text left-to-right.",
+ };
+ }
+ if (category === CATEGORY_TAG) {
+ return {
+ why: "Unicode tag characters encode a second ASCII string that models read and humans do not.",
+ remedy: "Strip U+E0020-U+E007F; inspect tag_payload for the smuggled text.",
+ };
+ }
+ if (category === CATEGORY_ZERO_WIDTH && context === "emoji") {
+ return {
+ why: "Zero-width joiner or invisible mark inside an emoji cluster; usually a legitimate emoji sequence.",
+ remedy: "Leave emoji ZWJ sequences unless you are sanitizing for a security boundary.",
+ };
+ }
+ if (category === CATEGORY_ZERO_WIDTH && context === "identifier") {
+ return {
+ why: "Zero-width character splits an identifier, breaking search and some compilers while looking unchanged.",
+ remedy: "Strip the zero-width character from the identifier.",
+ };
+ }
+ if (category === CATEGORY_ZERO_WIDTH) {
+ return {
+ why: "Invisible spacing or joining character that changes the byte stream without changing the glyphs.",
+ remedy: "Strip the zero-width character.",
+ };
+ }
+ if (category === CATEGORY_VARIATION_SELECTOR && context === "emoji") {
+ return {
+ why: "Variation selector tunes an emoji glyph; usually benign.",
+ remedy: "Keep emoji variation selectors unless you are stripping all hidden marks.",
+ };
+ }
+ if (severity === "high") {
+ return {
+ why: "Hidden or non-text codepoint that should not appear in ordinary source or prompts.",
+ remedy: "Strip the character.",
+ };
+ }
+ return {
+ why: CATEGORY_DESCRIPTIONS[category] || "Hidden or format codepoint that is invisible or renderer-defined.",
+ remedy: "Strip the character if this text crosses a trust boundary.",
+ };
+}
+
+function scanText(text, categories, language) {
+ const selected = categories ? new Set(categories) : null;
+ const roles = sourceRoles(text, language);
+ const counts = {};
+ const findings = [];
+ let total = 0;
+ let offset = 0;
+ while (offset < text.length) {
+ const cp = text.codePointAt(offset);
+ const length = cp > 0xffff ? 2 : 1;
+ const category = classify(cp);
+ if (category !== null && (selected === null || selected.has(category))) {
+ total += 1;
+ counts[category] = (counts[category] || 0) + 1;
+ const context = classifyContext(text, offset, roles[offset] || "code", category);
+ const severity = scoreSeverity(category, context);
+ const explained = explainFinding(category, context, severity);
+ findings.push({
+ offset,
+ length,
+ codepoint: cp,
+ category,
+ context,
+ severity,
+ why: explained.why,
+ remedy: explained.remedy,
+ });
+ }
+ offset += length;
+ }
+ return { total, counts, findings };
+}
+
+function cleanText(text, categories) {
+ const selected = categories ? new Set(categories) : null;
+ let out = "";
+ let removed = 0;
+ let offset = 0;
+ while (offset < text.length) {
+ const cp = text.codePointAt(offset);
+ const length = cp > 0xffff ? 2 : 1;
+ const category = classify(cp);
+ if (category !== null && (selected === null || selected.has(category))) {
+ removed += 1;
+ } else {
+ out += text.slice(offset, offset + length);
+ }
+ offset += length;
+ }
+ return { cleaned: out, removed };
+}
+
+function autofixTrojanSource(text) {
+ return cleanText(text, TROJAN_SOURCE_CATEGORIES);
+}
+
+function languageFromDocumentId(languageId) {
+ return normalizeLanguage(languageId);
+}
+
+const LANGUAGE_BY_SUFFIX = {
+ ".js": "javascript",
+ ".mjs": "javascript",
+ ".cjs": "javascript",
+ ".ts": "javascript",
+ ".jsx": "javascript",
+ ".tsx": "javascript",
+ ".c": "c",
+ ".h": "c",
+ ".hh": "c",
+ ".cc": "c",
+ ".cpp": "c",
+ ".cxx": "c",
+ ".java": "c",
+ ".go": "c",
+ ".rs": "c",
+ ".cs": "c",
+ ".css": "c",
+ ".py": "python",
+ ".pyi": "python",
+ ".sh": "python",
+ ".bash": "python",
+ ".zsh": "python",
+ ".yaml": "python",
+ ".yml": "python",
+ ".rb": "python",
+ ".toml": "python",
+ ".html": "html",
+ ".htm": "html",
+ ".xml": "html",
+ ".sql": "sql",
+};
+
+function languageFromPath(path) {
+ const name = String(path || "");
+ const slash = Math.max(name.lastIndexOf("/"), name.lastIndexOf("\\"));
+ const base = slash >= 0 ? name.slice(slash + 1) : name;
+ const dot = base.lastIndexOf(".");
+ if (dot <= 0) return "auto";
+ const suffix = base.slice(dot).toLowerCase();
+ return LANGUAGE_BY_SUFFIX[suffix] || "auto";
+}
+
+const FuckMarkScan = {
+ SCAN_ALGORITHM_VERSION,
+ SCAN_CATEGORIES,
+ CATEGORY_DESCRIPTIONS,
+ DEFAULT_SECURITY_CATEGORIES,
+ TROJAN_SOURCE_CATEGORIES,
+ classify,
+ classifyContext,
+ scoreSeverity,
+ normalizeLanguage,
+ languageFromDocumentId,
+ languageFromPath,
+ sourceRoles,
+ codepointToken,
+ scanText,
+ cleanText,
+ autofixTrojanSource,
+};
+
+if (typeof module !== "undefined" && module.exports) {
+ module.exports = FuckMarkScan;
+}
+if (typeof globalThis !== "undefined") {
+ globalThis.FuckMarkScan = FuckMarkScan;
+}
diff --git a/pyproject.toml b/pyproject.toml
index 21bc71b3..7f78157b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -50,4 +50,4 @@ where = ["."]
include = ["fuckmark*"]
[tool.setuptools.package-data]
-fuckmark = ["webui/*.html"]
+fuckmark = ["webui/*.html", "webui/*.js"]
diff --git a/specs/fuckmark-hidden-scan-v1.freeze.json b/specs/fuckmark-hidden-scan-v1.freeze.json
index 68084540..f3e4237d 100644
--- a/specs/fuckmark-hidden-scan-v1.freeze.json
+++ b/specs/fuckmark-hidden-scan-v1.freeze.json
@@ -1,9 +1,9 @@
{
"algorithm_version": "fuckmark-hidden-scan-v1",
- "freeze_record_version": "fuckmark-hidden-scan-v1-freeze-v1",
- "frozen_at_utc": "2026-09-01T23:30:00+00:00",
+ "freeze_record_version": "fuckmark-hidden-scan-v1-freeze-v3",
+ "frozen_at_utc": "2026-09-02T01:50:00+00:00",
"protocol_id": "fuckmark-hidden-scan-v1",
- "protocol_sha256": "859b1b138d8de3795cb09228ea78775b560aeb63520a309f9f9e59a7acc31046",
- "vectors_file_sha256": "a9fbdb7fc2bf30786d9d9199099e740da1e1b33b52518a2d61b038fd85d80610",
- "vectors_canonical_sha256": "2602ca87e5480813caf4452f8314e1ff2dbb43e94fa3cd81a6937b1595287915"
+ "protocol_sha256": "4e594b87ba2f68b929e62fc420b4543186ecc7f71f41f501f1bb646c33c23045",
+ "vectors_file_sha256": "acb8e4ddbd8849979b8f493fabb136fa63e8d4364ce60aedef3ce853006802a3",
+ "vectors_canonical_sha256": "763f2967f68882ec037b6245ad89e166f6b7482292b0e13d774475acb81c5e4a"
}
diff --git a/specs/fuckmark-hidden-scan-v1.protocol.md b/specs/fuckmark-hidden-scan-v1.protocol.md
index 64bacc47..38d5a71e 100644
--- a/specs/fuckmark-hidden-scan-v1.protocol.md
+++ b/specs/fuckmark-hidden-scan-v1.protocol.md
@@ -60,30 +60,47 @@ controls and related Cf assignments.
## 4. Context (heuristic)
-Context is assigned to each finding from immediate neighbors. It is not a
-language parser.
+Context is assigned to each finding from a lightweight source mask plus
+immediate neighbors. It is not a full language parser.
+Roles (optional `language` argument to `scan_hidden_characters`; default
+`auto`):
+
+- `comment`: `//` line comments and `/* */` block comments (`auto`,
+ `javascript`, `c`, `sql`). `#` line comments (`python`). `--` line comments
+ (`sql`). `` (`html`). `//` after `:` is not a comment (`http://`).
+- `string`: `"`, `'`, or backtick literals, with backslash escapes.
+
+Then:
+
+- `comment` / `string` from the role mask when the category is `bidi_control`
+ (Trojan Source encodings stay on that role even next to emoji).
- `emoji` if the previous or next scalar is emoji-ish (regional indicators
U+1F1E6-U+1F1FF, U+1F000-U+1FAFF, U+2600-U+27BF, a small BMP emoji set,
ZWJ, or a BMP variation selector U+FE00-U+FE0F).
+- else `comment` / `string` from the role mask.
- else `identifier` if the previous or next scalar is `_` or alphanumeric.
-- else `string` if a naive quote tracker currently has an open `"`, `'`, or
- backtick. The tracker flips on the current scalar before classification, so
- a hidden character inside quotes is `string` unless a stronger context
- applies. It does not understand escapes or nested languages.
- else `prose`.
+`language` may be passed on the vector as `"language"`. Omitted means `auto`.
+Lint infers language from the file suffix. This lexer does not nest block
+comments, does not parse raw strings, and does not understand every dialect.
+
## 5. Severity
-| Category | identifier | emoji | other |
+| Category | identifier / comment / string | emoji | other |
| --- | --- | --- | --- |
| `tag` | critical | critical | critical |
| `bidi_control` | critical | high | high |
-| `zero_width` | high | info | medium |
+| `zero_width` | high (identifier) / medium (comment, string) | info | medium |
| `variation_selector` | medium | info | medium |
| `control`, `noncharacter`, `surrogate` | high | high | high |
| remaining flagged categories | medium | medium | medium |
+`bidi_control` in identifier, comment, or string is the three Trojan Source
+encodings (CVE-2021-42574): identifier reordering, commenting-out, and
+stretched-string. `autofix_trojan_source` strips only `bidi_control`.
+
Each finding also carries a one-line `why` and `remedy`. Those strings are
informative; conformance is category, context, and severity.
@@ -106,8 +123,9 @@ Default security categories (lint, guard, normalize strip): `bidi_control`,
## 7. Conformance
Reconstruct each vector as `"".join(chr(cp) for cp in codepoints)` (Python
-scalars). Run `scan_hidden_characters`. For every expected finding, require
-matching `index`, `codepoint`, `category`, `context`, and `severity`, in
-order. Vectors that expect no findings must produce `detected is False`.
+scalars). Run `scan_hidden_characters` with `language` from the vector or
+`auto`. For every expected finding, require matching `index`, `codepoint`,
+`category`, `context`, and `severity`, in order. Vectors that expect no
+findings must produce `detected is False`.
Do not store the reconstructed strings in the vector file.
diff --git a/specs/fuckmark-hidden-scan-v1.vectors.json b/specs/fuckmark-hidden-scan-v1.vectors.json
index e8bb44bf..a3d6a78b 100644
--- a/specs/fuckmark-hidden-scan-v1.vectors.json
+++ b/specs/fuckmark-hidden-scan-v1.vectors.json
@@ -140,7 +140,7 @@
"codepoint": 8238,
"category": "bidi_control",
"context": "string",
- "severity": "high"
+ "severity": "critical"
}
]
},
@@ -185,6 +185,199 @@
99
],
"expect": []
+ },
+ {
+ "id": "bidi-stretched-string",
+ "codepoints": [
+ 34,
+ 8238,
+ 97,
+ 34
+ ],
+ "expect": [
+ {
+ "index": 1,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "string",
+ "severity": "critical"
+ }
+ ]
+ },
+ {
+ "id": "bidi-slash-comment",
+ "codepoints": [
+ 47,
+ 47,
+ 32,
+ 8238
+ ],
+ "expect": [
+ {
+ "index": 3,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "comment",
+ "severity": "critical"
+ }
+ ]
+ },
+ {
+ "id": "bidi-block-comment",
+ "codepoints": [
+ 47,
+ 42,
+ 32,
+ 8238,
+ 32,
+ 42,
+ 47
+ ],
+ "expect": [
+ {
+ "index": 3,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "comment",
+ "severity": "critical"
+ }
+ ]
+ },
+ {
+ "id": "bidi-hash-comment",
+ "language": "python",
+ "codepoints": [
+ 35,
+ 32,
+ 8238
+ ],
+ "expect": [
+ {
+ "index": 2,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "comment",
+ "severity": "critical"
+ }
+ ]
+ },
+ {
+ "id": "hash-not-comment-in-auto",
+ "codepoints": [
+ 35,
+ 32,
+ 8238
+ ],
+ "expect": [
+ {
+ "index": 2,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "prose",
+ "severity": "high"
+ }
+ ]
+ },
+ {
+ "id": "bidi-after-url-not-comment",
+ "codepoints": [
+ 104,
+ 116,
+ 116,
+ 112,
+ 58,
+ 47,
+ 47,
+ 8238
+ ],
+ "expect": [
+ {
+ "index": 7,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "prose",
+ "severity": "high"
+ }
+ ]
+ },
+ {
+ "id": "bidi-sql-line-comment",
+ "language": "sql",
+ "codepoints": [
+ 45,
+ 45,
+ 32,
+ 8238
+ ],
+ "expect": [
+ {
+ "index": 3,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "comment",
+ "severity": "critical"
+ }
+ ]
+ },
+ {
+ "id": "bidi-html-comment",
+ "language": "html",
+ "codepoints": [
+ 60,
+ 33,
+ 45,
+ 45,
+ 32,
+ 8238,
+ 45,
+ 45,
+ 62
+ ],
+ "expect": [
+ {
+ "index": 5,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "comment",
+ "severity": "critical"
+ }
+ ]
+ },
+ {
+ "id": "bidi-slash-comment-beside-emoji",
+ "codepoints": [
+ 47,
+ 47,
+ 128512,
+ 8238
+ ],
+ "expect": [
+ {
+ "index": 3,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "comment",
+ "severity": "critical"
+ }
+ ]
+ },
+ {
+ "id": "bidi-string-beside-emoji",
+ "codepoints": [
+ 34,
+ 128512,
+ 8238,
+ 34
+ ],
+ "expect": [
+ {
+ "index": 2,
+ "codepoint": 8238,
+ "category": "bidi_control",
+ "context": "string",
+ "severity": "critical"
+ }
+ ]
}
]
}
diff --git a/tests/test_hidden_scan_spec.py b/tests/test_hidden_scan_spec.py
index e8e8ec62..dba8f604 100644
--- a/tests/test_hidden_scan_spec.py
+++ b/tests/test_hidden_scan_spec.py
@@ -38,7 +38,7 @@ def test_hidden_scan_vectors_replay_on_python_engine() -> None:
payload = _load_vectors()
for vector in payload["vectors"]:
text = "".join(chr(code) for code in vector["codepoints"])
- result = scan_hidden_characters(text)
+ result = scan_hidden_characters(text, language=vector.get("language"))
expected = vector["expect"]
assert len(result.findings) == len(expected), vector["id"]
for finding, want in zip(result.findings, expected, strict=True):
@@ -62,7 +62,7 @@ def test_hidden_scan_js_replays_vector_verdicts() -> None:
const payload = JSON.parse(require("fs").readFileSync(0, "utf8"));
const out = payload.vectors.map((vector) => {
const text = String.fromCodePoint(...vector.codepoints);
- const result = scan.scanText(text);
+ const result = scan.scanText(text, null, vector.language || "auto");
return result.findings.map((finding) => ({
codepoint: finding.codepoint,
category: finding.category,
@@ -107,3 +107,11 @@ def test_scan_js_syntax() -> None:
subprocess.run([node, "--check", str(SCAN_JS)], check=True, timeout=15)
extension = ROOT / "editors" / "vscode" / "extension.js"
subprocess.run([node, "--check", str(extension)], check=True, timeout=15)
+ copies = (
+ ROOT / "docs" / "scan.js",
+ ROOT / "fuckmark" / "webui" / "scan.js",
+ )
+ canonical = SCAN_JS.read_text(encoding="utf-8")
+ for path in copies:
+ assert path.read_text(encoding="utf-8") == canonical
+ subprocess.run([node, "--check", str(path)], check=True, timeout=15)
diff --git a/tests/test_lint.py b/tests/test_lint.py
index 697f1dc8..a27f75af 100644
--- a/tests/test_lint.py
+++ b/tests/test_lint.py
@@ -66,6 +66,18 @@ def test_json_report_structure(tmp_path: Path) -> None:
assert result["locations"][0]["remedy"]
+def test_javascript_comment_context_from_suffix(tmp_path: Path) -> None:
+ (tmp_path / "note.js").write_text("// \u202e\n", encoding="utf-8")
+ code, out, err = _run(["--json", str(tmp_path)])
+ assert code == LINT_EXIT_FINDINGS
+ assert err == ""
+ report = json.loads(out)
+ location = report["results"][0]["locations"][0]
+ assert location["context"] == "comment"
+ assert location["severity"] == "critical"
+ assert location["codepoint"] == "U+202E"
+
+
def test_fix_rewrites_files_and_then_passes(tmp_path: Path) -> None:
target = tmp_path / "evil.js"
target.write_text(TROJAN, encoding="utf-8")
diff --git a/tests/test_scan.py b/tests/test_scan.py
index e1e0908e..bed0b7ba 100644
--- a/tests/test_scan.py
+++ b/tests/test_scan.py
@@ -13,6 +13,7 @@
CATEGORY_VARIATION_SELECTOR,
CATEGORY_ZERO_WIDTH,
SCAN_CATEGORIES,
+ autofix_trojan_source,
classify_hidden_codepoint,
clean_hidden_characters,
scan_dict,
@@ -20,6 +21,7 @@
scan_human_report,
scan_machine_line,
)
+from fuckmark.product.severity import language_from_path, source_roles
from fuckmark.web import scan_payload, serve_mark_web
@@ -209,6 +211,7 @@ def test_web_scan_payload_and_endpoint() -> None:
assert payload["cleaned"] == "Hello from the scanner."
assert int(payload["removed"]) > 0
assert payload["scan"]["found"] is True
+ assert payload["language"] == "auto"
seen: dict[str, object] = {}
@@ -233,6 +236,32 @@ def on_ready(url: str, port: int) -> None:
assert clean_data["reason"] == "clean"
assert clean_data["scan"]["found"] is False
+ def on_ready_language(url: str, port: int) -> None:
+ py_status, py_data = _post_json(
+ f"http://127.0.0.1:{port}/api/scan",
+ {"text": "# \u202e", "language": "python"},
+ )
+ seen["py_status"] = py_status
+ seen["py_data"] = py_data
+ auto_status, auto_data = _post_json(
+ f"http://127.0.0.1:{port}/api/scan",
+ {"text": "# \u202e"},
+ )
+ seen["auto_status"] = auto_status
+ seen["auto_data"] = auto_data
+
+ serve_mark_web(host="127.0.0.1", port=0, open_browser=False, serve_seconds=0.05, on_ready=on_ready_language)
+ assert seen["py_status"] == 200
+ py_data = seen["py_data"]
+ assert isinstance(py_data, dict)
+ assert py_data["language"] == "python"
+ assert py_data["scan"]["findings"][0]["context"] == "comment"
+ assert py_data["scan"]["findings"][0]["severity"] == "critical"
+ auto_data = seen["auto_data"]
+ assert isinstance(auto_data, dict)
+ assert auto_data["language"] == "auto"
+ assert auto_data["scan"]["findings"][0]["context"] == "prose"
+
def test_severity_is_context_aware() -> None:
ident = scan_hidden_characters("a\u202eb")
@@ -252,4 +281,65 @@ def test_severity_is_context_aware() -> None:
quoted = scan_hidden_characters('"\u202e"')
assert quoted.findings[0].context == "string"
- assert quoted.findings[0].severity == "high"
+ assert quoted.findings[0].severity == "critical"
+
+ comment = scan_hidden_characters("// \u202e")
+ assert comment.findings[0].context == "comment"
+ assert comment.findings[0].severity == "critical"
+
+
+def test_language_from_path_maps_suffixes() -> None:
+ assert language_from_path("app.js") == "javascript"
+ assert language_from_path("mod.py") == "python"
+ assert language_from_path("q.sql") == "sql"
+ assert language_from_path("page.html") == "html"
+ assert language_from_path("notes.txt") == "auto"
+
+
+def test_hash_comment_is_language_aware() -> None:
+ python = scan_hidden_characters("# \u202e", language="python")
+ assert python.findings[0].context == "comment"
+ assert python.findings[0].severity == "critical"
+ auto = scan_hidden_characters("# \u202e")
+ assert auto.findings[0].context == "prose"
+ assert auto.findings[0].severity == "high"
+
+
+def test_url_double_slash_is_not_a_comment() -> None:
+ roles = source_roles("http://x")
+ assert all(role == "code" for role in roles)
+ result = scan_hidden_characters("http://\u202e")
+ assert result.findings[0].context == "prose"
+ assert result.findings[0].severity == "high"
+
+
+def test_autofix_trojan_source_strips_only_bidi() -> None:
+ text = "a\u202e\u200bb"
+ cleaned, removed = autofix_trojan_source(text)
+ assert removed == 1
+ assert cleaned == "a\u200bb"
+ assert "\u202e" not in cleaned
+
+
+def test_bidi_in_comment_or_string_stays_critical_beside_emoji() -> None:
+ comment = scan_hidden_characters("//\U0001F600\u202e")
+ assert comment.findings[0].category == CATEGORY_BIDI_CONTROL
+ assert comment.findings[0].context == "comment"
+ assert comment.findings[0].severity == "critical"
+ quoted = scan_hidden_characters('"\U0001F600\u202e"')
+ assert quoted.findings[0].context == "string"
+ assert quoted.findings[0].severity == "critical"
+ emoji = scan_hidden_characters("// \U0001F468\u200d\U0001F469")
+ zwj = next(item for item in emoji.findings if item.category == CATEGORY_ZERO_WIDTH)
+ assert zwj.context == "emoji"
+ assert zwj.severity == "info"
+
+
+def test_highest_severity_includes_hits_past_findings_cap() -> None:
+ text = "\u200b" * 300 + "a\u202eb"
+ result = scan_hidden_characters(text, max_findings=256)
+ assert result.truncated is True
+ assert result.total == 301
+ assert len(result.findings) == 256
+ assert result.counts[CATEGORY_BIDI_CONTROL] == 1
+ assert result.highest_severity == "critical"
diff --git a/tests/test_web.py b/tests/test_web.py
index 8b18183d..fc4f8553 100644
--- a/tests/test_web.py
+++ b/tests/test_web.py
@@ -47,6 +47,26 @@ def test_packaged_mark_html_matches_docs_source() -> None:
assert web_root() == mark_html_path().parent
+def test_packaged_scan_page_matches_docs_and_editor_engine() -> None:
+ docs_html = (ROOT / "docs" / "scan.html").read_text(encoding="utf-8")
+ packaged_html = (ROOT / "fuckmark" / "webui" / "scan.html").read_text(encoding="utf-8")
+ assert packaged_html == docs_html
+ canonical_js = (ROOT / "editors" / "vscode" / "scan.js").read_text(encoding="utf-8")
+ assert (ROOT / "docs" / "scan.js").read_text(encoding="utf-8") == canonical_js
+ assert (ROOT / "fuckmark" / "webui" / "scan.js").read_text(encoding="utf-8") == canonical_js
+ assert 'script src="scan.js"' in docs_html
+ assert "FuckMarkScan" in docs_html
+ assert "autofixTrojanSource" in docs_html
+ assert "DEFAULT_SECURITY_CATEGORIES" in docs_html
+ assert "fromCodePoint(0x202E)" in docs_html
+ assert "truncated" in docs_html
+ assert "bidi_control" in docs_html
+ from fuckmark.product.scan import scan_hidden_characters
+
+ assert scan_hidden_characters(docs_html, language="html").detected is False
+ assert scan_hidden_characters(canonical_js, language="javascript").detected is False
+
+
def test_mark_html_miss_copy_is_english() -> None:
html = mark_html_path().read_text(encoding="utf-8")
assert "We did not detect a watermark in this text." in html
@@ -112,6 +132,11 @@ def on_ready(url: str, port: int) -> None:
seen["cache"] = response.headers.get("Cache-Control")
with urlopen(f"http://127.0.0.1:{port}/", timeout=2) as response:
seen["index"] = response.read().decode("utf-8")
+ with urlopen(f"http://127.0.0.1:{port}/scan.html", timeout=2) as response:
+ seen["scan_html"] = response.read().decode("utf-8")
+ seen["scan_cache"] = response.headers.get("Cache-Control")
+ with urlopen(f"http://127.0.0.1:{port}/scan.js", timeout=2) as response:
+ seen["scan_js"] = response.read().decode("utf-8")
health_status, health = _get_json(f"http://127.0.0.1:{port}/api/health")
seen["health_status"] = health_status
seen["health"] = health
@@ -145,6 +170,7 @@ def on_ready(url: str, port: int) -> None:
assert "FuckMark web:" in log
assert "Python API" in log
assert "/api/remove-marks" in log
+ assert "/scan.html" in log
body = str(seen["body"])
assert "FuckMark" in body
assert "detectFuckMark" in body
@@ -152,6 +178,11 @@ def on_ready(url: str, port: int) -> None:
assert "/api/remove-marks" in body
assert "We did not detect a watermark in this text." in body
assert "detectFuckMark" in str(seen["index"])
+ scan_html = str(seen["scan_html"])
+ assert "See the bytes." in scan_html
+ assert 'script src="scan.js"' in scan_html
+ assert "FuckMarkScan" in str(seen["scan_js"])
+ assert "no-store" in str(seen.get("scan_cache", "")).casefold()
assert "no-store" in str(seen.get("cache", "")).casefold()
assert seen["health_status"] == 200
health = seen["health"]
@@ -191,4 +222,5 @@ def test_cli_web_help_documents_browser_tool() -> None:
help_text = out.getvalue() + errors.getvalue()
assert "fuckmark web" in help_text
assert "Python API" in help_text
+ assert "scan.html" in help_text
assert "browser" in help_text.casefold() or "beginner" in help_text.casefold()