Language-aware Trojan Source analysis and zero-install scan page

CHANGELOG.md

@@ -11,7 +11,9 @@
 - LLM input guard (`fuckmark-guard-v1`). `protect()` / `inspect()` / `Guard` / `@Guard().wrap` sanitize strings, lists, and dicts (OpenAI-style `messages` included) before a model call. Default action is strip; `on_findings="refuse"` raises `HiddenTextRefused` without calling the inner function; `report` scans only. Unicode tag characters are decoded to ASCII as `tag_payload` on the receipt (`extract_tag_payload`). CLI: `fuckmark guard` (`--json`, `--refuse`, `--report`, `--receipt`, `--select`). HTTP: `POST /api/guard`. JS: `editors/vscode/guard.js`. This does not detect semantic prompt injection. Default categories are the shared `SECURITY_SCAN_CATEGORIES` also used by `fuckmark lint`. Reference: `docs/guard.md`.
 - Hidden-scan `format` classification uses the same explicit Cf ranges as `editors/vscode/scan.js` (including U+13430-U+1343F and U+0890-U+0891) instead of only `unicodedata.category`, so Python 3.11 (Unicode 14) agrees with the JS port and with Python 3.12+ (Unicode 15).
 - Hidden-scan findings now carry **context**, **severity**, and a one-line **why/remedy**. A bidi override inside an identifier is `critical` (Trojan Source); a ZWJ inside an emoji cluster is `info`. Lint JSON, `--scan` reports, and the VS Code hover/Problems panel all show those fields. Diagnostic rank: `critical` is Error, `high` is Warning.
-- Frozen classification spec `fuckmark-hidden-scan-v1` under `specs/` (`protocol.md`, `vectors.json`, `freeze.json`). Conformance vectors are codepoint arrays only; `tests/test_hidden_scan_spec.py` replays them on the Python engine and on `editors/vscode/scan.js`.
+- Frozen classification spec `fuckmark-hidden-scan-v1` under `specs/` (`protocol.md`, `vectors.json`, `freeze.json`). Conformance vectors are codepoint arrays only; `tests/test_hidden_scan_spec.py` replays them on the Python engine and on `editors/vscode/scan.js`. Freeze record **v3** adds language-aware comment/string roles: a bidi override in a comment or string is `critical` even next to emoji, `#` is a comment only for `python`, and `//` after `:` is not a comment.
+- Language-aware Trojan Source analysis and autofix. `scan_hidden_characters(..., language=)` (and lint, from the file suffix) classifies source as `comment` / `string` / code. `autofix_trojan_source` / `autofixTrojanSource` strip only `bidi_control`.
+- Zero-install hidden-Unicode scan page: `docs/scan.html` (packaged as `fuckmark/webui/scan.html`, engine copy `scan.js`). `fuckmark web` serves it at `/scan.html`. After deploy: `mark.q1z.org/scan.html`. Scanning runs in the browser; a local Python `/api/scan` confirm is optional when the CLI server is up.
 - `fuckmark normalize` (`fuckmark-normalize-v1`): NFC, optional identifier lookalike fold (UTS #39-inspired subset, not a full confusable map), then strip of the security category set, plus a JSON receipt (`input_sha256` / `output_sha256` / `steps` / `report_hash`). `--clean` remains the blunt strip with no receipt. HTTP: `POST /api/normalize`. Python: `normalize_text`, `skeleton_fold`. Reference: `docs/normalize.md`.
 - VS Code / Cursor: `fuckmark.cleanOnSave` (default off) strips hidden Unicode on save.
 - Frozen confirmation files, hashes, mix mechanisms, and the published wheel are unchanged. Package version stays 0.4.1.

README.md

@@ -26,6 +26,7 @@ Opens a local page at `http://127.0.0.1:8765/mark.html`. Paste text in the brows
 `fuckmark web` also serves a local Python API. Detect and strip POST to `/api/remove-marks` and run the same engine as `fuckmark --detect`. Opening `mark.html` with `file://` or the static website has no Python process, so those paths keep the in-browser fallback.
 
 - Product paste UI source: [`docs/mark.html`](docs/mark.html) (packaged as `fuckmark/webui/mark.html`) — after deploy: [mark.q1z.org/mark.html](https://mark.q1z.org/mark.html)
+- Hidden-Unicode scan (no install): [`docs/scan.html`](docs/scan.html) — `file://` works; `fuckmark web` serves `/scan.html`; after deploy: [mark.q1z.org/scan.html](https://mark.q1z.org/scan.html)
 - Research demo: [`docs/demo.html`](docs/demo.html) — `file://` works; after deploy: [mark.q1z.org/demo.html](https://mark.q1z.org/demo.html)
 
 `mark.html` runs a local closed-set FuckMark scan, strips approved insertions when found, and on a miss shows an English no-watermark card with contact to `Fhelp@q1z.org`. Under `fuckmark web` that scan is the Python detector; the JS copy is the fallback.
@@ -132,7 +133,7 @@ printf 'if (x != \u202eadmin\u202c) {\n' | .venv/bin/fuckmark --scan
 .venv/bin/fuckmark --clean --file suspect.txt -o clean.txt
 ```
 
-`--scan` prints a human report by default, a machine line with `-q`, and a `fuckmark-scan ...` status line to stderr with `--status`. Findings include context (`identifier` / `emoji` / `string` / `prose`) and severity (`critical` / `high` / `medium` / `info`): a bidi override inside an identifier is critical; a ZWJ inside an emoji cluster is info. `--clean` removes every flagged category (including FuckMark's own carriers, so `--clean` reverses a mix back to the visible text) and reports how many characters it removed. The same engine is available in the browser tool via `POST /api/scan`, and in Python through `scan_hidden_characters`, `clean_hidden_characters`, and `classify_hidden_codepoint`. Frozen spec: [`specs/fuckmark-hidden-scan-v1.protocol.md`](specs/fuckmark-hidden-scan-v1.protocol.md).
+`--scan` prints a human report by default, a machine line with `-q`, and a `fuckmark-scan ...` status line to stderr with `--status`. Findings include context (`identifier` / `comment` / `string` / `emoji` / `prose`) and severity (`critical` / `high` / `medium` / `info`): a bidi override inside an identifier, comment, or string is critical (Trojan Source); a ZWJ inside an emoji cluster is info. Lint infers comment syntax from the file suffix (`javascript`, `python`, `sql`, `html`). `--clean` removes every flagged category (including FuckMark's own carriers, so `--clean` reverses a mix back to the visible text) and reports how many characters it removed. `autofix_trojan_source` strips only bidirectional controls. The same engine is in the browser at [`docs/scan.html`](docs/scan.html) (`fuckmark web` → `/scan.html`), via `POST /api/scan`, and in Python through `scan_hidden_characters`, `clean_hidden_characters`, and `classify_hidden_codepoint`. Frozen spec: [`specs/fuckmark-hidden-scan-v1.protocol.md`](specs/fuckmark-hidden-scan-v1.protocol.md).
 
 Whitespace tab, newline, carriage return, and space are never flagged, and ordinary combining accents (`Mn`) are left alone. `--clean` strips emoji zero-width joiners and variation selectors as well, so pass a category subset to `clean_hidden_characters(...)` in Python if you need to keep emoji sequences intact.
 
@@ -176,7 +177,7 @@ Binary, non-UTF-8, oversized, and vendored/VCS paths are skipped; `--exclude GLO
 
 ## See it in your editor (VS Code / Cursor)
 
-The [`editors/vscode`](editors/vscode) extension reveals hidden Unicode inline as you read code: a red box and a visible `‹U+202E›` badge on every hidden character, hover text with why it matters, diagnostics in the Problems panel (`critical` as Error), a status-bar count, one-click clean, and optional clean-on-save (`fuckmark.cleanOnSave`). It is zero-dependency plain JavaScript with no build step, and its scanner is a faithful port of `fuckmark-hidden-scan-v1` — pinned to the Python engine across every Unicode codepoint by [`tests/test_vscode_scanner_parity.py`](tests/test_vscode_scanner_parity.py), so the editor and the CLI agree exactly. Load it with `code --extensionDevelopmentPath=editors/vscode` or open the folder and press `F5`.
+The [`editors/vscode`](editors/vscode) extension reveals hidden Unicode inline as you read code: a red box and a visible `‹U+202E›` badge on every hidden character, hover text with why it matters, diagnostics in the Problems panel (`critical` as Error, including Trojan Source in comments and strings), a status-bar count, one-click clean, and optional clean-on-save (`fuckmark.cleanOnSave`). It is zero-dependency plain JavaScript with no build step, and its scanner is a faithful port of `fuckmark-hidden-scan-v1` — pinned to the Python engine across every Unicode codepoint by [`tests/test_vscode_scanner_parity.py`](tests/test_vscode_scanner_parity.py), so the editor and the CLI agree exactly. Load it with `code --extensionDevelopmentPath=editors/vscode` or open the folder and press `F5`.
 
 ## Guard model input (LLM prompt-injection smuggling)
 

docs/cli.md

@@ -116,9 +116,9 @@ Existing files are read as UTF-8 bytes with no newline conversion. LF, CRLF, CR,
 
 `--scan` and `--clean` are the defensive inverse of the mix. The scan is general, not FuckMark-only. Flagged categories: `bidi_control` (Trojan Source, CVE-2021-42574), `zero_width`, `variation_selector`, `tag` (hidden-text / prompt-injection smuggling), `enclosing_mark`, `line_separator`, `deprecated` (interlinear annotation and deprecated format controls), `format` (other `Cf`), `control` (C0/C1), `private_use`, `noncharacter`, and `surrogate`. Tab, newline, carriage return, and space are never flagged; ordinary combining accents (`Mn`) are left alone.
 
-Findings include context (`identifier`, `emoji`, `string`, `prose`) and severity (`critical`, `high`, `medium`, `info`). A bidi override next to an identifier is `critical`; a ZWJ inside an emoji cluster is `info`. The frozen table is [`specs/fuckmark-hidden-scan-v1.protocol.md`](../specs/fuckmark-hidden-scan-v1.protocol.md).
+Findings include context (`identifier`, `comment`, `string`, `emoji`, `prose`) and severity (`critical`, `high`, `medium`, `info`). A bidi override in an identifier, comment, or string is `critical` (Trojan Source); a ZWJ inside an emoji cluster is `info`. Comment syntax follows `language` (`auto` treats `//` and `/* */` as comments; `python` also treats `#`; `sql` treats `--`; `html` treats `<!-- -->`). The frozen table is [`specs/fuckmark-hidden-scan-v1.protocol.md`](../specs/fuckmark-hidden-scan-v1.protocol.md).
 
-`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. The browser tool exposes the same engine at `POST /api/scan`.
+`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. `autofix_trojan_source` strips only `bidi_control`. The browser scan page is `docs/scan.html` (`fuckmark web` serves `/scan.html`); the API is `POST /api/scan` with optional `"language"`.
 
 ### `fuckmark normalize`
 
@@ -143,7 +143,7 @@ fuckmark guard --refuse --receipt --json < messages.json
 
 ### `fuckmark web`
 
-Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`.
+Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`. Hidden-Unicode reveal is `/scan.html` (same engine as `--scan`; zero-install copy at `docs/scan.html`).
 
 ```text
 fuckmark web

docs/lint.md

@@ -34,7 +34,9 @@ Pass `--select all` to include `variation_selector`, `enclosing_mark`,
 `line_separator`, `deprecated`, `format`, and `private_use`, or pass an explicit
 comma-separated subset. See [`cli.md`](cli.md) for the full category list.
 JSON locations include `context`, `severity`, `why`, and `remedy` from
-`fuckmark-hidden-scan-v1`.
+`fuckmark-hidden-scan-v1`. Context uses the file suffix: `.js`/`.ts` get
+JavaScript comments (`//`, `/* */`), `.py` gets `#` comments, `.sql` gets `--`,
+`.html` gets `<!-- -->`. A bidi override in a comment or string is `critical`.
 
 ### What is skipped