diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 16d7e82f..14bb931d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -167,3 +167,22 @@ jobs: name: middev-calibration-readiness path: calibration-readiness/ if-no-files-found: error + + wasm-scan: + name: WASM hidden-scan + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v7 + + - name: Install Rust + uses: dtolnay/rust-toolchain@stable + with: + targets: wasm32-unknown-unknown + + - name: Test crate and build wasm32 + run: | + cargo test --manifest-path crates/fuckmark-scan/Cargo.toml + cargo build --release --target wasm32-unknown-unknown --manifest-path crates/fuckmark-scan/Cargo.toml + test -f crates/fuckmark-scan/target/wasm32-unknown-unknown/release/fuckmark_scan.wasm + diff --git a/.gitignore b/.gitignore index b9e83619..b3bb9347 100644 --- a/.gitignore +++ b/.gitignore @@ -14,3 +14,4 @@ venv/ .env.* .DS_Store artifacts/ +target/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 57455b41..29cc3eff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ - Language-aware Trojan Source analysis and autofix. `scan_hidden_characters(..., language=)` (and lint, from the file suffix) classifies source as `comment` / `string` / code. `autofix_trojan_source` / `autofixTrojanSource` strip only `bidi_control`. - Zero-install hidden-Unicode scan page: `docs/scan.html` (packaged as `fuckmark/webui/scan.html`, engine copy `scan.js`). `fuckmark web` serves it at `/scan.html`. After deploy: `mark.q1z.org/scan.html`. Scanning runs in the browser; a local Python `/api/scan` confirm is optional when the CLI server is up. - Chromium browser extension under `editors/browser`: popup scanner, on-page reveal (badges plus CSS highlights), context-menu scan, per-tab toolbar badge, and optional paste-safe stripping that keeps real emoji ZWJ sequences (both neighbors emoji) and stops page paste handlers from seeing the raw clipboard. Zero remote code; `scan.js` matches the VS Code engine copy. Load unpacked from `chrome://extensions`. +- WASM build of `fuckmark-hidden-scan-v1` (`crates/fuckmark-scan`, committed `fuckmark_scan.wasm`). The scan page and the Chromium popup load it when `fetch` works so classification stays on-device; `scan.js` remains the `file://` fallback. Lone UTF-16 surrogates route through the JS engine (UTF-8 cannot carry them). Empty category arrays stay empty selections. The extension CSP allows `wasm-unsafe-eval` so Chromium can instantiate the module. Page reveal and paste-safe keep the synchronous JS port. Rebuild with `crates/fuckmark-scan/build-wasm.sh`. - `fuckmark normalize` (`fuckmark-normalize-v1`): NFC, optional identifier lookalike fold (UTS #39-inspired subset, not a full confusable map), then strip of the security category set, plus a JSON receipt (`input_sha256` / `output_sha256` / `steps` / `report_hash`). `--clean` remains the blunt strip with no receipt. HTTP: `POST /api/normalize`. Python: `normalize_text`, `skeleton_fold`. Reference: `docs/normalize.md`. - VS Code / Cursor: `fuckmark.cleanOnSave` (default off) strips hidden Unicode on save. - Frozen confirmation files, hashes, mix mechanisms, and the published wheel are unchanged. Package version stays 0.4.1. diff --git a/README.md b/README.md index 87bd2e24..30f4d261 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Opens a local page at `http://127.0.0.1:8765/mark.html`. Paste text in the brows `fuckmark web` also serves a local Python API. Detect and strip POST to `/api/remove-marks` and run the same engine as `fuckmark --detect`. Opening `mark.html` with `file://` or the static website has no Python process, so those paths keep the in-browser fallback. - Product paste UI source: [`docs/mark.html`](docs/mark.html) (packaged as `fuckmark/webui/mark.html`) — after deploy: [mark.q1z.org/mark.html](https://mark.q1z.org/mark.html) -- Hidden-Unicode scan (no install): [`docs/scan.html`](docs/scan.html) — `file://` works; `fuckmark web` serves `/scan.html`; after deploy: [mark.q1z.org/scan.html](https://mark.q1z.org/scan.html) +- Hidden-Unicode scan (no install): [`docs/scan.html`](docs/scan.html) — `file://` works; `fuckmark web` serves `/scan.html`; after deploy: [mark.q1z.org/scan.html](https://mark.q1z.org/scan.html). The page prefers a local WASM copy of the same classifier (`fuckmark_scan.wasm`) and falls back to `scan.js`. - Research demo: [`docs/demo.html`](docs/demo.html) — `file://` works; after deploy: [mark.q1z.org/demo.html](https://mark.q1z.org/demo.html) `mark.html` runs a local closed-set FuckMark scan, strips approved insertions when found, and on a miss shows an English no-watermark card with contact to `Fhelp@q1z.org`. Under `fuckmark web` that scan is the Python detector; the JS copy is the fallback. @@ -181,7 +181,7 @@ The [`editors/vscode`](editors/vscode) extension reveals hidden Unicode inline a ## See it in the browser (Chrome / Edge / Brave) -The [`editors/browser`](editors/browser) extension reveals hidden Unicode on the page you are reading. Load it unpacked from `chrome://extensions` (Developer mode). The popup scans pasted text or the current tab, **Reveal** paints `U+XXXX` badges on the page, and **Paste-safe** (off by default) strips hidden characters when you paste into a field while keeping emoji sequences. Nothing is uploaded; the scanner is the same `scan.js` as the editor extension. Full notes: [`editors/browser/README.md`](editors/browser/README.md). +The [`editors/browser`](editors/browser) extension reveals hidden Unicode on the page you are reading. Load it unpacked from `chrome://extensions` (Developer mode). The popup scans pasted text or the current tab, **Reveal** paints `U+XXXX` badges on the page, and **Paste-safe** (off by default) strips hidden characters when you paste into a field while keeping emoji sequences. Nothing is uploaded. The popup prefers `fuckmark_scan.wasm` when it can fetch it; on-page reveal and paste-safe keep `scan.js`. Full notes: [`editors/browser/README.md`](editors/browser/README.md). ## Guard model input (LLM prompt-injection smuggling) @@ -255,6 +255,7 @@ Frozen evidence, hashes, and protocols: [`docs/research.md`](docs/research.md). ```text python -m pip install -e ".[dev]" python -m pytest +cargo test --manifest-path crates/fuckmark-scan/Cargo.toml ``` ## License diff --git a/crates/fuckmark-scan/Cargo.lock b/crates/fuckmark-scan/Cargo.lock new file mode 100644 index 00000000..cc63e444 --- /dev/null +++ b/crates/fuckmark-scan/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "fuckmark-scan" +version = "0.4.1" diff --git a/crates/fuckmark-scan/Cargo.toml b/crates/fuckmark-scan/Cargo.toml new file mode 100644 index 00000000..f9044667 --- /dev/null +++ b/crates/fuckmark-scan/Cargo.toml @@ -0,0 +1,17 @@ +[package] +name = "fuckmark-scan" +version = "0.4.1" +edition = "2021" +license = "MIT" +publish = false +description = "fuckmark-hidden-scan-v1 classifier for native and wasm32" + +[lib] +crate-type = ["cdylib", "rlib"] + +[profile.release] +opt-level = "z" +lto = true +codegen-units = 1 +panic = "abort" +strip = true diff --git a/crates/fuckmark-scan/README.md b/crates/fuckmark-scan/README.md new file mode 100644 index 00000000..1dbb4c44 --- /dev/null +++ b/crates/fuckmark-scan/README.md @@ -0,0 +1,18 @@ +# fuckmark-scan (Rust / WASM) + +Native and `wasm32` build of `fuckmark-hidden-scan-v1`. Same category table as +`editors/vscode/scan.js` and `fuckmark.product.scan`. No network, no extra +crate dependencies. + +```text +rustup target add wasm32-unknown-unknown +./crates/fuckmark-scan/build-wasm.sh +``` + +That writes `fuckmark_scan.wasm` next to the scan page, the packaged web UI, +and the Chromium popup. The page and popup load it when `fetch` works and keep +`scan.js` as the `file://` fallback. + +`cargo test` in this directory covers classification, Trojan Source roles, and +clean. Conformance vectors are also replayed from Python via Node on the +committed module. diff --git a/crates/fuckmark-scan/build-wasm.sh b/crates/fuckmark-scan/build-wasm.sh new file mode 100755 index 00000000..a7e6a39e --- /dev/null +++ b/crates/fuckmark-scan/build-wasm.sh @@ -0,0 +1,22 @@ +#!/bin/sh +set -eu +root="$(CDPATH= cd -- "$(dirname "$0")/../.." && pwd)" +crate="$root/crates/fuckmark-scan" +cd "$crate" +rustup target add wasm32-unknown-unknown >/dev/null +cargo test --offline --quiet 2>/dev/null || cargo test --quiet +cargo build --release --target wasm32-unknown-unknown --quiet +src="$crate/target/wasm32-unknown-unknown/release/fuckmark_scan.wasm" +test -f "$src" +for dest in \ + "$root/editors/wasm/fuckmark_scan.wasm" \ + "$root/docs/fuckmark_scan.wasm" \ + "$root/fuckmark/webui/fuckmark_scan.wasm" \ + "$root/editors/browser/fuckmark_scan.wasm" +do + cp "$src" "$dest" +done +cp "$root/editors/wasm/scan_wasm.js" "$root/docs/scan_wasm.js" +cp "$root/editors/wasm/scan_wasm.js" "$root/fuckmark/webui/scan_wasm.js" +cp "$root/editors/wasm/scan_wasm.js" "$root/editors/browser/scan_wasm.js" +wc -c "$src" diff --git a/crates/fuckmark-scan/src/lib.rs b/crates/fuckmark-scan/src/lib.rs new file mode 100644 index 00000000..8844438f --- /dev/null +++ b/crates/fuckmark-scan/src/lib.rs @@ -0,0 +1,821 @@ +pub const SCAN_ALGORITHM_VERSION: &str = "fuckmark-hidden-scan-v1"; + +pub const CATEGORY_BIDI_CONTROL: &str = "bidi_control"; +pub const CATEGORY_ZERO_WIDTH: &str = "zero_width"; +pub const CATEGORY_VARIATION_SELECTOR: &str = "variation_selector"; +pub const CATEGORY_TAG: &str = "tag"; +pub const CATEGORY_ENCLOSING_MARK: &str = "enclosing_mark"; +pub const CATEGORY_LINE_SEPARATOR: &str = "line_separator"; +pub const CATEGORY_DEPRECATED: &str = "deprecated"; +pub const CATEGORY_FORMAT: &str = "format"; +pub const CATEGORY_CONTROL: &str = "control"; +pub const CATEGORY_PRIVATE_USE: &str = "private_use"; +pub const CATEGORY_NONCHARACTER: &str = "noncharacter"; +pub const CATEGORY_SURROGATE: &str = "surrogate"; + +pub const SCAN_CATEGORIES: [&str; 12] = [ + CATEGORY_BIDI_CONTROL, + CATEGORY_ZERO_WIDTH, + CATEGORY_VARIATION_SELECTOR, + CATEGORY_TAG, + CATEGORY_ENCLOSING_MARK, + CATEGORY_LINE_SEPARATOR, + CATEGORY_DEPRECATED, + CATEGORY_FORMAT, + CATEGORY_CONTROL, + CATEGORY_PRIVATE_USE, + CATEGORY_NONCHARACTER, + CATEGORY_SURROGATE, +]; + +const ALLOWED_WHITESPACE: [u32; 4] = [0x09, 0x0A, 0x0D, 0x20]; +const BIDI_EXTRA: [u32; 3] = [0x061C, 0x200E, 0x200F]; +const ZERO_WIDTH: [u32; 18] = [ + 0x00AD, 0x034F, 0x115F, 0x1160, 0x17B4, 0x17B5, 0x180E, 0x200B, 0x200C, 0x200D, 0x2060, 0x2061, + 0x2062, 0x2063, 0x2064, 0x3164, 0xFEFF, 0xFFA0, +]; +const FORMAT_RANGES: [(u32, u32); 10] = [ + (0x0600, 0x0605), + (0x06DD, 0x06DD), + (0x070F, 0x070F), + (0x0890, 0x0891), + (0x08E2, 0x08E2), + (0x110BD, 0x110BD), + (0x110CD, 0x110CD), + (0x13430, 0x1343F), + (0x1BCA0, 0x1BCA3), + (0x1D173, 0x1D17A), +]; +const EMOJI_BMP: [u32; 10] = [ + 0x00A9, 0x00AE, 0x203C, 0x2049, 0x2122, 0x2139, 0x3030, 0x303D, 0x3297, 0x3299, +]; +const ZWJ: u32 = 0x200D; + +fn in_range(cp: u32, lo: u32, hi: u32) -> bool { + cp >= lo && cp <= hi +} + +fn contains_u32(table: &[u32], cp: u32) -> bool { + table.iter().any(|item| *item == cp) +} + +pub fn classify(cp: u32) -> Option<&'static str> { + if contains_u32(&ALLOWED_WHITESPACE, cp) { + return None; + } + if contains_u32(&BIDI_EXTRA, cp) || in_range(cp, 0x202A, 0x202E) || in_range(cp, 0x2066, 0x2069) + { + return Some(CATEGORY_BIDI_CONTROL); + } + if in_range(cp, 0x206A, 0x206F) || cp == 0xFFF9 || cp == 0xFFFA || cp == 0xFFFB { + return Some(CATEGORY_DEPRECATED); + } + if contains_u32(&ZERO_WIDTH, cp) { + return Some(CATEGORY_ZERO_WIDTH); + } + if in_range(cp, 0xFE00, 0xFE0F) || in_range(cp, 0xE0100, 0xE01EF) { + return Some(CATEGORY_VARIATION_SELECTOR); + } + if in_range(cp, 0xE0000, 0xE007F) { + return Some(CATEGORY_TAG); + } + if in_range(cp, 0xFDD0, 0xFDEF) || (cp & 0xFFFF) == 0xFFFE || (cp & 0xFFFF) == 0xFFFF { + return Some(CATEGORY_NONCHARACTER); + } + if in_range(cp, 0xD800, 0xDFFF) { + return Some(CATEGORY_SURROGATE); + } + if in_range(cp, 0x0488, 0x0489) + || cp == 0x1ABE + || in_range(cp, 0x20DD, 0x20E0) + || in_range(cp, 0x20E2, 0x20E4) + || in_range(cp, 0xA670, 0xA672) + { + return Some(CATEGORY_ENCLOSING_MARK); + } + if cp == 0x2028 || cp == 0x2029 { + return Some(CATEGORY_LINE_SEPARATOR); + } + if in_range(cp, 0x00, 0x1F) || in_range(cp, 0x7F, 0x9F) { + return Some(CATEGORY_CONTROL); + } + if FORMAT_RANGES.iter().any(|(lo, hi)| in_range(cp, *lo, *hi)) { + return Some(CATEGORY_FORMAT); + } + if in_range(cp, 0xE000, 0xF8FF) + || in_range(cp, 0xF0000, 0xFFFFD) + || in_range(cp, 0x100000, 0x10FFFD) + { + return Some(CATEGORY_PRIVATE_USE); + } + None +} + +pub fn category_index(name: &str) -> Option { + SCAN_CATEGORIES + .iter() + .position(|item| *item == name) + .map(|index| index as i32) +} + +pub fn classify_index(cp: u32) -> i32 { + match classify(cp) { + Some(name) => category_index(name).unwrap_or(-1), + None => -1, + } +} + +pub fn normalize_language(language: &str) -> &'static str { + let key = language.trim().to_ascii_lowercase(); + match key.as_str() { + "" | "auto" => "auto", + "javascript" | "js" | "ts" | "typescript" | "jsx" | "tsx" => "javascript", + "c" | "h" | "cc" | "cpp" | "cxx" | "java" | "go" | "rs" | "rust" | "cs" | "css" + | "jsonc" => "c", + "python" | "py" | "pyi" | "hash" | "sh" | "bash" | "zsh" | "shell" | "shellscript" + | "yaml" | "yml" | "rb" | "ruby" | "toml" => "python", + "html" | "htm" | "xml" => "html", + "sql" => "sql", + _ => "auto", + } +} + +fn slash_comments(language: &str) -> bool { + matches!(language, "auto" | "javascript" | "c" | "sql") +} + +fn block_comments(language: &str) -> bool { + slash_comments(language) +} + +fn hash_comments(language: &str) -> bool { + language == "python" +} + +fn sql_line_comments(language: &str) -> bool { + language == "sql" +} + +fn html_comments(language: &str) -> bool { + language == "html" +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Role { + Code, + String, + Comment, +} + +fn role_name(role: Role) -> &'static str { + match role { + Role::Code => "code", + Role::String => "string", + Role::Comment => "comment", + } +} + +fn find_seq(chars: &[char], start: usize, needle: &[char]) -> Option { + if needle.is_empty() || start > chars.len() { + return None; + } + chars[start..] + .windows(needle.len()) + .position(|window| window == needle) + .map(|offset| start + offset) +} + +pub fn source_roles(text: &str, language: &str) -> Vec<&'static str> { + let lang = normalize_language(language); + let chars: Vec = text.chars().collect(); + let mut roles = vec![Role::Code; chars.len()]; + let length = chars.len(); + let mut index = 0; + let mut in_string: Option = None; + let mut escape = false; + while index < length { + let character = chars[index]; + if in_string.is_some() { + roles[index] = Role::String; + if escape { + escape = false; + index += 1; + continue; + } + if character == '\\' && index + 1 < length { + escape = true; + index += 1; + continue; + } + if Some(character) == in_string { + in_string = None; + } + index += 1; + continue; + } + let pair = if index + 1 < length { + Some((character, chars[index + 1])) + } else { + None + }; + if slash_comments(lang) + && pair == Some(('/', '/')) + && !(index > 0 && chars[index - 1] == ':') + { + let mut cursor = index; + while cursor < length && chars[cursor] != '\n' && chars[cursor] != '\r' { + roles[cursor] = Role::Comment; + cursor += 1; + } + index = cursor; + continue; + } + if sql_line_comments(lang) && pair == Some(('-', '-')) { + let mut cursor = index; + while cursor < length && chars[cursor] != '\n' && chars[cursor] != '\r' { + roles[cursor] = Role::Comment; + cursor += 1; + } + index = cursor; + continue; + } + if block_comments(lang) && pair == Some(('/', '*')) { + let close = find_seq(&chars, index + 2, &['*', '/']); + let stop = close.map(|pos| pos + 2).unwrap_or(length); + for role in roles.iter_mut().take(stop).skip(index) { + *role = Role::Comment; + } + index = stop; + continue; + } + if hash_comments(lang) && character == '#' { + let mut cursor = index; + while cursor < length && chars[cursor] != '\n' && chars[cursor] != '\r' { + roles[cursor] = Role::Comment; + cursor += 1; + } + index = cursor; + continue; + } + if html_comments(lang) && chars.get(index..index + 4) == Some(&['<', '!', '-', '-'][..]) { + let close = find_seq(&chars, index + 4, &['-', '-', '>']); + let stop = close.map(|pos| pos + 3).unwrap_or(length); + for role in roles.iter_mut().take(stop).skip(index) { + *role = Role::Comment; + } + index = stop; + continue; + } + if character == '"' || character == '\'' || character == '`' { + in_string = Some(character); + roles[index] = Role::String; + index += 1; + continue; + } + index += 1; + } + roles.into_iter().map(role_name).collect() +} + +fn is_emojiish(cp: i32) -> bool { + if cp < 0 { + return false; + } + let cp = cp as u32; + if in_range(cp, 0x1F1E6, 0x1F1FF) + || in_range(cp, 0x1F000, 0x1FAFF) + || in_range(cp, 0x2600, 0x27BF) + { + return true; + } + cp == ZWJ || contains_u32(&EMOJI_BMP, cp) +} + +fn is_ident_char(ch: Option) -> bool { + match ch { + Some('_') => true, + Some(character) => character.is_alphanumeric(), + None => false, + } +} + +fn is_bmp_vs(cp: i32) -> bool { + cp >= 0 && in_range(cp as u32, 0xFE00, 0xFE0F) +} + +pub fn classify_context(text: &str, index: usize, role: &str, category: &str) -> &'static str { + let chars: Vec = text.chars().collect(); + classify_context_chars(&chars, index, role, category) +} + +fn classify_context_chars( + chars: &[char], + index: usize, + role: &str, + category: &str, +) -> &'static str { + if index >= chars.len() { + return "prose"; + } + let prev = if index > 0 { + Some(chars[index - 1]) + } else { + None + }; + let next = chars.get(index + 1).copied(); + let prev_cp = prev.map(|ch| ch as u32 as i32).unwrap_or(-1); + let next_cp = next.map(|ch| ch as u32 as i32).unwrap_or(-1); + if (role == "comment" || role == "string") && category == CATEGORY_BIDI_CONTROL { + return if role == "comment" { + "comment" + } else { + "string" + }; + } + if is_emojiish(prev_cp) || is_emojiish(next_cp) || is_bmp_vs(prev_cp) || is_bmp_vs(next_cp) { + return "emoji"; + } + if role == "comment" { + return "comment"; + } + if role == "string" { + return "string"; + } + if is_ident_char(prev) || is_ident_char(next) { + return "identifier"; + } + "prose" +} + +pub fn score_severity(category: &str, context: &str) -> &'static str { + if category == CATEGORY_TAG { + return "critical"; + } + if category == CATEGORY_BIDI_CONTROL { + if matches!(context, "identifier" | "comment" | "string") { + return "critical"; + } + return "high"; + } + if category == CATEGORY_ZERO_WIDTH { + if context == "emoji" { + return "info"; + } + if context == "identifier" { + return "high"; + } + return "medium"; + } + if category == CATEGORY_VARIATION_SELECTOR { + return if context == "emoji" { "info" } else { "medium" }; + } + if matches!(category, "control" | "noncharacter" | "surrogate") { + return "high"; + } + "medium" +} + +pub fn explain_finding( + category: &str, + context: &str, + severity: &str, +) -> (&'static str, &'static str) { + if category == CATEGORY_BIDI_CONTROL && context == "identifier" { + return ( + "Bidirectional override sits inside an identifier, so the glyphs can read differently than the bytes (Trojan Source).", + "Strip the bidi control and keep the identifier left-to-right.", + ); + } + if category == CATEGORY_BIDI_CONTROL && context == "comment" { + return ( + "Bidirectional override sits inside a comment, so commented-out code can appear to run (Trojan Source commenting-out).", + "Strip the bidi control from the comment.", + ); + } + if category == CATEGORY_BIDI_CONTROL && context == "string" { + return ( + "Bidirectional override sits inside a string, so the literal can appear to close early (Trojan Source stretched-string).", + "Strip the bidi control from the string.", + ); + } + if category == CATEGORY_BIDI_CONTROL { + return ( + "Bidirectional override can reorder nearby glyphs (Trojan Source class, CVE-2021-42574).", + "Strip U+202A-U+202E / U+2066-U+2069 and rewrite the text left-to-right.", + ); + } + if category == CATEGORY_TAG { + return ( + "Unicode tag characters encode a second ASCII string that models read and humans do not.", + "Strip U+E0020-U+E007F; inspect tag_payload for the smuggled text.", + ); + } + if category == CATEGORY_ZERO_WIDTH && context == "emoji" { + return ( + "Zero-width joiner or invisible mark inside an emoji cluster; usually a legitimate emoji sequence.", + "Leave emoji ZWJ sequences unless you are sanitizing for a security boundary.", + ); + } + if category == CATEGORY_ZERO_WIDTH && context == "identifier" { + return ( + "Zero-width character splits an identifier, breaking search and some compilers while looking unchanged.", + "Strip the zero-width character from the identifier.", + ); + } + if category == CATEGORY_ZERO_WIDTH { + return ( + "Invisible spacing or joining character that changes the byte stream without changing the glyphs.", + "Strip the zero-width character.", + ); + } + if category == CATEGORY_VARIATION_SELECTOR && context == "emoji" { + return ( + "Variation selector tunes an emoji glyph; usually benign.", + "Keep emoji variation selectors unless you are stripping all hidden marks.", + ); + } + if severity == "high" { + return ( + "Hidden or non-text codepoint that should not appear in ordinary source or prompts.", + "Strip the character.", + ); + } + ( + "Hidden or format codepoint that is invisible or renderer-defined.", + "Strip the character if this text crosses a trust boundary.", + ) +} + +fn severity_rank(severity: &str) -> i32 { + match severity { + "info" => 0, + "medium" => 1, + "high" => 2, + "critical" => 3, + _ => -1, + } +} + +fn parse_categories(raw: &str) -> Option> { + let trimmed = raw.trim(); + if trimmed == "*" { + return None; + } + let mut selected = Vec::new(); + if trimmed.is_empty() { + return Some(selected); + } + for part in trimmed.split(',') { + let name = part.trim(); + if let Some(known) = SCAN_CATEGORIES.iter().copied().find(|item| *item == name) { + if !selected.contains(&known) { + selected.push(known); + } + } + } + Some(selected) +} + +fn json_escape(s: &str) -> String { + let mut out = String::from("\""); + for character in s.chars() { + match character { + '"' => out.push_str("\\\""), + '\\' => out.push_str("\\\\"), + '\n' => out.push_str("\\n"), + '\r' => out.push_str("\\r"), + '\t' => out.push_str("\\t"), + c if (c as u32) < 0x20 => out.push_str(&format!("\\u{:04x}", c as u32)), + c => out.push(c), + } + } + out.push('"'); + out +} + +#[derive(Clone)] +pub struct HiddenFinding { + pub index: usize, + pub codepoint: u32, + pub category: &'static str, + pub context: &'static str, + pub severity: &'static str, + pub why: &'static str, + pub remedy: &'static str, +} + +pub struct ScanResult { + pub source_length: usize, + pub total: usize, + pub truncated: bool, + pub highest_severity: String, + pub counts: Vec<(&'static str, usize)>, + pub findings: Vec, +} + +pub fn scan_text(text: &str, language: &str, categories: &str, max_findings: i32) -> ScanResult { + let selected = parse_categories(categories); + let roles = source_roles(text, language); + let chars: Vec = text.chars().collect(); + let mut counts = [0usize; 12]; + let mut findings = Vec::new(); + let mut total = 0usize; + let mut truncated = false; + let mut peak = ""; + let cap = if max_findings < 0 { + usize::MAX + } else { + max_findings as usize + }; + for (index, character) in chars.iter().enumerate() { + let code = *character as u32; + let Some(category) = classify(code) else { + continue; + }; + if let Some(allowed) = selected.as_ref() { + if !allowed.contains(&category) { + continue; + } + } + total += 1; + if let Some(slot) = category_index(category) { + counts[slot as usize] += 1; + } + let role = roles.get(index).copied().unwrap_or("code"); + let context = classify_context_chars(&chars, index, role, category); + let severity = score_severity(category, context); + let (why, remedy) = explain_finding(category, context, severity); + if severity_rank(severity) > severity_rank(peak) { + peak = severity; + } + if findings.len() < cap { + findings.push(HiddenFinding { + index, + codepoint: code, + category, + context, + severity, + why, + remedy, + }); + } else { + truncated = true; + } + } + let mut named_counts = Vec::new(); + for (index, name) in SCAN_CATEGORIES.iter().enumerate() { + if counts[index] > 0 { + named_counts.push((*name, counts[index])); + } + } + ScanResult { + source_length: chars.len(), + total, + truncated, + highest_severity: peak.to_string(), + counts: named_counts, + findings, + } +} + +pub fn clean_text(text: &str, categories: &str) -> (String, usize) { + let selected = parse_categories(categories); + let mut kept = String::new(); + let mut removed = 0usize; + for character in text.chars() { + let category = classify(character as u32); + let drop = match (category, selected.as_ref()) { + (Some(name), Some(allowed)) => allowed.contains(&name), + (Some(_), None) => true, + (None, _) => false, + }; + if drop { + removed += 1; + } else { + kept.push(character); + } + } + (kept, removed) +} + +pub fn scan_to_json(text: &str, language: &str, categories: &str, max_findings: i32) -> String { + let result = scan_text(text, language, categories, max_findings); + let mut out = String::from("{"); + out.push_str("\"algorithm_version\":"); + out.push_str(&json_escape(SCAN_ALGORITHM_VERSION)); + out.push_str(",\"source_length\":"); + out.push_str(&result.source_length.to_string()); + out.push_str(",\"total\":"); + out.push_str(&result.total.to_string()); + out.push_str(",\"truncated\":"); + out.push_str(if result.truncated { "true" } else { "false" }); + out.push_str(",\"highest_severity\":"); + out.push_str(&json_escape(&result.highest_severity)); + out.push_str(",\"counts\":{"); + for (index, (name, count)) in result.counts.iter().enumerate() { + if index > 0 { + out.push(','); + } + out.push_str(&json_escape(name)); + out.push(':'); + out.push_str(&count.to_string()); + } + out.push_str("},\"findings\":["); + for (index, finding) in result.findings.iter().enumerate() { + if index > 0 { + out.push(','); + } + out.push('{'); + out.push_str("\"index\":"); + out.push_str(&finding.index.to_string()); + out.push_str(",\"codepoint\":"); + out.push_str(&finding.codepoint.to_string()); + out.push_str(",\"category\":"); + out.push_str(&json_escape(finding.category)); + out.push_str(",\"context\":"); + out.push_str(&json_escape(finding.context)); + out.push_str(",\"severity\":"); + out.push_str(&json_escape(finding.severity)); + out.push_str(",\"why\":"); + out.push_str(&json_escape(finding.why)); + out.push_str(",\"remedy\":"); + out.push_str(&json_escape(finding.remedy)); + out.push('}'); + } + out.push_str("]}"); + out +} + +pub fn clean_to_json(text: &str, categories: &str) -> String { + let (cleaned, removed) = clean_text(text, categories); + let mut out = String::from("{\"removed\":"); + out.push_str(&removed.to_string()); + out.push_str(",\"cleaned\":"); + out.push_str(&json_escape(&cleaned)); + out.push('}'); + out +} + +fn pack_bytes(payload: &[u8]) -> *mut u8 { + let mut buffer = vec![0u8; 4 + payload.len()]; + buffer[..4].copy_from_slice(&(payload.len() as u32).to_le_bytes()); + buffer[4..].copy_from_slice(payload); + let ptr = buffer.as_mut_ptr(); + std::mem::forget(buffer); + ptr +} + +fn read_bytes<'a>(ptr: *const u8, len: u32) -> &'a [u8] { + if ptr.is_null() || len == 0 { + return &[]; + } + unsafe { std::slice::from_raw_parts(ptr, len as usize) } +} + +#[no_mangle] +pub extern "C" fn fm_alloc(size: u32) -> *mut u8 { + let mut buffer = vec![0u8; size as usize]; + let ptr = buffer.as_mut_ptr(); + std::mem::forget(buffer); + ptr +} + +#[no_mangle] +pub extern "C" fn fm_dealloc(ptr: *mut u8, size: u32) { + if ptr.is_null() { + return; + } + unsafe { + let _ = Vec::from_raw_parts(ptr, size as usize, size as usize); + } +} + +#[no_mangle] +pub extern "C" fn fm_classify(cp: u32) -> i32 { + classify_index(cp) +} + +#[no_mangle] +pub extern "C" fn fm_scan( + text_ptr: *const u8, + text_len: u32, + lang_ptr: *const u8, + lang_len: u32, + cats_ptr: *const u8, + cats_len: u32, + max_findings: i32, +) -> *mut u8 { + let text = match std::str::from_utf8(read_bytes(text_ptr, text_len)) { + Ok(value) => value.to_owned(), + Err(_) => return pack_bytes(br#"{"ok":false,"reason":"utf8"}"#), + }; + let language = std::str::from_utf8(read_bytes(lang_ptr, lang_len)) + .unwrap_or("auto") + .to_owned(); + let categories = std::str::from_utf8(read_bytes(cats_ptr, cats_len)) + .unwrap_or("") + .to_owned(); + let json = scan_to_json(&text, &language, &categories, max_findings); + pack_bytes(json.as_bytes()) +} + +#[no_mangle] +pub extern "C" fn fm_clean( + text_ptr: *const u8, + text_len: u32, + cats_ptr: *const u8, + cats_len: u32, +) -> *mut u8 { + let text = match std::str::from_utf8(read_bytes(text_ptr, text_len)) { + Ok(value) => value.to_owned(), + Err(_) => return pack_bytes(br#"{"ok":false,"reason":"utf8"}"#), + }; + let categories = std::str::from_utf8(read_bytes(cats_ptr, cats_len)) + .unwrap_or("") + .to_owned(); + let json = clean_to_json(&text, &categories); + pack_bytes(json.as_bytes()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn classifies_core_hidden_codepoints() { + assert_eq!(classify(0x202E), Some(CATEGORY_BIDI_CONTROL)); + assert_eq!(classify(0x200B), Some(CATEGORY_ZERO_WIDTH)); + assert_eq!(classify(0xE0061), Some(CATEGORY_TAG)); + assert_eq!(classify(0x20DD), Some(CATEGORY_ENCLOSING_MARK)); + assert_eq!(classify(0x13430), Some(CATEGORY_FORMAT)); + assert_eq!(classify(0x09), None); + assert_eq!(classify(0x61), None); + assert_eq!(classify(0x0301), None); + } + + #[test] + fn bidi_in_identifier_is_critical() { + let text: String = ['a', '\u{202E}', 'b'].into_iter().collect(); + let result = scan_text(&text, "auto", "*", -1); + assert_eq!(result.total, 1); + assert_eq!(result.findings[0].context, "identifier"); + assert_eq!(result.findings[0].severity, "critical"); + } + + #[test] + fn url_slash_slash_is_not_a_comment() { + let text = "http://example.com/\u{202E}"; + let roles = source_roles(text, "auto"); + assert!(roles.iter().all(|role| *role == "code")); + let result = scan_text(text, "auto", "*", -1); + assert_eq!(result.findings[0].context, "prose"); + assert_eq!(result.findings[0].severity, "high"); + } + + #[test] + fn python_hash_comment_is_critical_bidi() { + let text = "x = 1 # \u{202E} no"; + let result = scan_text(text, "python", "*", -1); + assert_eq!(result.findings[0].context, "comment"); + assert_eq!(result.findings[0].severity, "critical"); + let auto = scan_text(text, "auto", "*", -1); + assert_ne!(auto.findings[0].context, "comment"); + } + + #[test] + fn emoji_zwj_is_info() { + let text = "\u{1F468}\u{200D}\u{1F469}"; + let result = scan_text(text, "auto", "*", -1); + assert_eq!(result.findings[0].context, "emoji"); + assert_eq!(result.findings[0].severity, "info"); + } + + #[test] + fn clean_strips_bidi_only() { + let text = "a\u{202E}b\u{200B}c"; + let (cleaned, removed) = clean_text(text, "bidi_control"); + assert_eq!(removed, 1); + assert_eq!(cleaned, "ab\u{200B}c"); + } + + #[test] + fn empty_category_selection_finds_and_removes_nothing() { + let text = "a\u{202E}b\u{200B}c"; + let empty = scan_text(text, "auto", "", -1); + assert_eq!(empty.total, 0); + assert!(empty.findings.is_empty()); + let all = scan_text(text, "auto", "*", -1); + assert_eq!(all.total, 2); + let (cleaned, removed) = clean_text(text, ""); + assert_eq!(removed, 0); + assert_eq!(cleaned, text); + } + + #[test] + fn dense_zero_width_scan_stays_linear() { + let text = "\u{200B}".repeat(20_000); + let result = scan_text(&text, "auto", "*", -1); + assert_eq!(result.total, 20_000); + assert_eq!(result.findings.len(), 20_000); + } +} diff --git a/docs/cli.md b/docs/cli.md index d7919a20..72109046 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -118,7 +118,7 @@ Existing files are read as UTF-8 bytes with no newline conversion. LF, CRLF, CR, Findings include context (`identifier`, `comment`, `string`, `emoji`, `prose`) and severity (`critical`, `high`, `medium`, `info`). A bidi override in an identifier, comment, or string is `critical` (Trojan Source); a ZWJ inside an emoji cluster is `info`. Comment syntax follows `language` (`auto` treats `//` and `/* */` as comments; `python` also treats `#`; `sql` treats `--`; `html` treats ``). The frozen table is [`specs/fuckmark-hidden-scan-v1.protocol.md`](../specs/fuckmark-hidden-scan-v1.protocol.md). -`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. `autofix_trojan_source` strips only `bidi_control`. The browser scan page is `docs/scan.html` (`fuckmark web` serves `/scan.html`); the API is `POST /api/scan` with optional `"language"`. +`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. `autofix_trojan_source` strips only `bidi_control`. The browser scan page is `docs/scan.html` (`fuckmark web` serves `/scan.html`); it prefers the WASM engine (`fuckmark_scan.wasm`) and falls back to `scan.js`. The API is `POST /api/scan` with optional `"language"`. ### `fuckmark normalize` @@ -143,7 +143,7 @@ fuckmark guard --refuse --receipt --json < messages.json ### `fuckmark web` -Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`. Hidden-Unicode reveal is `/scan.html` (same engine as `--scan`; zero-install copy at `docs/scan.html`). +Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`. Hidden-Unicode reveal is `/scan.html` (WASM engine with `scan.js` fallback; zero-install copy at `docs/scan.html`). ```text fuckmark web diff --git a/docs/fuckmark_scan.wasm b/docs/fuckmark_scan.wasm new file mode 100755 index 00000000..5c459b0f Binary files /dev/null and b/docs/fuckmark_scan.wasm differ diff --git a/docs/scan.html b/docs/scan.html index 0b007977..6457e557 100644 --- a/docs/scan.html +++ b/docs/scan.html @@ -288,7 +288,9 @@

See the bytes.

    - Scanning is fuckmark-hidden-scan-v1. Copy uses the security category set + Scanning is fuckmark-hidden-scan-v1. The page prefers the local WASM engine + (fuckmark_scan.wasm) and keeps scan.js as the file:// + fallback. Copy uses the security category set (bidi_control, zero_width, tag, control, noncharacter, surrogate). Fix Trojan Source strips only bidirectional controls. Nothing is uploaded from a static page or file://. @@ -298,6 +300,7 @@

    See the bytes.

    + + diff --git a/editors/browser/popup.js b/editors/browser/popup.js index 28c27ebc..d3b7979a 100644 --- a/editors/browser/popup.js +++ b/editors/browser/popup.js @@ -158,6 +158,13 @@ pasteSafe.addEventListener("change", async () => { } else { scanPage.classList.add("primary"); } + if (typeof globalThis.loadFuckMarkScanWasm === "function") { + try { + await globalThis.loadFuckMarkScanWasm("fuckmark_scan.wasm"); + } catch (_err) { + void _err; + } + } if (typeof chrome !== "undefined" && chrome.storage) { const stored = await chrome.storage.sync.get({ pasteSafe: false }); pasteSafe.checked = Boolean(stored.pasteSafe); diff --git a/editors/browser/scan_wasm.js b/editors/browser/scan_wasm.js new file mode 100644 index 00000000..5f01e7cd --- /dev/null +++ b/editors/browser/scan_wasm.js @@ -0,0 +1,209 @@ +"use strict"; + +const WASM_CATEGORIES = [ + "bidi_control", + "zero_width", + "variation_selector", + "tag", + "enclosing_mark", + "line_separator", + "deprecated", + "format", + "control", + "private_use", + "noncharacter", + "surrogate", +]; + +function utf16Offsets(text) { + const map = []; + let offset = 0; + while (offset < text.length) { + map.push(offset); + const cp = text.codePointAt(offset); + offset += cp > 0xffff ? 2 : 1; + } + return map; +} + +function codepointToken(cp) { + return "U+" + Number(cp).toString(16).toUpperCase().padStart(4, "0"); +} + +function hasLoneSurrogate(text) { + const source = String(text || ""); + for (let index = 0; index < source.length; index += 1) { + const unit = source.charCodeAt(index); + if (unit >= 0xd800 && unit <= 0xdbff) { + if (index + 1 >= source.length) return true; + const next = source.charCodeAt(index + 1); + if (next < 0xdc00 || next > 0xdfff) return true; + index += 1; + continue; + } + if (unit >= 0xdc00 && unit <= 0xdfff) return true; + } + return false; +} + +function encodeCategories(categories) { + if (categories == null) return "*"; + if (Array.isArray(categories)) return categories.join(","); + return String(categories); +} + +function writeUtf8(memory, alloc, bytes) { + const ptr = alloc(bytes.length); + new Uint8Array(memory.buffer, ptr, bytes.length).set(bytes); + return ptr; +} + +function readPackedJson(memory, dealloc, ptr) { + const view = new DataView(memory.buffer, ptr, 4); + const length = view.getUint32(0, true); + const bytes = new Uint8Array(memory.buffer, ptr + 4, length); + const text = new TextDecoder("utf-8").decode(bytes); + dealloc(ptr, 4 + length); + return JSON.parse(text); +} + +function wrapInstance(instance, fallback) { + const { memory, fm_alloc, fm_dealloc, fm_classify, fm_scan, fm_clean } = instance.exports; + const encoder = new TextEncoder(); + + function callScan(text, language, categories, maxFindings) { + const textBytes = encoder.encode(String(text || "")); + const langBytes = encoder.encode(String(language || "auto")); + const catBytes = encoder.encode(encodeCategories(categories)); + const textPtr = writeUtf8(memory, fm_alloc, textBytes); + const langPtr = writeUtf8(memory, fm_alloc, langBytes); + const catPtr = writeUtf8(memory, fm_alloc, catBytes); + const resultPtr = fm_scan( + textPtr, + textBytes.length, + langPtr, + langBytes.length, + catPtr, + catBytes.length, + maxFindings == null ? -1 : maxFindings + ); + fm_dealloc(textPtr, textBytes.length); + fm_dealloc(langPtr, langBytes.length); + fm_dealloc(catPtr, catBytes.length); + return readPackedJson(memory, fm_dealloc, resultPtr); + } + + function callClean(text, categories) { + const textBytes = encoder.encode(String(text || "")); + const catBytes = encoder.encode(encodeCategories(categories)); + const textPtr = writeUtf8(memory, fm_alloc, textBytes); + const catPtr = writeUtf8(memory, fm_alloc, catBytes); + const resultPtr = fm_clean(textPtr, textBytes.length, catPtr, catBytes.length); + fm_dealloc(textPtr, textBytes.length); + fm_dealloc(catPtr, catBytes.length); + return readPackedJson(memory, fm_dealloc, resultPtr); + } + + function scanText(text, categories, language) { + const source = String(text || ""); + if (fallback && hasLoneSurrogate(source)) { + return fallback.scanText(source, categories, language); + } + const payload = callScan(source, language || "auto", categories, -1); + const offsets = utf16Offsets(source); + const findings = (payload.findings || []).map((finding) => { + const offset = offsets[finding.index] || 0; + const length = finding.codepoint > 0xffff ? 2 : 1; + return { + offset, + length, + index: finding.index, + codepoint: finding.codepoint, + category: finding.category, + context: finding.context, + severity: finding.severity, + why: finding.why, + remedy: finding.remedy, + }; + }); + return { + total: payload.total || 0, + counts: payload.counts || {}, + findings, + truncated: Boolean(payload.truncated), + highest_severity: payload.highest_severity || "", + source_length: payload.source_length || 0, + }; + } + + function cleanText(text, categories) { + const source = String(text || ""); + if (fallback && hasLoneSurrogate(source)) { + return fallback.cleanText(source, categories); + } + return callClean(source, categories); + } + + const api = Object.assign({}, fallback || {}, { + engine: "wasm", + SCAN_ALGORITHM_VERSION: "fuckmark-hidden-scan-v1", + classify(cp) { + const index = fm_classify(cp >>> 0); + return index < 0 ? null : WASM_CATEGORIES[index]; + }, + scanText, + cleanText, + autofixTrojanSource(text) { + return cleanText(text, ["bidi_control"]); + }, + codepointToken: (fallback && fallback.codepointToken) || codepointToken, + hasLoneSurrogate, + encodeCategories, + }); + return api; +} + +async function instantiateWasm(source) { + let buffer; + if (source instanceof ArrayBuffer) { + buffer = source; + } else if (ArrayBuffer.isView(source)) { + buffer = source.buffer.slice(source.byteOffset, source.byteOffset + source.byteLength); + } else if (typeof source === "string") { + const response = await fetch(source); + if (!response.ok) { + throw new Error("fuckmark_scan.wasm missing"); + } + buffer = await response.arrayBuffer(); + } else { + throw new Error("wasm source required"); + } + const result = await WebAssembly.instantiate(buffer, {}); + return result.instance; +} + +async function loadFuckMarkScanWasm(source) { + const fallback = typeof globalThis !== "undefined" ? globalThis.FuckMarkScan : undefined; + const instance = await instantiateWasm(source || "fuckmark_scan.wasm"); + const api = wrapInstance(instance, fallback); + if (typeof globalThis !== "undefined") { + globalThis.FuckMarkScan = api; + globalThis.FuckMarkScanWasm = api; + } + if (typeof module !== "undefined" && module.exports) { + module.exports.loadFuckMarkScanWasm = loadFuckMarkScanWasm; + module.exports.wrapInstance = wrapInstance; + } + return api; +} + +if (typeof module !== "undefined" && module.exports) { + module.exports.loadFuckMarkScanWasm = loadFuckMarkScanWasm; + module.exports.wrapInstance = wrapInstance; + module.exports.WASM_CATEGORIES = WASM_CATEGORIES; + module.exports.hasLoneSurrogate = hasLoneSurrogate; + module.exports.encodeCategories = encodeCategories; +} +if (typeof globalThis !== "undefined") { + globalThis.loadFuckMarkScanWasm = loadFuckMarkScanWasm; +} diff --git a/editors/wasm/fuckmark_scan.wasm b/editors/wasm/fuckmark_scan.wasm new file mode 100755 index 00000000..5c459b0f Binary files /dev/null and b/editors/wasm/fuckmark_scan.wasm differ diff --git a/editors/wasm/scan_wasm.js b/editors/wasm/scan_wasm.js new file mode 100644 index 00000000..5f01e7cd --- /dev/null +++ b/editors/wasm/scan_wasm.js @@ -0,0 +1,209 @@ +"use strict"; + +const WASM_CATEGORIES = [ + "bidi_control", + "zero_width", + "variation_selector", + "tag", + "enclosing_mark", + "line_separator", + "deprecated", + "format", + "control", + "private_use", + "noncharacter", + "surrogate", +]; + +function utf16Offsets(text) { + const map = []; + let offset = 0; + while (offset < text.length) { + map.push(offset); + const cp = text.codePointAt(offset); + offset += cp > 0xffff ? 2 : 1; + } + return map; +} + +function codepointToken(cp) { + return "U+" + Number(cp).toString(16).toUpperCase().padStart(4, "0"); +} + +function hasLoneSurrogate(text) { + const source = String(text || ""); + for (let index = 0; index < source.length; index += 1) { + const unit = source.charCodeAt(index); + if (unit >= 0xd800 && unit <= 0xdbff) { + if (index + 1 >= source.length) return true; + const next = source.charCodeAt(index + 1); + if (next < 0xdc00 || next > 0xdfff) return true; + index += 1; + continue; + } + if (unit >= 0xdc00 && unit <= 0xdfff) return true; + } + return false; +} + +function encodeCategories(categories) { + if (categories == null) return "*"; + if (Array.isArray(categories)) return categories.join(","); + return String(categories); +} + +function writeUtf8(memory, alloc, bytes) { + const ptr = alloc(bytes.length); + new Uint8Array(memory.buffer, ptr, bytes.length).set(bytes); + return ptr; +} + +function readPackedJson(memory, dealloc, ptr) { + const view = new DataView(memory.buffer, ptr, 4); + const length = view.getUint32(0, true); + const bytes = new Uint8Array(memory.buffer, ptr + 4, length); + const text = new TextDecoder("utf-8").decode(bytes); + dealloc(ptr, 4 + length); + return JSON.parse(text); +} + +function wrapInstance(instance, fallback) { + const { memory, fm_alloc, fm_dealloc, fm_classify, fm_scan, fm_clean } = instance.exports; + const encoder = new TextEncoder(); + + function callScan(text, language, categories, maxFindings) { + const textBytes = encoder.encode(String(text || "")); + const langBytes = encoder.encode(String(language || "auto")); + const catBytes = encoder.encode(encodeCategories(categories)); + const textPtr = writeUtf8(memory, fm_alloc, textBytes); + const langPtr = writeUtf8(memory, fm_alloc, langBytes); + const catPtr = writeUtf8(memory, fm_alloc, catBytes); + const resultPtr = fm_scan( + textPtr, + textBytes.length, + langPtr, + langBytes.length, + catPtr, + catBytes.length, + maxFindings == null ? -1 : maxFindings + ); + fm_dealloc(textPtr, textBytes.length); + fm_dealloc(langPtr, langBytes.length); + fm_dealloc(catPtr, catBytes.length); + return readPackedJson(memory, fm_dealloc, resultPtr); + } + + function callClean(text, categories) { + const textBytes = encoder.encode(String(text || "")); + const catBytes = encoder.encode(encodeCategories(categories)); + const textPtr = writeUtf8(memory, fm_alloc, textBytes); + const catPtr = writeUtf8(memory, fm_alloc, catBytes); + const resultPtr = fm_clean(textPtr, textBytes.length, catPtr, catBytes.length); + fm_dealloc(textPtr, textBytes.length); + fm_dealloc(catPtr, catBytes.length); + return readPackedJson(memory, fm_dealloc, resultPtr); + } + + function scanText(text, categories, language) { + const source = String(text || ""); + if (fallback && hasLoneSurrogate(source)) { + return fallback.scanText(source, categories, language); + } + const payload = callScan(source, language || "auto", categories, -1); + const offsets = utf16Offsets(source); + const findings = (payload.findings || []).map((finding) => { + const offset = offsets[finding.index] || 0; + const length = finding.codepoint > 0xffff ? 2 : 1; + return { + offset, + length, + index: finding.index, + codepoint: finding.codepoint, + category: finding.category, + context: finding.context, + severity: finding.severity, + why: finding.why, + remedy: finding.remedy, + }; + }); + return { + total: payload.total || 0, + counts: payload.counts || {}, + findings, + truncated: Boolean(payload.truncated), + highest_severity: payload.highest_severity || "", + source_length: payload.source_length || 0, + }; + } + + function cleanText(text, categories) { + const source = String(text || ""); + if (fallback && hasLoneSurrogate(source)) { + return fallback.cleanText(source, categories); + } + return callClean(source, categories); + } + + const api = Object.assign({}, fallback || {}, { + engine: "wasm", + SCAN_ALGORITHM_VERSION: "fuckmark-hidden-scan-v1", + classify(cp) { + const index = fm_classify(cp >>> 0); + return index < 0 ? null : WASM_CATEGORIES[index]; + }, + scanText, + cleanText, + autofixTrojanSource(text) { + return cleanText(text, ["bidi_control"]); + }, + codepointToken: (fallback && fallback.codepointToken) || codepointToken, + hasLoneSurrogate, + encodeCategories, + }); + return api; +} + +async function instantiateWasm(source) { + let buffer; + if (source instanceof ArrayBuffer) { + buffer = source; + } else if (ArrayBuffer.isView(source)) { + buffer = source.buffer.slice(source.byteOffset, source.byteOffset + source.byteLength); + } else if (typeof source === "string") { + const response = await fetch(source); + if (!response.ok) { + throw new Error("fuckmark_scan.wasm missing"); + } + buffer = await response.arrayBuffer(); + } else { + throw new Error("wasm source required"); + } + const result = await WebAssembly.instantiate(buffer, {}); + return result.instance; +} + +async function loadFuckMarkScanWasm(source) { + const fallback = typeof globalThis !== "undefined" ? globalThis.FuckMarkScan : undefined; + const instance = await instantiateWasm(source || "fuckmark_scan.wasm"); + const api = wrapInstance(instance, fallback); + if (typeof globalThis !== "undefined") { + globalThis.FuckMarkScan = api; + globalThis.FuckMarkScanWasm = api; + } + if (typeof module !== "undefined" && module.exports) { + module.exports.loadFuckMarkScanWasm = loadFuckMarkScanWasm; + module.exports.wrapInstance = wrapInstance; + } + return api; +} + +if (typeof module !== "undefined" && module.exports) { + module.exports.loadFuckMarkScanWasm = loadFuckMarkScanWasm; + module.exports.wrapInstance = wrapInstance; + module.exports.WASM_CATEGORIES = WASM_CATEGORIES; + module.exports.hasLoneSurrogate = hasLoneSurrogate; + module.exports.encodeCategories = encodeCategories; +} +if (typeof globalThis !== "undefined") { + globalThis.loadFuckMarkScanWasm = loadFuckMarkScanWasm; +} diff --git a/fuckmark/web.py b/fuckmark/web.py index 865b90df..03fba62f 100644 --- a/fuckmark/web.py +++ b/fuckmark/web.py @@ -171,6 +171,11 @@ def _web_parser() -> argparse.ArgumentParser: class _MarkHandler(http.server.SimpleHTTPRequestHandler): + extensions_map = { + **http.server.SimpleHTTPRequestHandler.extensions_map, + ".wasm": "application/wasm", + } + def log_message(self, format: str, *args) -> None: return diff --git a/fuckmark/webui/fuckmark_scan.wasm b/fuckmark/webui/fuckmark_scan.wasm new file mode 100755 index 00000000..5c459b0f Binary files /dev/null and b/fuckmark/webui/fuckmark_scan.wasm differ diff --git a/fuckmark/webui/scan.html b/fuckmark/webui/scan.html index 0b007977..6457e557 100644 --- a/fuckmark/webui/scan.html +++ b/fuckmark/webui/scan.html @@ -288,7 +288,9 @@

    See the bytes.

      - Scanning is fuckmark-hidden-scan-v1. Copy uses the security category set + Scanning is fuckmark-hidden-scan-v1. The page prefers the local WASM engine + (fuckmark_scan.wasm) and keeps scan.js as the file:// + fallback. Copy uses the security category set (bidi_control, zero_width, tag, control, noncharacter, surrogate). Fix Trojan Source strips only bidirectional controls. Nothing is uploaded from a static page or file://. @@ -298,6 +300,7 @@

      See the bytes.

      +