From 17a12485dd5751e458161a7464d895d968027119 Mon Sep 17 00:00:00 2001 From: Nick Date: Fri, 18 Sep 2026 14:41:04 +0300 Subject: [PATCH 1/3] docs: make CLI quickstart non-destructive and disclose current I/O limits --- README.md | 63 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 4e4e86c..d410f0c 100644 --- a/README.md +++ b/README.md @@ -3,14 +3,16 @@ # ๐Ÿงน DocStripper - > **AI-powered batch document cleaner** โ€” Remove noise from text documents automatically + > **Batch document cleaner for TXT, DOCX and PDF text** โ€” Rule-based cleanup with an optional browser AI mode [![Python Version](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://www.python.org/downloads/) [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE.txt) [![Product Hunt](https://img.shields.io/badge/Product%20Hunt-Featured-orange)](https://www.producthunt.com/products/docstripper) -**DocStripper** automatically removes noise from text documents. Remove page numbers, headers/footers, duplicate lines, and empty lines from `.txt`, `.docx`, and `.pdf` files. Choose between **Fast Clean** (instant) or **Smart Clean** (AI-powered). In the web app, DocStripper processes selected document files in your browser and does not upload their contents. The page still makes network requests for third-party resources, analytics, translation, and Smart Clean model downloads. No account is required. +**DocStripper** removes common text noise such as page numbers, headers/footers, duplicate lines, and empty lines. The browser app offers rule-based **Fast Clean** and optional **Smart Clean (Beta)** with an on-device model; the Python CLI uses rule-based text processing. DOCX and PDF support means text extraction, not preservation of the original document layout. + +In the web app, DocStripper processes selected document files in your browser and does not upload their contents. The page still makes network requests for third-party resources, analytics, translation, and Smart Clean model downloads. No account is required. **๐ŸŒ [Try it online โ†’](https://kikuai-lab.github.io/DocStripper/)** โ€” No installation needed! @@ -39,54 +41,67 @@ ### Web App (Recommended) 1. Visit [https://kikuai-lab.github.io/DocStripper/](https://kikuai-lab.github.io/DocStripper/) -2. Upload your files -3. Choose **Fast Clean** (instant) or **Smart Clean** (AI-powered) +2. Choose your local files +3. Choose **Fast Clean** (rule-based) or **Smart Clean (Beta)** (AI-powered) 4. Adjust **Cleaning Temperament** slider: Gentle (recommended), Moderate, Thorough, or Aggressive 5. Click "Start Cleaning" -6. Download or copy the cleaned results +6. Review the preview, then download or copy the cleaned results ### CLI Tool +> [!WARNING] +> Start with `--dry-run` and copies of your files. Without `--dry-run` or +> `--stdout`, the current CLI writes cleaned **plain text back to the input +> path**, including a `.docx` or `.pdf` path. It does not rebuild those document +> formats. Do not use in-place cleaning on original DOCX/PDF files. + #### Installation Options **Option 1: Homebrew (macOS)** ```bash brew tap KikuAI-Lab/docstripper brew install docstripper -docstripper document.txt +docstripper --dry-run document.txt ``` **Option 2: Manual Installation** ```bash git clone https://github.com/KikuAI-Lab/DocStripper.git cd DocStripper -python tool.py document.txt +python3 tool.py --dry-run document.txt ``` -See [INSTALL.md](INSTALL.md) for detailed installation instructions. +Replace `document.txt` with a copy of an existing local file. See +[INSTALL.md](INSTALL.md) for detailed installation instructions. CLI PDF text +extraction requires `pdftotext`; see [Supported Formats](#๏ธ-supported-formats). #### Usage ```bash -# Clean a file -python tool.py document.txt +# Preview changes without modifying the input +python3 tool.py --dry-run document.txt -# Clean multiple files -python tool.py file1.txt file2.txt file3.docx +# Preview multiple inputs without overwriting their formats +python3 tool.py --dry-run file1.txt file2.txt file3.docx -# Preview changes (dry-run) -python tool.py --dry-run document.txt +# Inspect PDF text in the terminal without modifying the source PDF +python3 tool.py --keep-headers input.pdf --stdout -# Undo last operation -python tool.py --undo - -# Pipe stdin to stdout (no file writes) -cat input.pdf | python tool.py - --stdout > output.txt +# Only after reviewing: clean a disposable plain-text copy in place +python3 tool.py document-copy.txt -# Keep headers/footers if needed -python tool.py --keep-headers input.pdf --stdout +# Attempt to restore the last logged in-place operation from its backup +python3 tool.py --undo ``` +**Current CLI I/O limitations:** `--stdout` avoids modifying the input, but +progress messages, separators, and statistics also go to stdout. Do not treat +redirected output as a clean machine-readable transcript. The CLI entry point +currently checks `-` as a filesystem path, so piped stdin is not a working +entry point; pass an existing filename instead. Piping raw PDF bytes would not +perform PDF extraction either. These are implementation limitations, not +features fixed by this documentation. + --- ## ๐Ÿ“– Example @@ -162,7 +177,7 @@ Important content here. - `--no-normalize-ws` โ€” disable whitespace normalization - `--no-normalize-unicode` โ€” disable Unicode punctuation normalization - `--keep-headers` โ€” keep headers/footers/page numbers -- `--stdout` โ€” write cleaned text to stdout instead of modifying files (supports `-` for stdin) +- `--stdout` โ€” print cleaned text without modifying the input; currently also includes diagnostics (see CLI I/O limitations above) **Protection Features:** - โœ… Lists are never merged or broken @@ -176,8 +191,8 @@ Important content here. | Format | Status | Notes | |--------|--------|-------| | `.txt` | โœ… Full | UTF-8, Latin-1 | -| `.docx` | โœ… Basic | Text extraction only (Web + CLI) | -| `.pdf` | โœ… Basic | Text extraction only (Web + CLI). Web uses PDF.js automatically. CLI requires `pdftotext` (poppler-utils) | +| `.docx` | โœ… Basic | Text extraction only (Web + CLI); use a non-writing CLI mode to preserve the source document | +| `.pdf` | โœ… Basic | Text extraction only (Web + CLI). Web uses PDF.js automatically. CLI requires `pdftotext` (poppler-utils); use a non-writing CLI mode to preserve the source document | **PDF Support:** - macOS: `brew install poppler` From 6621d7a6766df96b95d324c1e2c160564b3ececc Mon Sep 17 00:00:00 2001 From: Nick Date: Fri, 18 Sep 2026 14:41:39 +0300 Subject: [PATCH 2/3] docs: synchronize wiki quickstart with CLI safety boundaries --- docs/wiki/Home.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md index a11c8e9..26e0997 100644 --- a/docs/wiki/Home.md +++ b/docs/wiki/Home.md @@ -28,9 +28,9 @@ ### Web App 1. Visit https://kikuai-lab.github.io/DocStripper/ -2. Upload your files (.txt, .docx, .pdf) +2. Choose your local files (.txt, .docx, .pdf) 3. Click "Start Cleaning" -4. Download or copy results +4. Review the preview, then download or copy results ### CLI ```bash @@ -38,13 +38,21 @@ git clone https://github.com/KikuAI-Lab/DocStripper.git cd DocStripper -# Clean a file -python tool.py document.txt +# Preview an existing file without modifying it +python3 tool.py --dry-run document.txt -# Pipe stdin to stdout -cat file.pdf | python tool.py - --stdout > clean.txt +# Inspect extracted PDF text in the terminal; requires pdftotext +python3 tool.py file.pdf --stdout ``` +Use copies and start with `--dry-run`. The current CLI otherwise writes plain +text back to the input path, including DOCX/PDF paths; it does not reconstruct +those formats. `--stdout` preserves the input but also prints diagnostics, so +it is not a clean machine-readable export. The current entry point does not +support normal piped stdin via `-`. Pass a filename instead, and do not pipe +raw PDF bytes into a text-cleaning command. See the +[README CLI safety notes](https://github.com/KikuAI-Lab/DocStripper#cli-tool). + ## ๐Ÿ“– What Gets Removed? ### Default (Conservative) From fb0ac5a45c068ebb486803baebe3401d57a08b3c Mon Sep 17 00:00:00 2001 From: Nick Date: Fri, 18 Sep 2026 14:42:09 +0300 Subject: [PATCH 3/3] docs: remove broken stdin recipes and warn about format-overwriting CLI behavior --- docs/wiki/Usage.md | 91 +++++++++++++++++++++++++++++----------------- 1 file changed, 57 insertions(+), 34 deletions(-) diff --git a/docs/wiki/Usage.md b/docs/wiki/Usage.md index 49b8ff5..9ce269f 100644 --- a/docs/wiki/Usage.md +++ b/docs/wiki/Usage.md @@ -37,38 +37,44 @@ ## CLI Tool +> [!WARNING] +> Use copies and start with `--dry-run`. Without `--dry-run` or `--stdout`, +> the current CLI overwrites the input path with plain text. DOCX/PDF inputs +> are not rebuilt as DOCX/PDF documents. Do not use in-place cleaning on +> original files in those formats. + ### Basic Usage ```bash -# Clean a single file -python tool.py document.txt +# Preview a single file without modifying it +python3 tool.py --dry-run document.txt -# Clean multiple files -python tool.py file1.txt file2.txt file3.docx +# Preview multiple files without changing their formats +python3 tool.py --dry-run file1.txt file2.txt file3.docx -# Preview changes without modifying files -python tool.py --dry-run document.txt +# Inspect cleaned text in the terminal without modifying the input +python3 tool.py document.txt --stdout -# Undo last operation -python tool.py --undo +# Attempt to restore the last logged in-place operation from its backup +python3 tool.py --undo ``` ### Supported Formats - `.txt` - Plain text files -- `.docx` - Microsoft Word documents -- `.pdf` - PDF files (requires poppler-utils) +- `.docx` - Text extraction from Microsoft Word documents; layout is not preserved +- `.pdf` - Text extraction from PDF files; requires `pdftotext` from poppler-utils ### Command Options -```bash -python tool.py [OPTIONS] [FILES...] +```text +python3 tool.py [OPTIONS] [FILES...] Options: -h, --help Show help message - --dry-run Preview changes without modifying files - --undo Restore files from last operation - --stdout Write cleaned text to stdout (no file writes) + --dry-run Preview diagnostics without modifying files + --undo Restore files from the last logged operation when backups exist + --stdout Print cleaned text and diagnostics without modifying the input --keep-headers Keep headers/footers/page numbers --no-merge-lines Disable merging broken lines --no-dehyphenate Disable de-hyphenation across line breaks @@ -78,44 +84,61 @@ Options: ### Examples -#### Example 1: Clean a single document +#### Example 1: Clean a disposable plain-text copy ```bash -python tool.py report.txt +cp report.txt report-copy.txt +python3 tool.py --dry-run report-copy.txt +# Run only after reviewing the preview: +python3 tool.py report-copy.txt ``` -#### Example 2: Clean multiple documents +#### Example 2: Inspect multiple documents without overwriting them ```bash -python tool.py document1.txt document2.docx document3.pdf +python3 tool.py --dry-run document1.txt document2.docx document3.pdf ``` -#### Example 3: Preview before cleaning +#### Example 3: Keep headers during inspection ```bash -python tool.py --dry-run important_document.txt +python3 tool.py --keep-headers important_document.txt --stdout ``` -#### Example 4: Undo last operation +#### Example 4: Undo the last logged in-place operation ```bash -python tool.py --undo +python3 tool.py --undo ``` -#### Example 5: Use stdin/stdout +#### Example 5: Inspect extracted PDF text ```bash -cat report.pdf | python tool.py - --stdout > report.txt +python3 tool.py report.pdf --stdout ``` -### Output +### Current I/O limitations + +The main entry point checks `-` as a filesystem path, so normal piped stdin is +not currently supported. Pass an existing filename. Raw PDF bytes must go +through the PDF extraction path, not through a text decoder. + +The `--stdout` path currently includes progress messages, separators, and final +statistics. Redirecting it to a file does not produce a clean transcript for +another program. This documentation does not fix the CLI implementation; it +avoids advertising unsupported pipeline behavior. + +### Output and backups + +For normal in-place processing, the tool creates `.bak` files, writes cleaned +plain text over the originals, and logs operations in `.strip-log`. Keep an +independent copy: repeated operations may replace a previous backup. Restoring +with `--undo` depends on the relevant log and backup files still being present. -- Original files are backed up with `.bak` extension -- Processed files replace originals -- Statistics are shown in console -- Operation log saved to `.strip-log` +`--dry-run` does not write a cleaned input file. `--stdout` also avoids modifying +the input, but includes the diagnostic output described above. ## Best Practices -1. **Always test on copies first** - Use `--dry-run` or test on copies -2. **Backup important files** - The tool creates backups, but extra backups never hurt -3. **Review statistics** - Check what was removed before finalizing -4. **Use appropriate mode** - Fast Clean for simple documents, Smart Clean for complex ones +1. **Always test on copies first** - Start with `--dry-run`. +2. **Keep independent backups** - A `.bak` file is not a replacement for versioned originals. +3. **Review the extracted text** - DOCX/PDF support does not preserve layout or formatting. +4. **Choose the correct surface** - Browser modes and Python CLI behavior are not identical. ## Troubleshooting