diff --git a/README.md b/README.md index e9381b0..a1d136c 100644 --- a/README.md +++ b/README.md @@ -112,7 +112,7 @@ parser = UniParserClient( 提交解析任务时（`trigger_file` / `trigger_snip` / `trigger_url`），可分别设置 7 类语义元素的处理模式： | 字段 | 含义 | 枚举类型 | -|------|------|---------| +|------|------|------| | `textual` | 普通文本（段落、标题等） | `ParseModeTextual` | | `equation` | 数学公式 | `ParseMode` | | `table` | 表格 | `ParseMode` | @@ -127,19 +127,19 @@ parser = UniParserClient( |------|------|------| | `-3` / `-2` | `DumpHosting` / `DumpLocal` | 保留接口，默认关闭 | | `-1` | `DumpBase64` | 禁用解析，输出原始图像 Base64 | -| `0` | `Disable` | 禁用解析，不输出 | -| `1` | `OCRFast` | 快速 OCR（默认） | -| `2` | `OCRHighQuality` | 高质 OCR | +| `0` | `Disable` | 禁用解析，不输出 | +| `1` | `OCRFast` | 快速 OCR（默认） | +| `2` | `OCRHighQuality` | 高质 OCR | ### `ParseModeTextual`（仅用于 `textual`） | 取值 | 名称 | 含义 | |------|------|------| | `-1` | `DumpBase64` | 输出原始图像 Base64 | -| `0` | `Disable` | 不解析、不输出 | -| `1` | `OCRFast` | 快速 OCR | -| `2` | `OCRHighQuality` | 高质 OCR，支持行内公式 | -| `3` | `DigitalExported` | 从数字原生 PDF 直接抽取文字 | +| `0` | `Disable` | 不解析、不输出 | +| `1` | `OCRFast` | 快速 OCR | +| `2` | `OCRHighQuality` | 高质 OCR，支持行内公式 | +| `3` | `DigitalExported` | 从数字原生 PDF 直接抽取文字 | ## 快速开始 @@ -201,7 +201,7 @@ if result["status"] == "success": #### 输出格式（`FormatFlag`，仅作用于 `content` / `objects` 中的文本字段） | 取值 | 适用场景 | -|------|----------| +|------|------| | `FormatFlag.Plain` | 纯文本，适合检索 | | `FormatFlag.Markup` | 默认标记文本 | | `FormatFlag.Markdown` | ⭐ 推荐给 LLM | @@ -353,7 +353,7 @@ token = result["token"] 返回体字段约定： | 字段 | 出现场景 | 说明 | -|------|----------|------| +|------|------|------| | `status` | 始终存在 | `"success"` / `"error"`（见 `StatusFlag`） | | `token` | 触发/查询类接口 | 本次任务的 token，出错也会带上以便追溯 | | `description` | 错误时 | 业务层错误原因，通常取自 `ErrorFlag`（如 `Token_Invalid`、`File_Size_Exceeded`、`Domain_Not_Allowed`…）或本地 traceback | @@ -363,24 +363,30 @@ token = result["token"] ## 面向 AI Agent -本仓库提供 **Agent Skill**（`skills/UniParser-Tools/`），让 Cursor、Claude Code 等助手在对话中自动完成 PDF / 图片 / 公网 PDF 链接 → 结构化 Markdown 与版面 JSON 的解析。用户只需安装 Skill、配置 API Key，并用自然语言或下方触发词发起任务；具体执行步骤由 Skill 内的 `SKILL.md` 指导 Agent，无需手动敲命令。 +本仓库提供 **Agent Skill**（[skills/UniParser-Tools/](./skills/UniParser-Tools/)），让 Cursor、Claude Code 等助手自动完成 PDF / 图片 / 公网 PDF 链接 → Markdown 与版面 JSON 的解析。用户只需要安装 Skill、准备 API Key，然后在对话里提出解析需求；CLI 安装与具体执行步骤由 Agent 按 [SKILL.md](./skills/UniParser-Tools/SKILL.md) 自动完成。 ### 快速使用 Skill -**1. 为 Agent 安装 Skill** +**1. 安装 Skill** -将本仓库中的 `skills/UniParser-Tools/` 整个目录发送给 Agent，并让 Agent 安装该 Skill。 +使用 `skills` 命令安装： -安装后重启 Agent，确保 Skill 列表中出现 **uniparser-tools**。 +```bash +npx skills add dptech-corp/UniParser-Tools +``` + +也可以手动安装：将 [skills/UniParser-Tools/](./skills/UniParser-Tools/) 整个目录发送给 Agent，并让 Agent 安装该 Skill。安装后重启 Agent，确保 Skill 列表中出现 **uniparser-tools**。 -**2. 配置 API Key** +**2. 准备 API Key** -在 [https://uniparser.dp.tech/](https://uniparser.dp.tech/) 注册并申请 API Key，写入环境变量（Agent 终端需能读到）： +在 [https://uniparser.dp.tech/](https://uniparser.dp.tech/) 注册并申请 API Key。你可以让 Agent 按 Skill 指引配置，也可以提前设置环境变量： ```bash export UNIPARSER_API_KEY="your-api-key" ``` +不要把 API Key 直接粘贴到公开对话或代码仓库中。 + **3. 在 Agent 中使用 Skill** 在 Agent 对话中上传文件、粘贴公网 PDF 链接，或使用类似表述即可触发 Skill，例如： @@ -390,41 +396,20 @@ export UNIPARSER_API_KEY="your-api-key" 支持的输入：**本地 PDF**、**本地图片**（png / jpg 等）、**可公网访问的 PDF URL**。 -**4. 使用效果与结果位置** +**4. 查看输出结果** -解析成功后，Agent 会在回复中给出 **Markdown 文件路径**（以及需要时的 **版面结构文件路径**），并可将正文摘要或全文交付给你。典型效果包括： - -- 按阅读顺序输出的 **Markdown 全文**（`{源文件主名}.md`，如 `paper.pdf` → `paper.md`） -- **表格** 转为 Markdown 表格 -- **公式** 转为 LaTeX -- **图片 / 图表** 等以 base64 等形式出现在结果中（视文档类型而定） -- **版面结构树** `pages_tree.json`，便于需要章节、块级布局时使用 -- 面向科技文献的默认识别策略（高质量 OCR 等，由 Skill 配置） - -默认将结果写入用户主目录下： - -`~/Uni-Parser-Skill/<源文件主名>/` - -例如解析 `paper.pdf` 时，默认目录为 `~/Uni-Parser-Skill/paper/`。该目录在**解析成功完成后**才会写入文件，通常包含： +默认输出到 `~/Uni-Parser-Skill/<源文件主名>/`，通常包含： | 文件 | 说明 | |------|------| | `{源文件主名}.md` | 解析得到的完整 Markdown | | `pages_tree.json` | 结构化版面树（页面与语义块层次） | | `formatted_meta.json` | 元数据（不含全文 `content`） | +| `trigger_meta.json` | 任务 token 与解析参数（供 `uniparser fetch` 中断恢复） | -若你在对话中指定了输出目录，Agent 也可将结果保存到你提供的路径。若目标目录已存在，Agent 会先征求你是否覆盖后再继续。大文档解析可能耗时数分钟至十余分钟；重复提交同一文件时 Agent 会按 Skill 说明从已有任务恢复，而不会重复上传。 +Agent 会回复 Markdown 路径，并在需要版面结构时提供 `pages_tree.json`。大文档可能耗时数分钟至十余分钟；中断或重复任务可按 Skill 说明用 `trigger_meta.json` 中的 token 执行 `uniparser fetch`。 -Agent 实现细节、错误恢复与 SDK 安装说明见 `skills/UniParser-Tools/SKILL.md`。 - -### 参考文档 - -- `skills/UniParser-Tools/references/api-reference.md` -- `skills/UniParser-Tools/references/patterns.md` -- `skills/UniParser-Tools/references/data-classes.md` -- `skills/UniParser-Tools/references/layout-types.md` -- `skills/UniParser-Tools/references/utilities.md` -- `skills/UniParser-Tools/references/notes.md` +Agent 实现细节、CLI 命令与错误恢复见 [SKILL.md](./skills/UniParser-Tools/SKILL.md)。 ## MCP Server @@ -489,11 +474,10 @@ uv run python -m uniparser_mcp # 启动 MCP 服务（stdio 模式） ## 项目结构 ``` -uniparser_tools/cli/ # uniparser 命令行工具（见 uniparser_tools/cli/README.md） -├── commands/ # auth, parse, fetch, health, version -└── core/ # 配置、凭证、pipeline、输出 - uniparser_tools/ +├── cli/ # uniparser 命令行工具 +│ ├── commands/ # auth, parse, fetch, health, version +│ └── core/ # 配置、凭证、pipeline、输出 ├── api/ # API 客户端 ├── common/ # 通用常量和数据类 ├── tools/ # 工具模块 diff --git a/skills/UniParser-Tools/SKILL.md b/skills/UniParser-Tools/SKILL.md index e4b2afe..5acd85c 100644 --- a/skills/UniParser-Tools/SKILL.md +++ b/skills/UniParser-Tools/SKILL.md @@ -5,44 +5,61 @@ description: "Parse PDFs, document images, and public PDF URLs into structured M # UniParser-Tools Skill -Parse local PDFs, document images, and public PDF URLs into Markdown and structured layout JSON via [UniParser](https://uniparser.dp.tech/). Agents run the bundled CLI scripts—do not hand-write SDK code for the default workflow. +Parse local PDFs, document images, and public PDF URLs into Markdown and structured layout JSON via [UniParser](https://uniparser.dp.tech/). Agents use the installed `uniparser` CLI—do not hand-write SDK code for the default workflow. ## Installation -**Do not use** `pip install uniparser-tools` — the package is not reliably published on PyPI. - -Install once into the **same Python environment** that runs the scripts below: +Requires **Python 3.11+**. Install into the **same Python environment** that runs `uniparser`: ```bash pip install "git+https://github.com/dptech-corp/UniParser-Tools.git" ``` -`scripts/parse_document.py` and `scripts/fetch_by_token.py` check `import uniparser_tools` at startup; if missing, they exit with `CONFIG_ERROR` and the install command above (no automatic `pip install`). +PyPI is not published yet; use the git install above (no automatic `pip install`—if missing, exit with this command). -Manual verify: +Verify: ```bash -python3 -c "import uniparser_tools; print('ok')" +uniparser --help +uniparser version ``` +If `uniparser` is not found, the package is not installed in the active environment (no automatic `pip install`—exit with install guidance). + ## Configuration -**`UNIPARSER_API_KEY` is mandatory** (HTTP header `X-API-Key`). All requests use `https://uniparser.dp.tech/`. +All requests use `https://uniparser.dp.tech/`. An API key is **mandatory** (HTTP header `X-API-Key`). + +**Key sources** (highest priority first): -Check before parsing: +1. `uniparser --api-key YOUR_KEY …` (one-off) +2. Environment variable `UNIPARSER_API_KEY` +3. `~/.uniparser/config.yaml` (written by `uniparser auth`) + +### Interactive setup (user machines) ```bash -python3 -c "import os, sys; sys.exit(0 if os.getenv('UNIPARSER_API_KEY') else 1)" +uniparser auth +``` + +Apply for a key at [https://uniparser.dp.tech/](https://uniparser.dp.tech/) if needed. Prefer not pasting keys into chat. + +### Agent pre-check + +Before parsing, verify a key is configured: + +```bash +uniparser auth --verify ``` - Exit code **0** → continue. -- Exit code **1** → stop and guide the user to set the key (do not guess). +- Exit code **1** → stop; guide the user to `uniparser auth` or set `UNIPARSER_API_KEY`, then restart the terminal or Cursor. -If the key is missing: +Alternative (env only): -1. Apply at [https://uniparser.dp.tech/](https://uniparser.dp.tech/). -2. Set the environment variable (see below). -3. Restart the terminal or Cursor. +```bash +python3 -c "import os, sys; sys.exit(0 if os.getenv('UNIPARSER_API_KEY') else 1)" +``` **macOS / Linux:** @@ -56,44 +73,43 @@ export UNIPARSER_API_KEY="your-api-key" $env:UNIPARSER_API_KEY="your-api-key" ``` -Prefer environment variables over pasting keys into chat. - ## Usage -> **Working directory**: Run commands from this skill's root directory (folder containing `SKILL.md`). - ### CLI reference -**Pipeline** (fixed in script): `submit → poll get_result until success → fetch pages_tree + Markdown → save`. Default `sync=true` (trigger blocks until server done; script still polls and fetches). `--async` uses `sync=false` on submit only. +**Pipeline:** `submit → poll get_result until success → fetch pages_tree + Markdown → save`. Default `sync=true` (trigger blocks until server done; CLI still polls and fetches). `--async` uses `sync=false` on submit only. -**Input → flag** (use **one** per run): +**Input:** one positional `INPUT` per run—the CLI detects type by path suffix or `http(s)://` URL: -- Local PDF → `--file-path` -- Local image (.png, .jpg, …) → `--image-path` -- Public PDF URL → `--pdf-url` +- Local PDF → `uniparser parse "/path/to/document.pdf"` +- Local image (.png, .jpg, …) → `uniparser parse "/path/to/figure.png"` +- Public PDF URL → `uniparser parse "https://example.com/paper.pdf"` ```bash -python3 scripts/parse_document.py --file-path "/path/to/document.pdf" -python3 scripts/parse_document.py --image-path "/path/to/figure.png" -python3 scripts/parse_document.py --pdf-url "https://example.com/paper.pdf" +uniparser parse "/path/to/document.pdf" +uniparser parse "/path/to/figure.png" +uniparser parse "https://example.com/paper.pdf" ``` Optional flags: ```bash -python3 scripts/parse_document.py --file-path "./paper.pdf" --output-dir "./results" -python3 scripts/parse_document.py --file-path "./paper.pdf" --async -python3 scripts/parse_document.py --file-path "./paper.pdf" --overwrite +uniparser parse "./paper.pdf" -o "./results" +uniparser parse "./paper.pdf" --async +uniparser parse "./paper.pdf" --overwrite ``` Recovery (existing server job—see **Common issues**): ```bash -python3 scripts/fetch_by_token.py --file-path "/path/to/document.pdf" -python3 scripts/fetch_by_token.py --pdf-url "https://example.com/paper.pdf" +uniparser fetch --token "TASK_TOKEN_FROM_PRIOR_RUN" ``` -**Default output** (when `--output-dir` is omitted): `~/Uni-Parser-Skill//` +Token sources: stdout JSON from a prior `uniparser --json parse …`, `trigger_meta.json` under the output directory, or the `token` field in a failed parse stderr JSON. + +**Default output for** `fetch` (when `-o` / `--output-dir` is omitted): `~/Uni-Parser-Skill/token_/`, where `` is the first 8 characters of the token (e.g. `~/Uni-Parser-Skill/token_a1b2c3d4/token_a1b2c3d4.md`). To write into the same directory as a prior `parse`, pass `-o` explicitly (e.g. `-o ~/Uni-Parser-Skill/paper/`). + +**Default output for** `parse` (when `-o` / `--output-dir` is omitted): `~/Uni-Parser-Skill//` `` = local file stem (`paper.pdf` → `paper`); for URLs, the last path segment with only `.pdf`/image suffix removed (`…/2606.05847` → `2606.05847`, not `2606`). @@ -102,45 +118,107 @@ python3 scripts/fetch_by_token.py --pdf-url "https://example.com/paper.pdf" - `pages_tree.json` — structured layout tree - `{stem}.md` — full document Markdown - `formatted_meta.json` — metadata without full `content` +- `trigger_meta.json` — task token and `trigger_kwargs` (for `uniparser fetch` recovery) + +**Deliver to user:** open `{stem}.md` for full text; give the user the path and/or content. Mention `pages_tree.json` when layout structure matters. For reliable path extraction, prefer machine-readable output: + +```bash +uniparser --json parse "/path/to/document.pdf" +``` + +Then read stdout JSON fields `markdown_path`, `pages_tree_path`, `output_dir`, and `token`. + +**Parse options** (CLI flags; defaults match scientific-paper preset): -**Deliver to user:** read stdout JSON (`markdown_path`, `pages_tree_path`, `output_dir`); open `{stem}.md` for full text; give the user the path and/or content. Mention `pages_tree.json` when layout structure matters. -**Parse options** (fixed in script): +| Field | Flag | Default | +| ------------------- | -------------- | ---------- | +| Text | `--textual` | `ocr-hq` | +| Equation | `--equation` | `ocr-hq` | +| Table | `--table` | `ocr-hq` | +| Chart | `--chart` | `base64` | +| Figure | `--figure` | `base64` | +| Chemical expression | `--expression` | `base64` | +| Molecule | `--molecule` | `ocr-fast` | -- `textual`, `equation`, `table`: OCRHighQuality -- `chart`, `figure`, `expression`: DumpBase64 -- `molecule`: OCRFast -- `sync=true` by default (`--async` for `sync=false`) + +Choices: `disable`, `ocr-fast`, `ocr-hq`, `digital` (textual only), `base64`. `sync=true` by default (`--async` for `sync=false`). + +Example: + +```bash +uniparser parse paper.pdf --textual digital --molecule disable +``` + +## I/O contract + + +| Outcome | Exit code | stdout | stderr | +| -------------------- | --------- | ------------------------------- | ------------------------------------------------------------- | +| Success (human mode) | 0 | Human-readable paths | Progress (`Parsing... filename`) | +| Success (`--json`) | 0 | Single JSON object (`ok: true`) | Progress (same as above) | +| Failure | 1 | (empty or unused) | Single JSON line (`ok: false`, `error.code`, `error.message`) | + + +`--json` must appear **before** the subcommand: + +```bash +uniparser --json parse paper.pdf # correct +uniparser parse paper.pdf --json # wrong +``` + +`parse` **success JSON fields** (`uniparser --json parse INPUT`): + + +| Field | Meaning | +| ------------------- | ---------------------------------------- | +| `ok` | `true` on success | +| `output_dir` | Result directory (absolute path) | +| `markdown_path` | Main Markdown file | +| `pages_tree_path` | Layout tree JSON | +| `content_chars` | Markdown body length | +| `token` | Task token for `uniparser fetch --token` | +| `input_type` | `file` / `image` / `url` | +| `trigger_meta_path` | Path to `trigger_meta.json` | + + +**Common error codes** (stderr JSON): `CONFIG_ERROR`, `INPUT_ERROR`, `DIR_EXISTS`, `PARSE_ERROR`. ## Common issues On failure, show stderr JSON `error.message`. Do not substitute vision-only reading for UniParser output. -| Problem | Cause | Solution | -|---------|-------|----------| -| `CONFIG_ERROR` | Missing `UNIPARSER_API_KEY` or `uniparser_tools` not installed | **Configuration** + `pip install "git+https://github.com/dptech-corp/UniParser-Tools.git"`; re-check key one-liner | -| `DIR_EXISTS` | Output directory already exists | Ask user; re-run with `--overwrite` if they agree | -| `Token is duplicated` | Job for this API key + exact input already exists | Do **not** re-run `parse_document.py`. `python3 scripts/fetch_by_token.py` with the **same** flag and path/URL (e.g. `--file-path "/abs/path/paper.pdf"`) | -| Job not done / long wait / CLI interrupted / `processing` / poll timeout | Sync or poll still running; or local process stopped while server job continues | Wait; do **not** start a second `parse_document.py` for the same input. Re-run the same `fetch_by_token.py` command; files appear only after exit 0 | -| `502 Bad Gateway` on `--pdf-url` | Server failed fetching or processing remote PDF | Retry `--pdf-url` once; or download and `--file-path`; or `fetch_by_token.py --pdf-url "exact same url"` if a prior job exists | -| Wrong input flag | Images require `trigger_snip` | Use `--image-path`, not `--file-path` | -| `PARSE_ERROR` | Server `status: error` at trigger / poll / fetch | Read `error.message` and `stage`; match rows above; check `trigger_error.json` / `pages_tree_error.json` / `formatted_error.json` under output dir if present | -**Limits:** large PDFs may take 10–20+ minutes; public service ≤5 concurrent requests ([references/notes.md](./references/notes.md)); `--pdf-url` must be publicly accessible. Recovery via `fetch_by_token.py` must use the **same** input string as the original parse—URL and file path are not interchangeable. +| Problem | Cause | Solution | +| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `CONFIG_ERROR` | No API key or `uniparser` not installed | **Configuration** + `pip install "git+https://github.com/dptech-corp/UniParser-Tools.git"`; `uniparser auth --verify` | +| `DIR_EXISTS` | Output directory already exists | Ask user; re-run with `--overwrite` if they agree | +| `Token is duplicated` | Job for this API key + exact input already exists | Do **not** re-run `uniparser parse`. Read `token` from stderr JSON or `trigger_meta.json`; run `uniparser fetch --token TOKEN` | +| Job not done / long wait / CLI interrupted / `processing` / poll timeout | Sync or poll still running; or local process stopped while server job continues | Wait; do **not** start a second `uniparser parse` for the same input. Use saved `token` with `uniparser fetch --token TOKEN`; files appear only after exit 0 | +| `502 Bad Gateway` on URL input | Server failed fetching or processing remote PDF | Retry `uniparser parse "same url"` once; or download and `uniparser parse local.pdf`; or `uniparser fetch --token TOKEN` if a prior job exists | +| `PARSE_ERROR` | Server `status: error` at trigger / poll / fetch | Read `error.message` and `stage`; match rows above; check `trigger_error.json` / `pages_tree_error.json` / `formatted_error.json` under output dir if present | + + +**Limits:** large PDFs may take 10–20+ minutes; public service ≤5 concurrent requests ([Important notes](./references/notes.md)); PDF URLs must be publicly accessible. Save `token` from success JSON or `trigger_meta.json` for recovery after interrupt or duplicate-token errors. ## Advanced -For callbacks, custom `ParseMode`, or SDK examples, see [references/patterns.md](./references/patterns.md) and [references/api-reference.md](./references/api-reference.md). +For callbacks, custom `ParseMode`, or SDK examples, see [Common patterns](./references/patterns.md) and [API reference](./references/api-reference.md). + +Full CLI reference (flags, examples, JSON details): [CLI README](../../uniparser_tools/cli/README.md) in this repository. Optional MCP server setup is in the [UniParser-Tools GitHub repo](https://github.com/dptech-corp/UniParser-Tools); it is separate from this CLI workflow. ## Reference documents -| Topic | File | -|-------|------| -| API reference | [references/api-reference.md](./references/api-reference.md) | -| Common patterns | [references/patterns.md](./references/patterns.md) | -| Data classes | [references/data-classes.md](./references/data-classes.md) | -| Layout types | [references/layout-types.md](./references/layout-types.md) | -| Utilities | [references/utilities.md](./references/utilities.md) | -| Important notes | [references/notes.md](./references/notes.md) | + +| Topic | File | +| --------------- | ------------------------------------------------- | +| API reference | [api-reference.md](./references/api-reference.md) | +| Common patterns | [patterns.md](./references/patterns.md) | +| Data classes | [data-classes.md](./references/data-classes.md) | +| Layout types | [layout-types.md](./references/layout-types.md) | +| Utilities | [utilities.md](./references/utilities.md) | +| Important notes | [notes.md](./references/notes.md) | + + diff --git a/skills/UniParser-Tools/scripts/fetch_by_token.py b/skills/UniParser-Tools/scripts/fetch_by_token.py deleted file mode 100644 index b277220..0000000 --- a/skills/UniParser-Tools/scripts/fetch_by_token.py +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env python3 -""" -Fetch pages_tree + Markdown for an existing UniParser job (poll until success). - -Resolves the task token from --token or by computing to_token from the same input -used at submit time (--file-path / --image-path / --pdf-url). - -Usage: - python3 scripts/fetch_by_token.py --file-path "/path/to/document.pdf" - python3 scripts/fetch_by_token.py --pdf-url "https://example.com/paper.pdf" - python3 scripts/fetch_by_token.py --image-path "/path/to/figure.png" - python3 scripts/fetch_by_token.py --file-path paper.pdf --output-dir ./out --overwrite -""" - -from __future__ import annotations - -import argparse -import os -import sys -from pathlib import Path - - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -from lib_common import ( # noqa: E402 - DEFAULT_HOST, - fetch_markdown, - fetch_pages_tree, - parse_error, - poll_until_success, - print_success, - resolve_fetch_target, - resolve_output_dir, - run_startup_checks, - save_parse_results, -) - - -def main() -> int: - if (code := run_startup_checks()) is not None: - return code - - from uniparser_tools.api.clients import UniParserClient - - parser = argparse.ArgumentParser(description="Fetch UniParser results for an existing job (by input or token)") - input_group = parser.add_mutually_exclusive_group(required=True) - input_group.add_argument("--token", help="UniParser task token (optional; prefer input flags)") - input_group.add_argument("--file-path", help="Same local PDF path used in parse_document.py") - input_group.add_argument("--image-path", help="Same local image path used in parse_document.py") - input_group.add_argument("--pdf-url", help="Same public PDF URL used in parse_document.py") - parser.add_argument( - "--output-dir", - "-o", - help="Output directory (default: ~/Uni-Parser-Skill//)", - ) - parser.add_argument( - "--overwrite", - action="store_true", - help="Overwrite output directory if it already exists", - ) - args = parser.parse_args() - - client = UniParserClient(host=DEFAULT_HOST, api_key=os.environ["UNIPARSER_API_KEY"]) - - resolved = resolve_fetch_target( - token=args.token, - file_path=args.file_path, - image_path=args.image_path, - pdf_url=args.pdf_url, - client=client, - ) - if isinstance(resolved, int): - return resolved - token, source_stem = resolved - - out_dir, dir_code = resolve_output_dir(source_stem, args.output_dir, overwrite=args.overwrite) - if dir_code is not None: - return dir_code - - poll_result = poll_until_success(client, token) - if isinstance(poll_result, int): - return poll_result - - pages_tree = fetch_pages_tree(client, token) - if pages_tree.get("status") != "success": - return parse_error("get_result_pages_tree", pages_tree) - - formatted = fetch_markdown(client, token) - if formatted.get("status") != "success": - return parse_error("get_formatted", formatted) - - summary = save_parse_results( - out_dir=out_dir, - source_stem=source_stem, - pages_tree=pages_tree, - formatted=formatted, - ) - summary["fetched_by_token"] = True - print_success(summary) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/skills/UniParser-Tools/scripts/lib_common.py b/skills/UniParser-Tools/scripts/lib_common.py deleted file mode 100644 index 0924d02..0000000 --- a/skills/UniParser-Tools/scripts/lib_common.py +++ /dev/null @@ -1,275 +0,0 @@ -"""Shared helpers for UniParser skill CLI scripts.""" - -from __future__ import annotations - -import json -import os -import shutil -import sys -import time -from pathlib import Path -from typing import Any -from urllib.parse import urlparse - - -DEFAULT_HOST = "https://uniparser.dp.tech/" -INSTALL_CMD = "git+https://github.com/dptech-corp/UniParser-Tools.git" - -# Hard-coded polling (not documented in SKILL.md as tunable). -POLL_INTERVAL_SEC = 3 -POLL_TIMEOUT_SEC = 1800 - -PENDING_STATUSES = frozenset({"undefined", "waiting", "processing"}) - - -def emit_json_stderr(payload: dict) -> None: - print(json.dumps(payload, ensure_ascii=False), file=sys.stderr) - - -def config_error(message: str) -> int: - emit_json_stderr({"ok": False, "error": {"code": "CONFIG_ERROR", "message": message}}) - return 1 - - -def dir_exists_error(output_dir: Path) -> int: - emit_json_stderr( - { - "ok": False, - "error": { - "code": "DIR_EXISTS", - "message": ( - f"Output directory already exists: {output_dir}. " - "Ask the user whether to continue parsing. " - "If they agree, re-run the same command with --overwrite." - ), - "output_dir": str(output_dir), - }, - } - ) - return 1 - - -def parse_error(stage: str, result: dict) -> int: - emit_json_stderr( - { - "ok": False, - "error": { - "code": "PARSE_ERROR", - "message": result.get("description") or result.get("message") or str(result), - "stage": stage, - }, - "token": result.get("token"), - } - ) - return 1 - - -def get_api_key() -> str | None: - return (os.getenv("UNIPARSER_API_KEY") or "").strip() or None - - -def check_api_key() -> int | None: - if get_api_key(): - return None - return config_error( - "UNIPARSER_API_KEY is not set. " - "Register at https://uniparser.dp.tech/ then set the environment variable. " - "See SKILL.md Configuration section for platform-specific commands." - ) - - -def check_uniparser_installed() -> int | None: - try: - import uniparser_tools # noqa: F401 - except ImportError: - return config_error(f'uniparser_tools is not installed. Run once: pip install "{INSTALL_CMD}"') - return None - - -def run_startup_checks() -> int | None: - """API key + package presence. Returns exit code if checks fail.""" - code = check_api_key() - if code is not None: - return code - return check_uniparser_installed() - - -def source_stem_from_path(path: Path) -> str: - return path.stem or "document" - - -def source_stem_from_url(url: str) -> str: - """Last URL path segment; strip only known doc extensions (not arXiv ID dots).""" - segment = urlparse(url).path.rstrip("/").rsplit("/", 1)[-1] - if not segment: - return "url_document" - lower = segment.lower() - for ext in (".pdf", ".png", ".jpg", ".jpeg", ".webp"): - if lower.endswith(ext): - segment = segment[: -len(ext)] - break - return segment or "url_document" - - -def default_output_dir(source_stem: str) -> Path: - return (Path.home() / "Uni-Parser-Skill" / source_stem).expanduser().resolve() - - -def resolve_fetch_target( - *, - token: str | None, - file_path: str | None, - image_path: str | None, - pdf_url: str | None, - client, -) -> tuple[str, str] | int: - """Resolve task token and source_stem for fetch_by_token. Returns exit code on error.""" - provided = sum(1 for value in (token, file_path, image_path, pdf_url) if value is not None) - if provided != 1: - return config_error("Provide exactly one of --token, --file-path, --image-path, or --pdf-url.") - - if token is not None: - resolved = token.strip() - if not resolved: - return config_error("Token must not be empty.") - return resolved, f"token_{resolved[:8]}" - - if file_path is not None: - path = Path(file_path).expanduser().resolve() - if not path.is_file(): - return config_error(f"File not found: {path}") - task_id = str(path) - return client.to_token(task_id), source_stem_from_path(path) - - if image_path is not None: - path = Path(image_path).expanduser().resolve() - if not path.is_file(): - return config_error(f"Image not found: {path}") - task_id = str(path) - return client.to_token(task_id), source_stem_from_path(path) - - task_id = pdf_url.strip() - if not task_id: - return config_error("PDF URL must not be empty.") - return client.to_token(task_id), source_stem_from_url(task_id) - - -def resolve_output_dir( - source_stem: str, - output_dir: str | None, - *, - overwrite: bool, -) -> tuple[Path | None, int | None]: - out = Path(output_dir).expanduser().resolve() if output_dir else default_output_dir(source_stem) - if out.exists() and not overwrite: - return None, dir_exists_error(out) - if out.exists() and overwrite: - shutil.rmtree(out) - return out, None - - -def scientific_paper_trigger_kwargs(*, sync: bool = True) -> dict: - from uniparser_tools.common.constant import ParseMode, ParseModeTextual - - return { - "sync": sync, - "textual": ParseModeTextual.OCRHighQuality, - "equation": ParseMode.OCRHighQuality, - "table": ParseMode.OCRHighQuality, - "chart": ParseMode.DumpBase64, - "figure": ParseMode.DumpBase64, - "expression": ParseMode.DumpBase64, - "molecule": ParseMode.OCRFast, - } - - -def poll_until_success(client, token: str) -> dict | int: - """Poll get_result until status is success. Returns result dict or exit code.""" - deadline = time.time() + POLL_TIMEOUT_SEC - last: dict[str, Any] = {} - - while time.time() < deadline: - last = client.get_result( - token, - content=False, - objects=False, - pages_dict=False, - pages_tree=False, - ) - status = last.get("status") - if status == "success": - return last - if status == "error": - return parse_error("get_result_poll", last) - if status in PENDING_STATUSES or status is None: - time.sleep(POLL_INTERVAL_SEC) - continue - return parse_error("get_result_poll", last) - - return parse_error( - "get_result_poll", - { - "status": "error", - "description": f"Timed out after {POLL_TIMEOUT_SEC}s waiting for parsing to finish.", - "token": token, - "last_status": last.get("status"), - }, - ) - - -def fetch_pages_tree(client, token: str) -> dict: - return client.get_result(token, pages_tree=True, objects=False) - - -def fetch_markdown(client, token: str) -> dict: - from uniparser_tools.common.constant import FormatFlag - - return client.get_formatted( - token, - content=True, - textual=FormatFlag.Markdown, - table=FormatFlag.Markdown, - equation=FormatFlag.Latex, - ) - - -def save_parse_results( - *, - out_dir: Path, - source_stem: str, - pages_tree: dict, - formatted: dict, -) -> dict[str, Any]: - out_dir.mkdir(parents=True, exist_ok=True) - stem = source_stem or "document" - - pages_tree_path = out_dir / "pages_tree.json" - pages_tree_path.write_text( - json.dumps(pages_tree, indent=2, ensure_ascii=False, default=str), - encoding="utf-8", - ) - - md_path = out_dir / f"{stem}.md" - content = formatted.get("content", "") - md_path.write_text(content, encoding="utf-8") - - meta = {k: v for k, v in formatted.items() if k != "content"} - (out_dir / "formatted_meta.json").write_text( - json.dumps(meta, indent=2, ensure_ascii=False, default=str), - encoding="utf-8", - ) - - return { - "ok": True, - "output_dir": str(out_dir), - "pages_tree_path": str(pages_tree_path), - "markdown_path": str(md_path), - "content_chars": len(content), - } - - -def print_success(summary: dict) -> None: - print(json.dumps(summary, ensure_ascii=False, indent=2)) - print(f"Pages tree saved to: {summary['pages_tree_path']}", file=sys.stderr) - print(f"Markdown saved to: {summary['markdown_path']}", file=sys.stderr) - print(f"Output directory: {summary['output_dir']}", file=sys.stderr) diff --git a/skills/UniParser-Tools/scripts/parse_document.py b/skills/UniParser-Tools/scripts/parse_document.py deleted file mode 100644 index 5048dbf..0000000 --- a/skills/UniParser-Tools/scripts/parse_document.py +++ /dev/null @@ -1,147 +0,0 @@ -#!/usr/bin/env python3 -""" -Parse a local PDF, local image, or public PDF URL with UniParser-Tools. - -Flow: submit → token → poll get_result until success → save pages_tree + Markdown. - -Usage: - python3 scripts/parse_document.py --file-path document.pdf - python3 scripts/parse_document.py --pdf-url "https://example.com/paper.pdf" - python3 scripts/parse_document.py --file-path paper.pdf --async - python3 scripts/parse_document.py --file-path paper.pdf --overwrite -""" - -from __future__ import annotations - -import argparse -import json -import os -import sys -from pathlib import Path - - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -from lib_common import ( # noqa: E402 - DEFAULT_HOST, - config_error, - fetch_markdown, - fetch_pages_tree, - parse_error, - poll_until_success, - print_success, - resolve_output_dir, - run_startup_checks, - save_parse_results, - scientific_paper_trigger_kwargs, - source_stem_from_path, - source_stem_from_url, -) - - -def main() -> int: - if (code := run_startup_checks()) is not None: - return code - - from uniparser_tools.api.clients import UniParserClient - - parser = argparse.ArgumentParser(description="Parse documents with UniParser-Tools (https://uniparser.dp.tech/)") - input_group = parser.add_mutually_exclusive_group(required=True) - input_group.add_argument("--file-path", help="Local PDF file path") - input_group.add_argument("--image-path", help="Local image path (snippet)") - input_group.add_argument("--pdf-url", help="Public PDF URL") - parser.add_argument( - "--output-dir", - "-o", - help="Output directory (default: ~/Uni-Parser-Skill//)", - ) - parser.add_argument( - "--async", - dest="async_mode", - action="store_true", - help="Submit with sync=false and poll get_result until success (default: sync=true)", - ) - parser.add_argument( - "--overwrite", - action="store_true", - help="Overwrite output directory if it already exists (use after user confirms)", - ) - args = parser.parse_args() - - if args.file_path: - path = Path(args.file_path).expanduser().resolve() - if not path.is_file(): - return config_error(f"File not found: {path}") - source_stem = source_stem_from_path(path) - input_type = "file" - elif args.image_path: - path = Path(args.image_path).expanduser().resolve() - if not path.is_file(): - return config_error(f"Image not found: {path}") - source_stem = source_stem_from_path(path) - input_type = "image" - else: - source_stem = source_stem_from_url(args.pdf_url) - input_type = "url" - - out_dir, dir_code = resolve_output_dir(source_stem, args.output_dir, overwrite=args.overwrite) - if dir_code is not None: - return dir_code - - client = UniParserClient(host=DEFAULT_HOST, api_key=os.environ["UNIPARSER_API_KEY"]) - trigger_kwargs = scientific_paper_trigger_kwargs(sync=not args.async_mode) - - if args.file_path: - trigger = client.trigger_file(file_path=str(path), **trigger_kwargs) - stage = "trigger_file" - elif args.image_path: - trigger = client.trigger_snip(snip_path=str(path), **trigger_kwargs) - stage = "trigger_snip" - else: - trigger = client.trigger_url(pdf_url=args.pdf_url, **trigger_kwargs) - stage = "trigger_url" - - if trigger.get("status") != "success": - out_dir.mkdir(parents=True, exist_ok=True) - (out_dir / "trigger_error.json").write_text( - json.dumps(trigger, indent=2, ensure_ascii=False), - encoding="utf-8", - ) - return parse_error(stage, trigger) - - token = trigger["token"] - poll_result = poll_until_success(client, token) - if isinstance(poll_result, int): - out_dir.mkdir(parents=True, exist_ok=True) - return poll_result - - pages_tree = fetch_pages_tree(client, token) - if pages_tree.get("status") != "success": - out_dir.mkdir(parents=True, exist_ok=True) - (out_dir / "pages_tree_error.json").write_text( - json.dumps(pages_tree, indent=2, ensure_ascii=False), - encoding="utf-8", - ) - return parse_error("get_result_pages_tree", pages_tree) - - formatted = fetch_markdown(client, token) - if formatted.get("status") != "success": - out_dir.mkdir(parents=True, exist_ok=True) - (out_dir / "formatted_error.json").write_text( - json.dumps(formatted, indent=2, ensure_ascii=False), - encoding="utf-8", - ) - return parse_error("get_formatted", formatted) - - summary = save_parse_results( - out_dir=out_dir, - source_stem=source_stem, - pages_tree=pages_tree, - formatted=formatted, - ) - summary["input_type"] = input_type - print_success(summary) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tests/unit/test_skill_cli.py b/tests/unit/test_skill_cli.py deleted file mode 100644 index eac13d8..0000000 --- a/tests/unit/test_skill_cli.py +++ /dev/null @@ -1,45 +0,0 @@ -"""Smoke tests for skills/UniParser-Tools CLI scripts (no network).""" - -from __future__ import annotations - -import json -import os -import subprocess -import sys -from pathlib import Path - - -SKILL_ROOT = Path(__file__).resolve().parents[2] / "skills" / "UniParser-Tools" -PARSE_SCRIPT = SKILL_ROOT / "scripts" / "parse_document.py" -FETCH_SCRIPT = SKILL_ROOT / "scripts" / "fetch_by_token.py" - - -def _env_without_api_key() -> dict[str, str]: - return {k: v for k, v in os.environ.items() if k != "UNIPARSER_API_KEY"} - - -def _run_script(script: Path, *args: str) -> subprocess.CompletedProcess[str]: - return subprocess.run( - [sys.executable, str(script), *args], - capture_output=True, - text=True, - env=_env_without_api_key(), - cwd=SKILL_ROOT, - check=False, - ) - - -class TestSkillCliConfigError: - def test_parse_document_without_api_key(self) -> None: - result = _run_script(PARSE_SCRIPT, "--file-path", "/tmp/nonexistent.pdf") - assert result.returncode == 1 - payload = json.loads(result.stderr.strip()) - assert payload["ok"] is False - assert payload["error"]["code"] == "CONFIG_ERROR" - - def test_fetch_by_token_without_api_key(self) -> None: - result = _run_script(FETCH_SCRIPT, "--token", "dummy-token") - assert result.returncode == 1 - payload = json.loads(result.stderr.strip()) - assert payload["ok"] is False - assert payload["error"]["code"] == "CONFIG_ERROR" diff --git a/uniparser_tools/cli/README.md b/uniparser_tools/cli/README.md index dc7d1de..51b0201 100644 --- a/uniparser_tools/cli/README.md +++ b/uniparser_tools/cli/README.md @@ -8,15 +8,21 @@ ## 安装 -在仓库根目录执行（**必须**用这一条，才会安装 `uniparser` 命令）： +需要 **Python 3.11+**。在运行 `uniparser` 的同一 Python 环境中执行： + +```bash +pip install "git+https://github.com/dptech-corp/UniParser-Tools.git" +``` + +> PyPI 尚未发布，请使用上述 git 安装。只运行 `pip install -r requirements.txt` **不会**出现 `uniparser` 命令。 + +在本仓库内开发时，可在仓库根目录执行： ```bash cd /path/to/UniParser-Tools pip install -e . ``` -> 只运行 `pip install -r requirements.txt` **不会**出现 `uniparser` 命令。 - 检查是否安装成功： ```bash @@ -444,7 +450,7 @@ uniparser --json parse paper.pdf 2>/dev/null | python3 -c "import sys,json; prin **命令 `uniparser` 找不到** -确认已执行 `pip install -e .`，且与当前终端使用的是同一 Python 环境。 +确认已执行 `pip install "git+https://github.com/dptech-corp/UniParser-Tools.git"`（或在本仓库内 `pip install -e .`），且与当前终端使用的是同一 Python 环境。 **解析时间较长** @@ -456,7 +462,7 @@ uniparser --json parse paper.pdf 2>/dev/null | python3 -c "import sys,json; prin ```bash # 1. 安装并配置 -pip install -e . +pip install "git+https://github.com/dptech-corp/UniParser-Tools.git" uniparser auth # 2. 解析