AI Cost Tracking

• 🤖 LLM usage: $6.5296 (172 commits)
• 👤 Human dev: ~$7134 (71.3h @ $100/h, 30min dedup)

Generated on 2026-07-06 using openrouter/qwen/qwen3-coder-next

---

curllm = curl + LLM

Intelligent Browser Automation with Local LLMs

Quick Start • Features • Examples • Documentation • API

---

🎯 What is curllm?

curllm is a powerful CLI tool that combines browser automation with local LLMs (like Ollama's Qwen, Llama, Mistral) to intelligently extract data, fill forms, and automate web workflows - all running locally on your machine with complete privacy.

🆕 v2 LLM-DSL Architecture! Dynamic element detection, semantic goal understanding, no hardcoded selectors. 388 tests passing.

# Extract products with prices from any e-commerce site
curllm "https://shop.example.com" -d "Find all products under $100"

# Fill contact forms automatically
curllm --stealth "https://example.com/contact" -d "Fill form: name=John, email=john@example.com"

# Extract all emails from a page
curllm "https://example.com" -d "extract all email addresses"

✨ Features

Feature	Description
🧠 Local LLM	Works with 8GB GPUs (Qwen 2.5, Llama 3, Mistral)
🎯 Smart Extraction	LLM-guided DOM analysis - no hardcoded selectors
📝 Form Automation	Auto-fill forms with intelligent field mapping
🥷 Stealth Mode	Bypass anti-bot detection
👁️ Visual Mode	See browser actions in real-time
🔍 BQL Support	Browser Query Language for structured queries
📊 Export Formats	JSON, CSV, HTML, XLS output
🔒 Privacy-First	Everything runs locally - no cloud APIs needed

🧠 LLM-DSL Architecture

curllm v2 uses LLM-DSL (LLM Domain Specific Language) - a dynamic approach that eliminates hardcoded selectors:

┌─────────────────────────────────────────────────────────────┐
│                     LLM-DSL Flow                            │
├─────────────────────────────────────────────────────────────┤
│  1. Goal Detection (semantic)                               │
│     "Find RAM DDR5" → FIND_PRODUCTS                         │
│                                                             │
│  2. Strategy Selection                                      │
│     FIND_PRODUCTS → use search flow                         │
│     FIND_CART → find link by semantic scoring               │
│                                                             │
│  3. Element Finding (LLM-first)                             │
│     LLM analysis → Statistical scoring → Fallback           │
│                                                             │
│  4. Dynamic Selector Generation                             │
│     Analyze DOM → Score elements → Generate selector        │
└─────────────────────────────────────────────────────────────┘

Key Benefits

Feature	Traditional	LLM-DSL
Selectors	Hardcoded CSS/XPath	Dynamic generation
Keywords	Static lists	Semantic analysis
Language	English only	Multi-language (PL, EN)
Maintenance	Manual updates	Self-adapting

🚀 Quick Start

Installation

pip install -U curllm
curllm-setup      # One-time setup (installs Playwright browsers)
curllm-doctor     # Verify installation

Requirements

• Python 3.10+
• GPU: NVIDIA with 6-8GB VRAM (RTX 3060/4060) or CPU mode
• Ollama: For local LLM inference

# Install Ollama (if not installed)
curl -fsSL https://ollama.ai/install.sh | sh
ollama pull qwen2.5:7b

📖 Examples

Continuous verification and autonomy (Linux user service)

Use an environment with the declared MCP SDK 1.x, pytest, Playwright Chromium and Planfile installed. Stage a version already merged into origin/main, then activate its user timer explicitly:

python scripts/install-autonomy.py --python /absolute/path/to/venv/bin/python --activate
systemctl --user status curllm-autonomy.timer
journalctl --user -u curllm-autonomy.service -n 20 --no-pager

The installer exports the merged revision into ~/.local/share/curllm-autonomy/releases/<sha>, preserving uncommitted primary files. Each cycle checks source digests, then runs real Chromium field/DSL, MCP transport/v2 and monitoring regressions. The next cycle starts five minutes after completion. Source checksums are checked before Python starts. Omit --activate to stage without changing running services. Existing differing operator files are retained; reconcile version/configuration changes before reinstalling. To stop: systemctl --user disable --now curllm-autonomy.timer. Disabling the timer lets an already running bounded cycle finish; incident state, development intake and versioned releases remain available.

Configuration lives in ~/.config/curllm-autonomy/config.json. Status is available through the deployed interpreter, from the release directory:

python -m curllm_core.cli.autonomy status --config ~/.config/curllm-autonomy/config.json

The private state directory ~/.local/state/curllm-autonomy/ contains incidents.sqlite, structured DSL events in events.jsonl (one retained rotated file), and a dedicated development-intake/ Planfile queue. Two consecutive failures open an incident; two successful observations resolve it. Diagnostics retain output hashes, sizes, exit codes and timing, without raw output. For a JSON probe use "oracle": "json"; it requires explicit success: true without errors or a malformed/failing status, in addition to a zero exit code.

Automatic runtime repairs must be explicitly configured on a probe:

"repair": {
  "argv": ["/usr/bin/systemctl", "--user", "restart", "your-owned-service.service"],
  "authority_ref": "operator:your-owned-service-restart",
  "timeout_seconds": 30,
  "max_attempts": 1,
  "cooldown_seconds": 300
}

No repairs are inferred from test output. Attempts are reserved before execution, limited across recurring incidents for that probe configuration, and confirmed by rerunning the probe. An interrupted effect remains unknown for operator reconciliation and is never automatically replayed. A command's zero exit code alone never confirms repair. The default installation only runs tests and intake.

Development findings are deduplicated Planfile tickets with state awaiting_controller. The observer does not edit source or approve/merge PRs. Autonomous coding needs a separately admitted repository profile, canonical ticket/worktree, protected fenced lease and independent Validator. Monitor recovery does not close a development ticket as delivered. A missing intake dependency leaves a durable outbox_pending for the next cycle.

Extract Data

# Extract all links
curllm "https://example.com" -d "extract all links"

# Extract emails
curllm "https://example.com/contact" -d "extract all email addresses"
# Output: {"emails": ["info@example.com", "sales@example.com"]}

# Extract products with price filter
curllm --stealth "https://shop.example.com" -d "Find all products under 500zł"

Form Automation

# Fill contact form
curllm --visual --stealth "https://example.com/contact" \
  -d "Fill form: name=John Doe, email=john@example.com, message=Hello"

# Login automation
curllm --visual "https://app.example.com/login" \
  -d '{"instruction":"Login", "credentials":{"user":"admin", "pass":"secret"}}'

Export Results

# Export to CSV
curllm "https://example.com" -d "extract all products" --csv -o products.csv

# Export to HTML
curllm "https://example.com" -d "extract all links" --html -o links.html

# Export to Excel
curllm "https://example.com" -d "extract all data" --xls -o data.xlsx

Screenshots

# Take screenshot
curllm "https://example.com" -d "screenshot"

# Visual mode (watch browser)
curllm --visual "https://example.com" -d "extract all links"

BQL Queries

curllm --bql -d 'query {
  page(url: "https://news.ycombinator.com") {
    title
    links: select(css: "a.titlelink") { text url: attr(name: "href") }
  }
}'

🌐 Web Interface

curllm-web start   # Start web UI at http://localhost:5000
curllm-web status  # Check status
curllm-web stop    # Stop server

Features:

• 🎨 Modern responsive UI
• 📝 19 pre-configured prompts
• 📊 Real-time log viewer
• 📤 File upload support

🔧 Configuration

Environment variables (.env):

CURLLM_MODEL=qwen2.5:7b          # LLM model
CURLLM_OLLAMA_HOST=http://localhost:11434
CURLLM_HEADLESS=true             # Run browser headlessly
CURLLM_STEALTH_MODE=false        # Anti-detection
CURLLM_LOCALE=en-US              # Browser locale

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         curllm CLI                              │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌────────────────┐    ┌────────────────┐    ┌───────────────┐  │
│  │  DSL Executor  │───▶│ Knowledge Base │───▶│ Strategy YAML │  │
│  │  (Orchestrator)│    │   (SQLite)     │    │    Files      │  │
│  └────────────────┘    └────────────────┘    └───────────────┘  │
│          │                                                      │
│          ▼                                                      │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │                    DOM Toolkit (Pure JS)                   │ │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────────┐  │ │
│  │  │Structure │  │ Patterns │  │Selectors │  │   Prices   │  │ │
│  │  │ Analyzer │  │ Detector │  │Generator │  │  Detector  │  │ │
│  │  └──────────┘  └──────────┘  └──────────┘  └────────────┘  │ │
│  └────────────────────────────────────────────────────────────┘ │
│          │                                                      │
│          ▼                                                      │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │              Playwright Browser Engine                     │ │
│  │         (Chromium with Stealth & Anti-Detection)           │ │
│  └────────────────────────────────────────────────────────────┘ │
│          │                                                      │
│          ▼                                                      │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │                 Ollama / LiteLLM                           │ │
│  │      (Local LLM: Qwen 2.5, Llama 3, Mistral, GPT, etc)     │ │
│  └────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Key Components

Component	Description	LLM Calls
URL Resolver	Smart navigation with goal detection	0-1
Goal Detector	Semantic intent understanding	0-1
Element Finder	Dynamic selector generation	0-1
DOM Toolkit	Pure JavaScript atomic queries	0
SPA Hydration	Wait for CSR/SPA content	0

📖 Full Architecture Documentation →

🧬 DSL System (Strategy-Based Extraction)

Note: The YAML DSL system works alongside the newer LLM-DSL. YAML strategies are used for known sites with proven extraction patterns, while LLM-DSL handles unknown sites dynamically.

curllm automatically learns and saves successful extraction strategies as YAML files:

# dsl/ceneo_products.yaml - Auto-generated from successful extraction
url_pattern: "*.ceneo.pl/*"
task: extract_products
algorithm: statistical_containers

selector: div.product-card
fields:
  name: h3.title
  price: span.price
  url: a[href]

metadata:
  success_rate: 0.95
  use_count: 42

How It Works

1. First visit - LLM-DSL dynamically analyzes page, extracts data
2. Successful - Strategy saved to dsl/*.yaml, recorded in Knowledge Base
3. Next visit - Knowledge Base loads saved strategy (fast path)
4. Unknown site - Falls back to LLM-DSL dynamic discovery

┌─────────────────────────────────────────────────────────┐
│                   Request Flow                          │
├─────────────────────────────────────────────────────────┤
│  URL → Knowledge Base lookup                            │
│        │                                                │
│        ├─ Found? → Load YAML strategy (fast)            │
│        │                                                │
│        └─ Not found? → LLM-DSL dynamic (flexible)       │
│                        │                                │
│                        └─ Success? → Save to YAML       │
└─────────────────────────────────────────────────────────┘

Algorithms

Algorithm	Best For	Speed
statistical_containers	Product grids	⚡ Fast
pattern_detection	Lists, tables	⚡ Fast
llm_guided	Complex layouts	🐢 Slower
form_fill	Contact forms	⚡ Fast

📖 DSL System Documentation →

🤝 Multi-Provider LLM Support

curllm supports multiple LLM providers via LiteLLM:

from curllm_core import LLMConfig

# OpenAI
config = LLMConfig(provider="openai/gpt-4o-mini")

# Anthropic
config = LLMConfig(provider="anthropic/claude-3-haiku-20240307")

# Google Gemini
config = LLMConfig(provider="gemini/gemini-2.0-flash")

# Local Ollama (default)
config = LLMConfig(provider="ollama/qwen2.5:7b")

📚 Documentation

Getting Started

• 📖 Full Documentation
• ⚙️ Installation Guide
• 📝 Examples & Tutorials

Architecture

• 🏗️ System Architecture
• 🧬 DSL System - Strategy-based extraction
• ⚛️ DOM Toolkit - Pure JS queries
• 🧩 Components - Module overview
• 🔗 LLM-DSL URL Resolution - Smart URL navigation

Reference

• 🔌 API Reference
• 🤖 MCP (agents) — curllm-mcp for Cursor / Claude Desktop
• 🛠️ Configuration
• ❓ Troubleshooting

🧪 Development

# Clone and install
git clone https://github.com/wronai/curllm.git
cd curllm
make install

# Run tests (388 tests passing)
make test

# Run URL resolver examples
cd examples/url_resolver && python run_all.py

# Run with Docker
docker compose up -d

📄 License

Apache License 2.0 - see LICENSE

🙏 Acknowledgments

Built with:

• Playwright - Browser automation
• Ollama - Local LLM inference
• LiteLLM - Multi-provider LLM support
• Flask - Web framework

---

⭐ Star this repo if you find it useful!

Made with ❤️ by wronai

License

Licensed under Apache-2.0.