BuildCanada Defuddle Worker

A Cloudflare Worker that extracts the main content of any web page — plus special handling for X/Twitter and Facebook posts — and returns clean Markdown (or JSON). Built on defuddle v0.19.2, and protected by a static API key.

This is derived from thieung/defuddle, updated to the latest defuddle release and hardened with API-key auth. It is a standalone project (not a GitHub fork).

What's different from the upstream project

• defuddle bumped 0.15.0 → 0.19.2 — picks up the content-sanitization fixes (extractor HTML sanitization, data:/blob: URL rejection, SVG <style> stripping), plus footnote/math/table/image extraction improvements and many site-specific fixes.
• Static API key auth on all endpoints (except OPTIONS and /health).
• API-only — the public demo landing page and static-asset binding were removed so nothing is served unauthenticated.
• X/Twitter and Facebook extractors are retained unchanged (they are self-contained and do not depend on defuddle).

Architecture

• Generic pages → fetched, parsed with linkedom, extracted with defuddle, then converted to Markdown with turndown.

  Note: markdown is produced by Turndown rather than defuddle's native serializer, matching the upstream approach — linkedom lacks some DOM APIs defuddle's built-in markdown path expects. Extraction quality still comes from defuddle 0.19.2.

• X/Twitter → fetched via the FxTwitter API and rendered to Markdown (text, media, polls, quotes, long-form Articles via a DraftJS converter, engagement stats).
• Facebook → fetched via the post embed plugin with an og:meta fallback.

Setup

npm install

Set the API key. For production (recommended — stored as an encrypted secret):

npx wrangler secret put API_KEY
# paste your key when prompted

For local development, copy .dev.vars.example to .dev.vars and set a key there (this file is gitignored):

cp .dev.vars.example .dev.vars

Run locally

npm run dev

Deploy

npm run deploy

Verify / typecheck

npm run typecheck                       # tsc --noEmit
npx wrangler deploy --dry-run --outdir dist   # confirm it bundles

Usage

All requests require the API key via either header:

• X-API-Key: <key>, or
• Authorization: Bearer <key>

Browser clients may use HTTP Basic Auth. Enter the API key as the username and leave the password blank. An unauthenticated request receives a Basic Auth challenge so browsers show their credential prompt.

Browser navigation also supports HTTP Basic Auth. Enter the API key as the username and leave the password blank when prompted. Requests whose Accept header includes text/html receive a styled HTML rendering; JSON and Markdown clients retain their existing response formats.

JSON responses include html (the raw extracted HTML before Markdown conversion) and css (the companion stylesheet used by the browser view).

GET /{url} — convert a page

The URL is the path after the leading slash. Use Accept: text/html for the styled browser document, Accept: application/json for structured output, or any other Accept value for Markdown.

# Regular page → Markdown
curl -H "X-API-Key: $KEY" https://<your-worker>/https://stephango.com/vault

# X/Twitter post
curl -H "X-API-Key: $KEY" https://<your-worker>/https://x.com/user/status/123

# Facebook post
curl -H "X-API-Key: $KEY" https://<your-worker>/https://facebook.com/...

# JSON output
curl -H "X-API-Key: $KEY" -H "Accept: application/json" \
  https://<your-worker>/https://stephango.com/vault

JSON output includes the extracted metadata, content as Markdown, html as the raw extracted HTML fragment, and css as the companion browser stylesheet.

POST /api/convert — JSON in, JSON out

curl -X POST https://<your-worker>/api/convert \
  -H "X-API-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://stephango.com/vault"}'

Body fields: url (required), cookies, contentSelector, language, includeReplies.

GET /health or /healthz — unauthenticated liveness check

Returns {"status":"ok"} without authentication.

Optional query params (GET) / body fields (POST)

Param	Meaning
contentSelector	CSS selector to force as the main content element
language	Preferred content language (BCP-47, e.g. en, fr)
includeReplies	Include replies/comments (true/false)

For GET, page cookies can be forwarded with the X-Custom-Cookie header (useful for content behind a login). Treat that as sensitive — only send it to a worker instance you control.

Notes & caveats

• This runs on linkedom (like the upstream worker and defuddle.md), so there is no JavaScript execution and no CSS/mobile-styles heuristic — heavily client-rendered SPAs may extract thin. Clean server-rendered articles and the dedicated X/Facebook paths work well.
• The 5 MB page-size cap and localhost host block from upstream are retained.