feat(docs): serve Steel Docs under steel.dev/docs and redirect docs.steel.dev

[nibzard]

Summary

Prepares Steel Docs to move from https://docs.steel.dev to https://steel.dev/docs with Vercel Microfrontends, and adds the redirect project that will own docs.steel.dev after the move.

• Path mode. One URL seam (lib/docs-path.ts) with two build-time variables: NEXT_PUBLIC_DOCS_PATH_PREFIX (empty or /docs) and NEXT_PUBLIC_DOCS_ORIGIN (default https://docs.steel.dev). In path mode, every canonical, og:url, og:image, JSON-LD id, sitemap <loc>, llms.txt/llms-full.txt URL, API catalog link, Link header, internal redirect, rendered MDX link and media URL, and client navigation uses https://steel.dev/docs/.... withMicrofrontends is applied only in path mode. No Next basePath.
• Root mode is the default and does not change. With the variables unset, the build serves docs.steel.dev as today, so this PR is safe to merge before the cutover.
• Redirect project for the old host. docs-host-redirect/ is a static Vercel project (only vercel.json, robots.txt and an old-URL sitemap.xml). It maps 660 old URLs one to one to steel.dev/docs with 308s (307 where the docs redirect was temporary), keeps query strings, and returns 404 for unknown paths. scripts/generate-docs-host-redirects.ts writes it; --check fails if it is stale.

Cross-linked with the steel-web PR: https://github.com/steel-dev/steel-web/pull/18

Reused from the September pilot

The 5 pilot commits from feat/docs-path (3e0f9da, d2ede44, 2d67604, ade730eb, 3f5247b4) are cherry-picked onto current main (322ad57, 8 commits newer). There was one conflict, in app/(home)/_pages/page.en.tsx: main switched the logo import to SteelMark and the pilot added a docsUrl import, so the resolution keeps both. The pilot's app/favicon.ico → public/favicon.ico move carried main's new favicon bytes (same blob as main). The lockfile is regenerated with the pinned bun 1.4.2 because the auto-merged lockfile failed --frozen-lockfile.

New in this PR:

• fix: /docs/overview and /docs/changelog redirected to steel-web paths (/ and /changelog/...) in path mode. These now go through docsPath (the same fix as the uncommitted change in the docs-path worktree, which this PR leaves alone). The interactive badge navigation is also prefixed.
• microfrontends.json now matches steel-web on Next.js (port 3000, fallback steel.dev, routes /docs and /docs/:path*).
• More path-boundary tests, and ABOUTME headers on the pilot's files.
• The redirect project, its generator, its coverage tests, and scripts/verify-docs-host-redirect.ts.

The pilot's plan 004 preferred a standalone redirect-only Vercel service outside the Microfrontends group over host-conditioned middleware in the docs app. This PR follows that. Why in this repo: the map comes from this repo's source (Fumadocs pages, public/, next.config.mjs redirects), so the coverage test can fail on the same PR that adds a page or a redirect. Why vercel.json only: no runtime, no build, no code fork of the docs. 660 entries is under Vercel's 2,048 limit for redirects.

Redirect inventory (660)

Kind	Count	Example
Pages (HTML)	186 + /	/overview/stealth/captcha-solving → https://steel.dev/docs/overview/stealth/captcha-solving
Markdown pages (.md, only pages without llm: false)	157	/overview/steel-cli.md
Open Graph images	188	/og/overview/steel-cli
Public files (images, favicon, section llms.txt, agent-skill archives, API catalog)	83	/images/logo.png, /.well-known/agent-skills/index.json
App routes (/llms.txt is also a public file)	5	/llms-full.txt, /AGENTS.md, /DESIGN.md, /changelog, /api/search
Old redirects from next.config.mjs and /overview, resolved to one hop	40	/cookbook/playwright-ts → https://steel.dev/docs/cookbook/playwright#typescript; /api-reference and /openapi.json go straight to their external targets

Two content pages (/overview/llms-full.txt, /overview/self-hosting/render) are hidden by old redirects, as on the live site, so they keep the redirect.

Sources and checks:

• Pages from the Fumadocs source; the test fails if a page, a tracked public file or an old redirect has no entry.
• tests/fixtures/docs-host-known-urls.txt: 167 paths from the live sitemap (150 URLs) and every Search Console page (16 months) that did not return 404. The test fails if one is not mapped.
• All 660 sources probed on live docs.steel.dev on 2026-10-08: 619 × 200, 39 × 308, 2 × 307, 0 × 404. Every live redirect matches the map's final destination, except /changelog (mapped to /docs/changelog, which still redirects to the latest entry) and a %2F encoding difference in the Render URL.
• robots.txt and sitemap.xml on the old host are served by the redirect project. The sitemap lists the old page URLs so crawlers revisit them and see the redirects.
• Spanish (es): idioma.json lists es as a target, but no es content exists. docs.steel.dev/es returns 404, no page emits hreflang, and Search Console has no es URL. So there is nothing to map. The prefix seam applies to every path, so a later es tree is covered when the map is regenerated.
• Not mapped on purpose: /llms.mdx/* (internal rewrite target), /_next/*, /api/feedback. Search Console also shows 54 URLs that already return 404 on docs.steel.dev (for example /overview/integrations/*, /overview/guides/connect-with-*, /integrations/*/quickstart). They stay 404. If redirects for them are added to next.config.mjs before the cutover, a regenerate picks them up.

SEO migration checklist

• 1:1 redirect map for every live URL, one hop, permanent, query kept, unknown → 404
• Canonical, og:url, og:image, JSON-LD ids → https://steel.dev/docs/... in path mode (verified on 620 composed paths, see below)
• New sitemap at https://steel.dev/docs/sitemap.xml with only steel.dev/docs URLs; steel-web's robots.txt names it (steel-web PR)
• Old sitemap and permissive robots.txt stay on docs.steel.dev so crawlers see the redirects
• llms.txt, llms-full.txt, AGENTS.md, API catalog, agent-skills index, Link headers → new URLs
• hreflang: none emitted, no es pages (see above)
• Titles and meta descriptions unchanged (compared page by page with live docs.steel.dev)
• GSC Change of Address is not available for a subdomain → subfolder move. Rely on the redirects, the new sitemap, and the old sitemap. sc-domain:steel.dev already covers both hosts.
• Monitoring plan (below)

Owner cutover steps

None of these were done. Order matters.

Before the cutover (no visible change)

1. Review and merge this PR. Root mode is the default, so production docs.steel.dev does not change. The redirect directory is inert until a project points at it.
2. Create the Vercel Microfrontends group (the pilot group steel no longer exists; GET /v1/microfrontends/groups returns none): default app steel-web, child docs. Creating a group changes nothing until a microfrontends.json reaches steel-web production. Check the request pricing that the pilot recorded.
3. On the docs project, remove the stale feat/docs-path Preview variables. For preview tests, set Preview variables NEXT_PUBLIC_DOCS_PATH_PREFIX=/docs and NEXT_PUBLIC_DOCS_ORIGIN=https://steel.dev on a docs branch whose name matches the steel-web branch (for example push this branch also as niko/docs-microfrontend), so branch routing pairs them. LLMS_BASE_URL is not needed; it derives from the two variables.
4. On the composed steel-web branch URL, run npm run verify:docs -- <url> --redirect-map ../docs/docs-host-redirect/vercel.json --preview --compare-old-host (steel-web) and bun scripts/verify-docs-browser.ts <url> (here). Both must pass.
5. Re-run bun run generate && bun run generate-docs-host-redirects on the final main, commit any change, and confirm --check and bun test pass.
6. Create a Vercel project docs-host-redirect, outside the Microfrontends group: Git repo steel-dev/docs, Root Directory docs-host-redirect, framework Other. Deploy it and run bun scripts/verify-docs-host-redirect.ts https://<project>.vercel.app. Confirm especially /.well-known/* (Vercel documents /.well-known as reserved for vercel.json redirects) and the fragment destinations (query before #).

Cutover (same session, in this order)
7. Set Production variables on docs: NEXT_PUBLIC_DOCS_PATH_PREFIX=/docs, NEXT_PUBLIC_DOCS_ORIGIN=https://steel.dev. Redeploy and promote. docs.steel.dev keeps working: a path-mode build still serves root paths and /docs/*, with canonicals already on steel.dev/docs (checked locally).
8. Merge the steel-web PR (after the interim redirect PR, see its body). Run npm run verify:docs -- https://steel.dev --redirect-map ... --compare-old-host.
9. Move the docs.steel.dev domain from the docs project to docs-host-redirect. No DNS change is needed if both projects are in nen-labs. Run bun scripts/verify-docs-host-redirect.ts https://docs.steel.dev. Do this only after step 8 passes; the other order creates a loop of 404s.
10. Search Console: submit https://steel.dev/docs/sitemap.xml; keep https://docs.steel.dev/sitemap.xml submitted; URL-inspect and request indexing for the top pages (/docs/overview/stealth/captcha-solving, /docs, /docs/overview/pricinglimits, /docs/overview/stealth/proxies, /docs/overview/self-hosting/docker).
11. Analytics: the docs GTM tags now see steel.dev as the host. Check GA4/Clarity filters that use docs.steel.dev.
12. Keep docs-host-redirect and its domain for at least a year.

Monitoring

• Weekly for 12 weeks in Search Console, Performance by page: docs.steel.dev clicks and impressions should fall as steel.dev/docs rises. Watch /overview/stealth/captcha-solving (≈3.3M impressions in 16 months), /overview/pricinglimits, /overview/stealth/proxies, /overview/self-hosting/docker, /llms-full.txt.
• Page indexing report: docs.steel.dev URLs move to "Page with redirect"; steel.dev/docs URLs to "Indexed". Investigate any "Not found (404)" on steel.dev/docs.
• Vercel Observability on docs-host-redirect: 404s show old URLs that the map misses. Add a redirect in next.config.mjs and regenerate.
• Rollback trigger: steel.dev/docs 5xx, broken navigation, or a sustained drop in total docs clicks of more than 30% after 3 weeks.

Rollback

• Before step 9: Instant Rollback the steel-web production deployment (removes /docs routing) and the docs production deployment (back to root mode). docs.steel.dev was never changed.
• After step 9: move docs.steel.dev back to the docs project first, then roll back docs to a root-mode deployment, then steel-web. Browsers cache 308s, so visitors who already followed a redirect keep going to steel.dev/docs. Keep steel.dev/docs serving during any rollback.

Verification done

• bun install --frozen-lockfile (bun 1.4.2), bun run test (359 pass, 0 fail), bun run typecheck, bun run check and the CI form with --error-on-warnings (263 files, clean), bun run validate-links (0 errors), git diff --check.
• Root-mode bun run build: pass. Compared with live docs.steel.dev (same base commit 322ad57) on all 150 sitemap pages: title, description, canonical, og:url, og:image and links are equal. Differences are only GTM (the variable is not set locally) and the home page's recently-updated list, which depends on Git history depth. robots.txt, sitemap.xml and llms.txt are byte-equal.
• Path-mode build (/docs, origin https://steel.dev): pass. Composed locally with the steel-web branch through microfrontends proxy 2.4.0. The steel-web verifier passed on 620 mapped paths with --compare-old-host: every page 200 with canonical and og:url on https://steel.dev/docs/..., no link out of /docs, no /docs/docs, Markdown and images correct, machine endpoints, negotiation (GET and HEAD), true 404s, feedback validation 400, robots, changelog redirect. Assets load from /vc-ap-e3e2a9/_next/....
• Path-mode build served directly on its own host (the cutover window for docs.steel.dev): root paths, /docs/* and prefixed assets return 200.
• The vercel.json map compiles with @vercel/routing-utils (692 routes) and matches the expected sources.
• Not done (needs owner access): hosted composed preview, verify-docs-browser.ts against it, a deployment of docs-host-redirect.

Open risks

• Path mode sets Cache-Control: private, no-store on negotiated pages (pilot decision, because Next can overwrite Vary on prerendered HTML). This turns off CDN caching of docs HTML on steel.dev/docs. Measure TTFB and cost on the preview.
• /.well-known/* redirects on the redirect project (6 entries) may be refused by Vercel. If so, those agent-discovery URLs on the old host return 404.
• Some content still names docs.steel.dev in code samples and inline code (overview/browser-tools/overview.mdx uses it as a sample scrape target; overview/steel-cli.mdx shows https://docs.steel.dev/llms-full.txt). Links in rendered HTML and Markdown are rewritten; these code strings are not. A content follow-up can update them after the cutover.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 8, 2026 8:51am UTC