Skip to content

fix(worker): stop bundling every docs page twice (#395) - #397

Merged
zhuangjianguo merged 11 commits into
mainfrom
claude/project-thread-u7eoxi
Oct 29, 2026
Merged

zhuangjianguo merged 11 commits into
mainfrom
claude/project-thread-u7eoxi

Conversation

@zhuangjianguo

@zhuangjianguo zhuangjianguo commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Requested by Steedos · project thread
Fixes #395
Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:Worker 里每个文档页面的编译模块原本打包了两份:一份给页面渲染,一份给 6 个 route handler(搜索、og、sitemap、llms 三个)。原因是这些 handler 引用的 lib/source.ts 带着每个页面的懒加载 import。现在 handler 用一个只有 frontmatter 的集合(构建时从 fumadocs-mdx 生成的文件改写),llms 用的英文 Markdown 在构建时生成;页面渲染仍从完整集合加载正文。预算步骤新增每页成本预测。
  • 为什么改:每个译文页面约 170–235 KiB,Other locales, phase 2: ja to full coverage (the remaining 79 pages, seven serial PRs) #388 的 PR 3 会超出 61,440 KiB 预算,日文全量会超出 Cloudflare 64 MiB 上限。
  • 风险与回滚:Worker 上传 59,173 → 38,909 KiB。预渲染输出 9,759 个文件与 main 对比,llms 正文完全一致;只有代码高亮颜色和中日文分享卡图片有差异,而 main 自己构建两次也有同样的差异。回滚即 revert 本 PR。
  • 先看:apps/docs/scripts/gen-route-data.mjs 的 lightCollection(),然后看 lib/source.ts 和 lib/page-body.ts。
  • 两分钟验证:看 CI 的 Worker bundle fits the size budget 摘要,新增「Per-page cost」表,每个译文页约 86 KiB,预算还能容纳约 260 页,再加一个完整语言也放得下。
  • 需要你做的:无,席位落地。改动超出认领范围的部分见 Acceptance notes。

What changes

File Change
apps/docs/scripts/gen-route-data.mjs (renamed from gen-search-data.mjs) Still writes the search text. Now also writes .route-data/collection.ts: .source/server.ts with each page's () => import("…?collection=docs") replaced by () => body("<file>"). It throws unless every page's import was replaced. It also writes markdown.ts, the English pages' processed Markdown (remarkLLMs added at the end of the same remark chain). With --dev, collection.ts re-exports .source/server.ts instead, so next dev still sees pages added while it runs. Output moved from .search/ to .route-data/.
apps/docs/lib/source.ts Builds the loader on @/.route-data/collection. getLLMText(page, processed?) takes the Markdown when the caller already has it.
apps/docs/lib/page-body.ts (new) loadPageBody(page): the only module that imports the full collection.
apps/docs/app/[lang]/docs/[[...slug]]/page.tsx Loads the body through loadPageBody, and passes the loaded _markdown to getLLMText for a translation's Copy Markdown.
apps/docs/lib/packed.ts (new) unpack(): the gzip+base64 reader, moved out of the search route so the collection can use it too.
apps/docs/app/api/search/route.ts Imports from .route-data/ and uses unpack(). No behaviour change.
apps/docs/package.json build, type-check and dev (--dev) run gen-route-data.mjs.
.github/scripts/check-worker-page-cost.mjs (new) Reads OpenNext's handler.mjs.meta.json. It prints each locale's page-module and search-text KiB per page, what one more translated page costs (the most expensive locale's per-page KiB plus 10 KiB of routing data, calibrated below), how many such pages the budget still holds, and whether one more full locale fits. It fails only when it cannot read the bundle (unreadable, no-pages). --self-test has 7 cases.
.github/workflows/ci.yml Worker bundle fits the size budget keeps its verdict (exit "$VERDICT") and prints the projection after it, including when over budget.
tools/ci-scripts/run-self-tests.mjs check-worker-page-cost.mjs added to SELF_TESTED.

The branch carries one -s ours merge of the landed #396 branch head so the push fast-forwards. The diff against main is only these 11 files.

Where the per-page cost went

These numbers come from the esbuild metafile of the packaged handler, with #388's PR 3 applied (61,529 KiB upload):

KiB
page modules, page graph (chunks/ssr/content_docs_*) 19,826
page modules again, route-handler graph (chunks/content_docs_*) 19,847
everything else ~21,900

The second copy exists because route handlers are compiled as their own graph, and .source/server.ts's docsLazy map holds an import() of every page. That import() is bundled whether or not a handler calls load(). The handlers only need frontmatter, the page tree, search text and English processed Markdown, so this PR removes the copy. Highlighting and the prerendered HTML/RSC were not the cause: prerendered output is in .open-next/cache and assets, which do not count toward the upload.

Tests

Worker upload (wrangler deploy --dry-run), all builds on the same machine:

build Total Upload gzip
main f7e4a57 (43 ja pages) 59,173.45 KiB 10,035.53 KiB
this PR (43 ja pages) 38,907.70 KiB 7,754.59 KiB
this PR in CI (run 40880611657) 38,916.49 KiB 7,755.89 KiB
this PR, every ja page filled in (98 ja: 55 English copies) 43,699.18 KiB 8,537.71 KiB
main + full ja (measured on #394, for comparison) 66,779.77 KiB 10,276.88 KiB

Headroom with ja fully covered is 61,440 − 43,699 = 17,741 KiB. One more full locale costs about 8,421 KiB by the projection, which leaves about 9,300 KiB after it. Two more would fit too.

The projection, as the step prints it on this PR's build (CI prints the same table; its headroom is 22,523.51 KiB, leaving 14,102 KiB after one more locale):

| locale | page modules | module KiB | search KiB | KiB per page |
| en      | 98 | 4315.55 | 338.48 | 47.49 |
| zh-Hans | 98 | 5660.60 | 376.04 | 61.60 |
| zh-Hant | 98 | 5684.68 | 377.47 | 61.86 |
| ja      | 43 | 3116.37 | 148.63 | 75.93 |
A translated page costs about 85.93 KiB: ja's 75.93 KiB per page, the most of any locale, plus 10 KiB of routing data.
The budget holds 262 more such pages (22532.30 KiB of headroom).
One more fully translated locale (98 pages) would add about 8421 KiB: it fits, leaving 14111 KiB.

Calibration: going from 43 to 98 ja pages added 4,791 KiB of upload. Of that, 4,019 KiB was page modules and 238 KiB search text, so the remaining 534 KiB over 55 pages is about 10 KiB each. The projection for those 55 pages is 4,726 KiB, 1.4% under the measured growth.

  • Prerendered output. I compared all 9,759 .html/.rsc/.body/.meta files under .next/server/app between main and this PR, normalizing build id and static chunk names. 9,698 are identical. Every llms.mdx body, llms-full.txt, llms.txt and sitemap.xml is identical. The 61 that differ are of two kinds, and two builds of main differ the same way (43 files on the second main build):
    • Code-highlight colours on a few pages (record-access, views, webhooks, backup here; cel, backup between the two main builds). With <script> and the colours removed, the HTML of all 9 differing pages is identical to main. It is a token-colour flicker in shiki's light theme, for example a SQL string drawn red instead of blue. It predates this PR.
    • og share-card PNGs for ja/zh pages, whose CJK glyphs come from a dynamic font fetch (Failed to load dynamic font in every build log here).
  • Browser. One page per section (index, build/interface/actions, configure/permissions, cloud/storage-and-limits, deploy/docker, operate/backup, reference/cel, resources/glossary, start/community, editions/compare, use, architecture) in en and ja. Each was loaded with Playwright from opennextjs-cloudflare preview (workerd) of main and of this PR. The status, article text length and console errors (0) match on all 24. 20 full-page screenshots are byte-identical. The other 4 (operate/backup and reference/cel, in en and ja) differ in one code line's colours, the flicker above.
  • Worker preview of this PR. smoke-docs.mjs --base passes. /docs/build/agents.mdx, /llms-full.txt, /llms.txt, /sitemap.xml, /og/docs/ja/build/agents/image.png and /api/search?locale=ja all answer 200.
  • next dev. /ja/docs/build/agents, /docs/build/agents.mdx and /api/search answer 200, and .route-data/collection.ts re-exports the full collection.
  • Gates.
    • check-search-heap.mjs: level at about 53 MB.
    • check-search-locales.mjs, check-locale-surface.mjs, check-translations.mjs, check-positioning.mjs, check-doc-samples.mjs and gen-zh-hant.mjs --check pass.
    • pnpm turbo run type-check and pnpm turbo run test (14 self-tests) pass.
    • I ran the budget step's script locally with budget 61,440 (exit 0, projection printed) and with budget 30,000 (exit 1, projection still printed).

Acceptance notes

  • After landing, the post-deploy smoke check needs to be green.
  • Beyond the claimed surface:
    • app/api/search/route.ts: import path and unpack() only. The generator it depends on was renamed, and the unpacking moved to lib/packed.ts so the collection shares it.
    • lib/packed.ts and lib/page-body.ts: new siblings of lib/source.ts. page-body.ts has to be a separate module from lib/source.ts, because anything the route handlers import must not reference the full collection.
    • apps/docs/package.json scripts: to run the renamed generator.
    • All three are needed for the fix.
  • Unrelated finding, not fixed: shiki colours some code tokens differently from one build to the next (two builds of main differ in reference/cel and operate/backup). Readers see a string drawn red on one deploy and blue on the next.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

zhuangjianguo and others added 11 commits October 29, 2026 10:17
…ions, use)

Phase 1 of the other-locale plan. Adds ja translations of the 19
entry-path pages (index, why, start/*, editions/*, use/*) and the ja
sidebar labels for start/ and editions/. Every heading keeps its
English id, fences match the English byte for byte, terms follow the
glossary's ja column, and each page carries a --stamp.

check-translations: ja 0 stale, 79 missing (was 98).

Fixes #386

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2 of the other-locale plan, first batch: ja for the build/ top
level (index, agents, ai-builder, ai-skills, extend-existing-systems,
marketplace, packages, reports, templates) and build/data/ (index,
relationships, formulas, validation-rules). Same rules as #386: English
heading ids, fences byte for byte, glossary ja column, UI labels in
English, one line per paragraph, each page stamped.

check-translations: ja 0 stale, 66 missing (was 79).

Refs #388

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
…388, PR 2 of 7)

Adds ja translations for the 11 automation and interface pages:
build/automation/{index,flows,workflows,approvals} and
build/interface/{index,apps,views,forms,dashboards,pages,actions}.
Both sections' meta.ja.json already match the English pages list.

Refs #388

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
…tput (#393)

/api/search kept every locale's Orama index (30-40 MB each) and every
page module page.data.load() imported, so one Worker isolate grew from
58 MB after its first locale to 283 MB after all eight, and #390's ja
pages tipped it past 128 MB into 503s.

- scripts/gen-search-data.mjs computes each page's structuredData before
  next build with fumadocs-mdx's own remark chain and writes one gzipped
  module per locale to apps/docs/.search/; the route reads that instead
  of importing page modules.
- The route keeps the most recently used locale's index only.
- check-search-heap.mjs asks the built route in every locale twice and
  fails above 100 MB or when the second pass keeps growing; it runs in
  ci.yml after check-search-locales and its --self-test is in SELF_TESTED.

Measured on the built route: levels at 50-54 MB across both passes.

Fixes #393

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
The tree is unchanged: the diff against main is the #393 commit only.

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
…388, PR 2 of 7)

Reverts #392, which reverted #390 because /api/search ran out of Worker
memory. #394 (#393) fixed that, so the same 11 ja pages come back
unchanged.

Refs #388

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
The tree is unchanged: the diff against main is the re-landed ja pages only.

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
Six route handlers (search, og, sitemap, llms.txt, llms-full.txt,
llms.mdx) import lib/source.ts, whose fumadocs-mdx collection holds a
lazy import() of every compiled page. Turbopack compiles route handlers
as their own module graph, so the Worker carried a second copy of every
page module: ~20 MB of upload, and each translated page cost ~170-235 KiB
instead of ~86. #388's PR 3 would have gone over the 61,440 KiB budget.

- scripts/gen-route-data.mjs (was gen-search-data.mjs) also writes
  .route-data/collection.ts: .source/server.ts with each page's body
  import replaced by a read of generated data, and markdown.ts, the
  English pages' processed Markdown the llms routes serve. With --dev it
  re-exports the full collection so next dev sees new pages.
- lib/source.ts builds its loader on that collection; the page route
  loads bodies through lib/page-body.ts, the only module that imports
  the full collection.
- lib/packed.ts unpacks the generated gzip+base64 strings.
- The Worker budget step prints a per-page projection from the bundle's
  metafile (check-worker-page-cost.mjs, in SELF_TESTED).

Measured, same machine: main f7e4a57 59,173 KiB, this change 38,909 KiB;
with every ja page filled in, 43,700 KiB.

Fixes #395

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
The tree is unchanged: the diff against main is the #395 commit only.

Claude-Session: https://claude.ai/code/session_01QsR2jRsjbApbf83iyTDDXm

Co-Authored-By: Claude <noreply@anthropic.com>
@zhuangjianguo zhuangjianguo self-assigned this Oct 29, 2026
@objectstack-fleet

Copy link
Copy Markdown
Contributor

席位核对 · ACCEPT · 2026-10-29T13:21Z · session_01QsR2jTaZ2GeyfjANMwstiw · seat repo:docs.objectos.ai#1

  1. Shape. Draft PR against main f7e4a57, head 6769cd3. 11 files, +514/−142, no content files. The body says Fixes #395. Claim 6496835010, branch claude/project-thread-u7eoxi. The tree diff against main is exactly these 11 files. The history merges already-landed heads only, and the queue squashes it.
  2. Surface. Four files are outside the claimed surface: app/api/search/route.ts (import path and unpack() only), the new lib/packed.ts, the new lib/page-body.ts, and the apps/docs/package.json scripts. The seat accepts them. They follow directly from the cut: search and the light collection share the packer, the page route needs the one full-collection import, and the build scripts call the renamed generator. The PR body explains each.
  3. Cause and cut. Six route handlers import lib/source.ts, and the full fumadocs-mdx collection's lazy import() of every page put a second copy of every compiled page in the handlers' graph. gen-route-data.mjs rewrites .source/server.ts into a body-less collection for the handlers. It throws unless the number of page imports it replaced equals the number of pages and none are left. A fumadocs-mdx change to what it writes therefore goes red at build, not silently fat. Bodies load only in the page route, through lib/page-body.ts. The llms routes read English processed Markdown that is generated at build time, and all 98 llms.mdx bodies plus llms.txt and llms-full.txt are prerendered.
  4. Seat re-run at 6769cd3.
    • Build. The local build is green, and the generator wrote search data for en 98, zh-Hans 98, zh-Hant 98, ja 43, and English Markdown for 98 pages.
    • check-search-heap. It passes: under 100 MB, and not growing.
    • check-search-locales. All 8 locales answer 200, and every locale finds its own pages by title.
  5. CI, run 40880611657. build success, with all 14 self-tests passing, including the new check-worker-page-cost.
    • Worker size. Total Upload is 38,916 KiB, down from 59,172 KiB on main. That leaves 22,524 KiB of headroom under the 61,440 KiB budget.
    • Per-page projection. A translated page now costs about 86 KiB (it was about 172–235). 262 more pages fit, and one more full locale would add about 8,421 KiB, leaving 14,102 KiB.
    • Pre-merge preview smoke. It passed: 4 pages render, and search answers in all 8 locales in 0.3–0.5 s.
  6. The dev's parity evidence.
    • Of 9,759 prerendered files, 9,698 are identical, including every llms body and the sitemap. The 61 that differ are shiki token colours and CJK og PNGs, which also differ between two builds of main.
    • 24 pages were browsed in en and ja on the workerd preview. 20 screenshots are byte-identical, and the other 4 differ only in code-line colour.
    • A full-ja simulation measured 43,699 KiB.
  7. Landing. Tier none. It lands through the merge queue under the standing go-ahead, serially, and the post-deploy smoke check must be green. Other locales, phase 2: ja to full coverage (the remaining 79 pages, seven serial PRs) #388 then resumes with PR 3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Each translated page adds about 172 KiB to the Worker upload, so full ja coverage (#388) would exceed Cloudflare's 64 MiB Worker limit

1 participant