Repository navigation
Task/ccl 8779 txt files for llms fixed - #67
Merged
Merged
Conversation
The per-page .txt files used by the MCP server and AI agents were generated on every deploy, but diverged from the HTML pages in two ways: - Content rendered by React components was silently dropped: <McpNotice/> (added on 41 pages in UOD-430), <PriceBlock/> and <BlogLink/>. They are now rendered to text; prices are fetched once at build time from the same public endpoint the site uses, and omitted (never failing the build) if the request fails. - Docs following the Docusaurus category-index convention (mcp/mcp.mdx, external-services/external-services.mdx, extension/captcha-parameters/captcha-parameters.mdx) got URLs like /docs/mcp/mcp/ (404) and were published at /docs/mcp/mcp.txt instead of /docs/mcp.txt next to the real page URL. Also normalise CRLF so local runs on Windows produce the same output as the Linux build, keep component-generated lines out of llms.txt page descriptions, and add --no-prices for offline/deterministic runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- scripts/verify-llms.mjs (npm run verify-llms): post-build check that llms.txt, llms-full.txt and every /docs/**.txt exist, that each txt's URL header points to a page that exists in the build, that the file count matches the publishable source docs, and that no raw MDX (imports, JSX tags, admonition markers) leaked into the text. - .github/workflows/llms-txt.yml: on pull requests and pushes to master, run the same build command as production and then the verifier. Algolia env vars are placeholders only; real keys stay in the Cloudflare Pages project settings. - README: document that the site is deployed by Cloudflare Pages from master, how the txt files are produced, and how to check them locally. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
generate-llms.mjs - Rewrite internal links in a second pass with the full set of generated page URLs: a link becomes ".txt" only when that file exists; links to category (generated-index) pages keep their page URL instead of pointing at a non-existent captchas.txt / category/api.txt / extension.txt. - Links to .md/.mdx files resolve relative to the source file and collapse the category-index name (extension-main -> captcha-parameters.txt). - Extension-less relative links (../recaptcha-v3-task/) resolve against the page URL like a browser does, fixing two links that previously escaped their folder. - Absolute-link rewrite honours the locale prefix (/ru/docs/ too). - <PriceBlock/> lines carry their label (title ?? item.Name) so multiple prices on one page are distinguishable; <PriceBlockWrap> emits its "View full price" link. - collapseCategoryIndex compares names case-insensitively. - firstParagraph keeps inline-code text instead of deleting it. verify-llms.mjs - Check in-body doc links as well as llms.txt: .txt targets must exist as files, page targets must exist as <path>/index.html. - Fix the no-op regex escape; strip BOM before draft detection. - Warn (not fail) when no page contains a price line, so a prices API outage during a build is visible. README: drop the unverified claim about preview builds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
<ParamItem required="required (if Task is not filled)"> was reduced to a bare "required", losing the condition; parameters without `required` showed no marker although the page prints "optional". Mirror the component: string value verbatim, boolean -> "required", otherwise "optional", nothing when `response` is set. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- generate-llms.mjs: strip trailing whitespace before collapsing blank lines, so indented JSX children no longer leave whitespace-only runs; skip the "View full price" link when prices are unavailable. - verify-llms.mjs: write the BOM as instead of a raw character; ignore links to non-page assets (e.g. a PDF) under the docs prefix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.