Repository navigation
docs site: any single-segment path containing a dot renders the homepage with 200 (soft-404 class) #12233
Description
One-liner
/foo.txt, /ads.txt, /sitemap_index.xml, /security.txt — every single-segment path with a dot in it returns 200 with the full homepage. Paths without a dot 404 correctly. So the site publishes an unbounded set of duplicate homepages at exactly the URLs crawlers probe by default.
Measured (local dev server, same routing in production)
/foo.txt 200 title=<title>ObjectStack — Apps small enough for AI to hold whole.</title>
/ads.txt 200 title=<title>ObjectStack — Apps small enough for AI to hold whole.</title>
/sitemap_index.xml 200 title=<title>ObjectStack — Apps small enough for AI to hold whole.</title>
/security.txt 200 title=<title>ObjectStack — Apps small enough for AI to hold whole.</title>
/this-page-does-not-exist 404 ← no dot, correct
Root cause
apps/docs/proxy.ts matcher excludes dotted paths from locale rewriting:
matcher: ['/((?!api|_next/static|_next/image|favicon.ico|.*\\..*).*)'],so /robots.txt skips the proxy and falls straight through to the single-segment dynamic route app/[lang]/page.tsx, which matches with lang = "robots.txt" and renders HomePage() — it never checks whether lang is a real locale. A dotless path is rewritten to /en/<path>, becomes two segments, matches nothing, and 404s properly.
Expected
[lang] accepts only the locales in lib/i18n.ts. Either export const dynamicParams = false alongside the existing generateStaticParams, or an explicit if (!i18n.languages.includes(lang)) notFound() in app/[lang]/layout.tsx — whichever the Next 16 app router honours for both the static and the dev path. Verify both, do not assume.
Why it matters
Soft-404s at this scale burn crawl budget and split ranking signal across junk URLs; and the two paths that matter most (/robots.txt, /sitemap.xml) are in the affected set, so this card and the robots/sitemap card must both land for either to be worth anything.
Acceptance
-
/foo.txt,/ads.txt,/security.txt,/anything.htmlall return 404 -
/robots.txtand/sitemap.xmlstill resolve to their real routes (regression check against the sibling card) -
/docs,/docs/<any real page>,/blogunchanged - a test or gate pins the behaviour so the catch-all cannot silently return
Source
Found in an SEO review of the docs site (apps/docs) run on 2026-08-25, measured against the local dev server and against production. The canonical origin is https://objectstack.ai — maintainer ruling recorded in #10659:
这个仓的文档站规范 URL 是 https://objectstack.ai
Activity
os-zhuang commentedon Aug 25, 2026
Claim: PM loop round 1 (epic PM for #12243)
Session:
session_f9f0958b-ab68-46cc-801c-216aa7ee2107Branch:
claude/issue-12233-dotted-soft-404Worktree:
objectstack-issue-12233Domain:
domain:devxFile surface:
apps/docs/proxy.tsandapps/docs/app/[lang]/layout.tsx(or the[lang]page, whichever the Next 16 router actually honours) plus a regression test (stop on breach; explain in the report)Container & model: M,
mode:subagent,model: opus— no path-derived mandate; routing semantics need measuring in both dev and build modes rather than reasoningClause-②: no
Serial constraints cleared: none. Disjoint from #12232 (which only adds files) and from #12236 (content only)
os-zhuang commentedon Aug 25, 2026
Dev claim (os-dev subagent, dispatched by the PM claim above).
session_f9f0958b-ab68-46cc-801c-216aa7ee2107(same session as the epic PM — I am theos-devsubagent it dispatched, not a second claimant)claude/issue-12233-dotted-soft-404(pushed empty at start, base20b0fdb568c648374963cc13fdab89f4f74594bf)objectstack-issue-12233proxy.tsmatcher arrived intact — it is byte-identical toapps/docs/proxy.tsonorigin/main(blob4c8540cfdbe284598b68d1d48592e5431b54559b), so no reconstruction was needed.Plan: re-measure both halves of the asymmetry on a dev server first, confirm or correct the root-cause reading, then measure the candidate fixes in both
next devand a productionnext build+next start, since the card explicitly does not assume they agree.os-zhuang commentedon Aug 25, 2026
os-dev-report
{ "issue": 12233, "status": "done", "branch": "claude/issue-12233-dotted-soft-404", "pr": "https://github.com/objectstack-ai/objectstack/pull/12258", "premise_still_valid": true, "summary": "Both PM assumptions confirmed by measurement, not reasoning. (1) The asymmetry reproduces on both halves: /foo.txt, /ads.txt, /security.txt, /anything.html, /sitemap_index.xml, /robots.txt, /sitemap.xml all returned 200 with the full homepage; /this-page-does-not-exist correctly 404'd. (2) The root-cause reading is exactly right: the dev request log prints a `proxy.ts:` timing only for paths the proxy actually handled, and dotted paths have none, while they DO show `generate-params` (generateStaticParams from app/[lang]/layout.tsx) -- and a temporary probe in the layout printed lang=\"foo.txt\", lang=\"ads.txt\", lang=\"robots.txt\", lang=\"sitemap.xml\" against lang=\"en\" for /docs, /blog and /. The issue body's quoted matcher arrived INTACT -- byte-identical to apps/docs/proxy.ts on origin/main (blob 4c8540cfd), no reconstruction needed. (3) On assumption 3: Next 16 honours BOTH candidates in dev -- measured separately, each alone, each 404ing every dotted path with every real page unchanged. I shipped the explicit guard, not dynamicParams=false: the guard states the contract at the point it is violated, is render-mode-independent, and cannot silently degrade if generateStaticParams is later edited, whereas dynamicParams=false derives its correctness from generateStaticParams and is a segment config that also governs the nested [lang]/docs/[[...slug]] and [lang]/blog/[[...slug]] segments -- a wider blast radius than this card for no externally visible difference (both yield 404). It remains available as a follow-up optimization. proxy.ts is deliberately UNCHANGED: making its matcher rewrite dotted paths would send /llms.txt, /llms-full.txt, /og/**/*.png and /docs/**.mdx to /en/<same> where nothing matches, 404ing exactly the agent-reader endpoints the epic protects. Fix: lib/i18n.ts exports isSupportedLanguage() derived from i18n.languages; app/[lang]/layout.tsx calls notFound() before rendering when lang is not a declared locale. app/[lang]/page.tsx was NOT touched (PR #12218 owns it) and did not need to be -- the layout is the right home anyway, since it also wraps [lang]/docs/** and [lang]/blog/**. FINDING FOR #12232: its precedence assumption HOLDS, measured not assumed -- app/robots.ts and app/sitemap.ts were added temporarily, the site rebuilt WITH this guard active, and /robots.txt returned 200 text/plain and /sitemap.xml 200 application/xml (build route table: `o /robots.txt`, `o /sitemap.xml`) while /ads.txt and /foo.txt stayed 404. Metadata routes beat the [lang] catch-all; this guard does not interfere. The probe files were removed and the tree left clean. Same precedence already visible pre-fix for the existing static dotted route app/llms.txt/route.ts, which never reached [lang] in either direction. Docs-only, no changeset; `skip-changeset` added additively and read back as [\"skip-changeset\"].", "tests": "All at final commit 6f5d011 (git rev-parse --short HEAD on the pushed branch), except runs explicitly against base 20b0fdb56.\n\nBEFORE/AFTER STATUS CODES -- measured both ways in BOTH modes (the prod 'before' column is a real next build of the pre-fix tree, so 'after' reads as a change, not a state):\n path dev-before | dev-after | prod-before | prod-after\n /foo.txt 200 homepage | 404 | 200 homepage | 404\n /ads.txt 200 homepage | 404 | 200 homepage | 404\n /security.txt 200 homepage | 404 | 200 homepage | 404\n /anything.html 200 homepage | 404 | 200 homepage | 404\n /sitemap_index.xml 200 homepage | 404 | 200 homepage | 404\n /robots.txt 200 homepage | 404* | 200 homepage | 404*\n /sitemap.xml 200 homepage | 404* | 200 homepage | 404*\n /this-page-does-not-exist 404 | 404 | 404 | 404\n /docs 200 'Documentation | ObjectStack' | same | 200 | 200\n /docs/getting-started/quick-start 200 'Anatomy of an ObjectStack App' | same | 200 | 200\n /docs/getting-started/quick-start.mdx 200 text/markdown | same | 200 | 200\n /en 307 -> / | same | 307 | 307\n /en/docs 307 -> /docs | same | 307 | 307\n /blog 200 'Blog | ObjectStack' | same | 200 | 200\n /blog/protocol-first-development 200 | same | 200 | 200\n / 200 homepage | same | 200 | 200\n /llms.txt 200 text/plain | same | 200 | 200\n /llms-full.txt 200 text/plain | same | 200 | 200\n /og/docs/image.png 200 image/png | same | 200 | 200\n /og/docs/getting-started/quick-start/image.png 200 image/png | same | 200 | 200\n /api/search?query=object 200 application/json | same | 200 | 200\n * /robots.txt and /sitemap.xml 404 only because those routes do not exist yet (#12232's work); the precedence probe above shows they return 200 with their real content once they do.\n\nREGRESSION PIN -- scripts/check-docs-locale-catch-all.mjs, wired as `pnpm check:docs-locale-catch-all` in Lint & Repo Gates. It encodes ONE CONDITIONAL INVARIANT rather than a text pin: if a dotted single-segment path bypasses proxy.ts's matcher, then every top-level dynamic segment under apps/docs/app/ must reject a non-locale parameter before rendering. Requirement is on the SEGMENT class, so a future app/[slug]/ is caught too.\n - self-test: `check-docs-locale-catch-all --self-test` exit 0 -- verdict line: '12 assertions over a temp fixture (real checkApp path); every limb -- deleted guard, guard behind the return, hollowed predicate, a new unguarded segment, an uncompilable matcher -- observed FAILING, and the proxy condition observed flipping the requirement off.'\n - REAL RED AGAINST REAL DATA (not construction): run against the actual pre-fix files from 20b0fdb56 it reports 2 findings ('app/[lang]/ is a top-level catch-all and nothing rejects a non-locale `lang`...' and 'lib/i18n.ts does not export `isSupportedLanguage`'); 0 on this branch.\n - live tree: exit 0 -- verdict line 'check-docs-locale-catch-all: 1 top-level dynamic segment(s), 1 guarded; dotted paths bypass proxy.ts: true'\n\nGATES (exit codes captured before any pipe; verdict lines quoted from the gate's own output):\n pnpm --filter @objectstack/docs typecheck exit 0 (script name echoed: 'fumadocs-mdx && next typegen && tsc --noEmit'; 'Types generated successfully')\n pnpm --filter @objectstack/docs exec next build exit 0 ('Compiled successfully in 25.4s'; 'Generating static pages using 2 workers (1220/1220)')\n eslint . --no-inline-config (repo-wide, 8G heap) exit 0, zero findings\n pnpm check:nul-bytes exit 0 ('OK (scanned 6766 text file(s))')\n pnpm check:docs-redirects exit 0 (92 entries)\n pnpm check:entry-guard / check:parse-guard exit 0\n check-self-test-wired / -workflow-commands exit 0 (133 scripts; the new self-test IS run by CI)\n check-step-collectors / -aggregator-roster / -ci-filter-parity / -whole-set-label-write exit 0\n pnpm check:required-contexts / check:workflow-status-functions / check:pnpm-filter-targets exit 0\n pnpm check:cross-package-test-inputs / check:agent-test-spelling exit 0\nFamily list re-derived AFTER the edits with `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack` (27 families matched over the 5 changed paths), not taken from the dispatch prompt. That run reports the tree as STALE against a fast-moving origin/main; the files it named stale (check-driver-conformance.mjs, gen-sdui-manifest.sh, os-regen-merge.sh, publish-smoke.sh) are unrelated to this change surface.\n\nNOT MEASURED (reported as such, not as red): `pnpm lint` as spelled aborts on this host with 'FATAL ERROR: Ineffective mark-compacts near heap limit' at ~4055 MB, exit 134 -- a V8 heap OOM, not a lint result. The identical sweep with --max-old-space-size=8192 is green with zero findings, which is the row above. Filed as #12261.\n\nDECLARED NARROWING: scripts/pm/os-verify-lock.sh ran in DECLARED UNLOCKED MODE on this host (macOS, no usable flock -- the shared verify lock is Linux-only). Every heavy command went through the entry point and its official disclosure wording is pasted in the PR body. VERDICT lines read 'command-exit 0 . UNLOCKED (declared)'.\n\nMUTATION/RESTORE DISCIPLINE: three temporary mutations, each confirmed on disk before measuring and each restored by `git checkout HEAD -- <path>` with a git-hash-object comparison against the HEAD blob: (a) the lang probe in layout.tsx (restored to f1c6ad457, probe grep back to 0, `git diff HEAD` empty); (b) the dynamicParams=false variant; (c) the pre-fix revert for the production reverse verification (restored to 9202a8f25 / ae20a3c32, guard grep back to 1, `git diff HEAD` empty). The #12232 precedence probe files carried a `trap ... EXIT INT TERM` and their absence was confirmed by ls + `git status --porcelain` empty.\n\nENVIRONMENT NOTE: the host disk sat at 99% (11-15 GiB free) throughout. Turbopack's dev cache logged 'No space left on device (os error 28)' and disabled persistence, and during that window two consecutive /blog requests returned a transient 404 mid-HMR-recompile. A 20-request burst afterwards was 20/20 200, and the production build -- which has no HMR -- returns /blog 200 in both the before and after runs, so this was an environment artifact, not the change. Flagged rather than swallowed.\n\nCLEANUP: dev server (39233) and all four next-start servers (39234/5/6/7) stopped by recorded PID, never by process name; all five ports confirmed free and all six PIDs confirmed gone. .next and node_modules removed, `git worktree remove` succeeded WITHOUT --force on a clean tree with 0 unpushed commits.", "open_questions": [], "out_of_scope_findings": [ "filed as #12261: `pnpm lint` aborts with a V8 heap OOM (exit 134) at the default old-space limit -- the script pins --stack-size=4000 but not --max-old-space-size, and lint.yml runs it with no NODE_OPTIONS; green with an 8G heap, so headroom rather than a live outage (label: finding, unassigned)", "already filed by another agent as #12255 (independently confirmed here, no duplicate opened): apps/docs/app/page.tsx is a shadowed route -- the proxy rewrites / to /en before it can run, so `/` is served by app/[lang]/page.tsx and the redirect('/docs') in app/page.tsx is dead code" ] }```os-zhuang commentedon Aug 25, 2026
ACCEPT — PR #12258, reviewed against the diff rather than the report.
main, 5 files, +465/-1.skip-changeseton,Check Changesetgreen with it.isSupportedLanguage()derived fromi18n.languages, called inapp/[lang]/layout.tsxbefore render. It also covers[lang]/docs/**and[lang]/blog/**, andapp/[lang]/page.tsxwas correctly left alone.proxy.tsdeliberately unchanged, and the reasoning is better than my card's. I listedproxy.tsin the file surface as if widening the matcher were an option; it is not. Rewriting dotted paths would send/llms.txt,/llms-full.txt,/og/**/*.pngand/docs/**.mdxto/en/<same>, where nothing matches — 404ing exactly the agent-reader endpoints the epic exists to protect. Refusing the surface I handed it is the right call and I am recording it as such.dynamicParams = falsewas measured and rejected on blast radius, not on taste: it is a segment config that would also govern the nested[lang]/docsand[lang]/blogsegments, and it derives its correctness fromgenerateStaticParamsstaying correct. Both candidates were measured to 404 every dotted path; the explicit guard states the contract where it is violated./robots.txtand/sitemap.xmldo not exist — both answer 200 with the homepage HTML #12232: this dev independently addedapp/robots.ts/app/sitemap.tsas throwaway probes, rebuilt with its own guard active, and measured/robots.txt→200 text/plainand/sitemap.xml→200 application/xmlwhile/ads.txtand/foo.txtstayed 404 — then removed the probes and confirmed the tree clean. Two independent agents now agree that metadata routes outrank the[lang]catch-all, and the two changes compose. That is the single most useful thing in this report.app/[slug]/is caught too. It was run red against the real pre-fix commit and reported both real findings, which is ablation on real data rather than on a fixture..github/workflows/lint.ymland rootpackage.jsonto register a newcheck:*gate. They will conflict textually. Whichever enters the queue second must mergemainand re-push first — the queue rebuilds against currentmainand will eject it otherwise.CI at review time: 21 green, 0 red, 8 still converging. Ready and queue happen on full green, required or not.
os-zhuang commentedon Aug 25, 2026
Correction to the landing-order warning above — #12258 and #12262 do not conflict. I asserted they would without testing it.
Measured, read-only:
git merge-tree --write-treebetween the two branch heads exits 0 with no conflict, because the two new gate steps land at different line ranges in.github/workflows/lint.yml(around 1111 and 1071) and theirpackage.jsonadditions are one line each in different places. Both may be queued together; neither needs to wait for the other.The general shape of the warning was right — two PRs registering a gate in the same workflow file is worth checking — but I stated the outcome instead of measuring it, which is the failure mode this repo's own rules name. Recording it here rather than deleting the claim.