ci: keep studio.geolibre.app out of search results - #1843
Conversation
The gated instance is meant to be reached by invitation, and nothing on it is useful to a crawler — every route renders the sign-in card until a session exists. Write a robots.txt disallowing everything, and append a noindex meta to <head> for crawlers that ignore it. The two deliberately do not compose: a crawler honouring the Disallow never fetches the page and so never reads the noindex, which is why a URL linked from elsewhere can still be listed bare. Noted inline, with the switch to make (relax robots.txt to Allow) if studio is ever linked publicly. The meta is injected into generated markup, so the step greps for it afterwards and fails rather than publishing an indexable page if a future index.html stops matching.
📝 WalkthroughWalkthroughThe studio deployment workflow now blocks crawler access, adds ChangesDeployment protection
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔍 Cloudflare PR preview
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| # to "Allow: /" so the noindex is the rule that applies. | ||
| # Appended to the end of <head> rather than the start, so the charset | ||
| # declaration keeps its place as the first thing in the document. | ||
| sed -i '0,/<\/head>/s// <meta name="robots" content="noindex, nofollow" \/>\n <\/head>/' index.html |
There was a problem hiding this comment.
Minor/informational, low confidence: vite-plugin-pwa's workbox globPatterns includes html (see vite.config.ts around the workbox: block), so the service-worker precache manifest's revision hash for index.html is computed at build time from the pre-sed content. This sed runs after that build step, so the deployed index.html (with the noindex meta tag) will differ from what the manifest hashed.
In practice this is likely harmless — Workbox precaching fetches the resource by URL at install time rather than validating byte-for-byte against the revision hash, so the service worker should still end up caching the actual (modified) file. Flagging only because it's a subtle build-step-ordering interaction worth being aware of if precache behavior ever looks stale for this deployment specifically.
Code reviewBugs: None found. The Security: None applicable — static config/markup changes only, no user input, no new external hosts, no secrets touched. Performance: None of note. Robots.txt/meta tag are trivially small relative to the existing 900 MB Pages size guard. Quality: One low-confidence, informational note posted inline ( CLAUDE.md: No violations. The added comments explain why (crawler behavior, injection ordering, fail-loudly rationale) rather than restating what, consistent with this repo's convention and the surrounding file's existing style. Overall this is a well-scoped, carefully validated change (the PR body documents manual testing against the real built |
🔍 GitHub Pages PR preview
Note GitHub Pages built this preview successfully, but its serving edge returned HTTP 403 when checked. The links may still be propagating. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/studio-deploy.yml:
- Around line 145-149: Strengthen the validation around the robots-tag injection
in the deployment workflow: record the number of </head> markers before and
after the sed edit and fail unless the count is unchanged, then validate that
exactly one meta element with name="robots" and content="noindex, nofollow"
exists inside the head rather than matching arbitrary text. Keep the deployment
blocked whenever the expected document structure or exact tag is absent.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: fe27d2ec-6401-489f-b806-d0042045054d
📒 Files selected for processing (1)
.github/workflows/studio-deploy.yml
| sed -i '0,/<\/head>/s// <meta name="robots" content="noindex, nofollow" \/>\n <\/head>/' index.html | ||
| # The tag is injected into generated markup, so fail loudly if a | ||
| # future index.html no longer matches rather than publishing an | ||
| # indexable page. | ||
| grep -q 'name="robots"' index.html |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Validate the exact tag and document structure before publishing.
At Line 149, grep -q 'name="robots"' only checks a substring. It can pass on an existing tag, a comment, or text outside <head>, even when Line 145 did not add the required noindex, nofollow tag. Line 145 also uses the first textual </head> occurrence instead of validating the parsed document. This can publish an indexable studio after a template change. Validate exactly one required meta element inside <head>, and compare the </head> count before and after the edit.
Suggested minimum guard
+ head_count_before=$(grep -oF '</head>' index.html | wc -l || true)
+ if [ "$head_count_before" -eq 0 ]; then
+ echo "::error::index.html has no </head> element."
+ exit 1
+ fi
sed -i '0,/<\/head>/s// <meta name="robots" content="noindex, nofollow" \/>\n <\/head>/' index.html
- grep -q 'name="robots"' index.html
+ head_count_after=$(grep -oF '</head>' index.html | wc -l || true)
+ robots_count=$(grep -oF '<meta name="robots" content="noindex, nofollow" />' index.html | wc -l || true)
+ if [ "$head_count_after" -ne "$head_count_before" ] || [ "$robots_count" -ne 1 ]; then
+ echo "::error::index.html failed robots meta validation."
+ exit 1
+ fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/studio-deploy.yml around lines 145 - 149, Strengthen the
validation around the robots-tag injection in the deployment workflow: record
the number of </head> markers before and after the sed edit and fail unless the
count is unchanged, then validate that exactly one meta element with
name="robots" and content="noindex, nofollow" exists inside the head rather than
matching arbitrary text. Keep the deployment blocked whenever the expected
document structure or exact tag is absent.
Summary
The gated instance is meant to be reached by invitation, and nothing on it is useful to a crawler — every route renders the sign-in card until a session exists.
The deploy step now writes a
robots.txtdisallowing everything, and appends<meta name="robots" content="noindex, nofollow">to<head>.Why both, and the caveat
They deliberately do not compose. A crawler that honours the
Disallownever fetches the page, so it never reads thenoindex— which is why a URL linked from elsewhere can still be listed (bare, no title). TheDisallowstops crawling; the meta covers crawlers that ignore robots.txt.For this deployment that is the right trade: nothing links to
studio.geolibre.app, so nothing should discover it in the first place. The inline comment records the switch to make if that ever changes — relaxrobots.txttoAllow: /so thenoindexbecomes the rule that applies.Notes
<head>so the charset declaration keeps its place as the first thing in the document.index.htmlstops matching the pattern.Validation
Ran the exact
sedagainst the real builtapps/geolibre-desktop/dist/index.html: injected once,</head>count unchanged, charset still first, and the document still parses as<html> <head> </head> <body> </body> </html>.check-yamland the fullnpm buildpass.Summary by CodeRabbit
robots.txtandnoindex, nofollowmetadata.