Data Preparation Pipeline Redesign — Master Tracking Issue

[EBukin]

Overview

Redesign the devPTIpack data preparation pipeline and scaffolding workflow to be simpler, more transparent, and AI/human-comprehensible. A deployer starting from scratch — with no existing country data — should be able to scaffold a working PTI app, load default boundaries, run the full data pipeline, review a data quality report, and deploy to Posit Connect with minimal manual editing and no invisible steps.

This is a sibling of #9 (Architecture Redesign), which covers internal package plumbing. This issue covers the deployer-facing workflow: the template files a user interacts with when building a new country PTI app.

Goals

1. Zero-friction project setup: create_new_pti() injects the app name everywhere, emits a clear next-steps checklist, and optionally opens the project.
2. One-line boundary loading: get_country_shapes("ISO3") replaces 40 lines of manual GeoJSON renaming and spatial joins.
3. Hex data on by default: Step 4 runs as part of the normal pipeline — not commented out.
4. No manual Excel editing: The var_overrides block in Step 5 is the single place deployers control what goes into the PTI and Explorer.
5. Data quality website: The full Quarto pipeline renders a navigable website into docs/ that shows boundary maps, data summaries, and the per-variable metadata report — ready to publish to GitHub Pages.
6. Documentation updated in sync: All website tutorials (Steps 0–6) updated to match the new template files.
7. AI-ready from day one: Every scaffolded project ships CLAUDE.md and an agent skill so an AI assistant has full pipeline context immediately.

Sub-issues

Setup

• A — create_new_pti(): inject app_name + compact next-steps CLI with open-project prompt — A: create_new_pti() — inject app_name into scaffolded files + compact next-steps CLI with open-project prompt #152
• B — Template: CHECKLIST.md with pre-run / pre-deploy task lists — B: Template — add CHECKLIST.md with pre-run / pre-deploy task lists #150

Helpers (prerequisite for all step changes)

• C — New helper functions: pti_plot_boundaries(), pti_plot_histogram(), pti_summary_table() — C: New helper functions — pti_plot_boundaries(), pti_plot_histogram(), pti_summary_table() #151

Pipeline steps

• D — 02a-user-zonal-stats.qmd: document output contract and explicit feed to Step 3 — E: Rename 03-metadata.qmd → 03-user-data.qmd; data contract header + pti_patch_admin_sheet() helper #153
• E — Rename 03-metadata.qmd → 03-user-data.qmd; data contract header + pti_patch_admin_sheet() — F: 04-hex-data.qmd — remove eval: false guards; render hex variable summary table #154
• F — 04-hex-data.qmd: remove eval: false guards; render hex variable summary table — D: 02a-user-zonal-stats.qmd — document output contract and explicit feed into Step 3 #155
• G — 05-compile.qmd: auto-detect metadata-hex.xlsx; var_overrides block; bird's-eye reactable summary — I: Template — scaffold _quarto.yml Quarto website with sidebar nav (step pages + Metadata Report + App iframe) #156
• H — 05-compile-report.qmd (new): boundary overview + per-variable HTML report + downloadable PDF — H: 05-compile-report.qmd (new) — boundary overview + per-variable HTML report + downloadable PDF #157

Website and deployment

• I — Template: scaffold _quarto.yml website with sidebar nav — G: 05-compile.qmd — auto-detect metadata-hex.xlsx; var_overrides block; bird's-eye reactable summary #158
• J — 00-master.R: add APP_URL param; render full Quarto site to docs/; un-comment Step 04 — J: 00-master.R — add APP_URL param; render full Quarto site to docs/; un-comment Step 04 #159
• K — 06-deploy.R: add GitHub Pages deployment instructions for docs/ — K: 06-deploy.R — add GitHub Pages deployment instructions for docs/ #160

App content

• L — app.R + landing-page.md: update contact to Nga Thi Viet Nguyen; generic {{APP_NAME}} template — Doc: Update all website tutorial vignettes (Steps 0–6) to match redesigned pipeline and new helper functions #161

Documentation (independent — flag for parallel work)

• Doc — Update all website tutorials (Steps 0–6 vignettes) to match the redesigned template files and new helper functions — L: app.R + landing-page.md — update contact; generic template with {{APP_NAME}}; wire HTML metadata report #162

AI tooling

• P — Bundle CLAUDE.md + agent skill .agents/skills/pti-data-prep/SKILL.md with scaffolded project — P: Bundle AI skills and CLAUDE.md with the scaffolded PTI project template #165

Maintenance rule

Any PR that changes the pipeline interface (new helper functions, renamed files, new steps, changed function signatures) must also update CLAUDE.md and the skill file (#165). This is a hard acceptance criterion for all sub-issues above.

Infrastructure dependencies

This issue depends on the WB Boundaries infrastructure being planned (not necessarily complete):

• Inf-1: ETL vignette — convert WB Official Boundaries to per-admin-level parquet and upload to WB Data Catalog #145 (Inf-1) — ETL parquets
• Inf-2: inst/wb_shapes_registry.yaml — bundled catalog of available countries, admin levels, and parquet base URL #146 (Inf-2) — shapes registry YAML
• Inf-3: list_country_shapes() — return tibble of available countries and admin levels from bundled registry #147 (Inf-3) — list_country_shapes()
• Inf-4: get_country_shapes() — fetch WB Official Boundaries parquet and return package-ready named sf list with hex grid #148 (Inf-4) — get_country_shapes()

Sub-issues C (#151), D (#153), I (#158), L (#161) can start immediately. Sub-issues A (#152), B (#150) require #143. The full pipeline (E–H, J–K) requires C (#151). Issue P (#165) requires A (#152) and a stable pipeline (D–J).

Related standalone issues

• N: End-to-end automated pipeline test — two countries (Rwanda + Ethiopia), fresh environment #163 — API coordination: pti_patch_admin_sheet() vs generate_metadata_from_csv() (Refactor validation into step-by-step, user-facing data checks with CSV support #7)
• M: API coordination — pti_patch_admin_sheet() vs generate_metadata_from_csv() (#7) #164 — End-to-end automated pipeline test: Rwanda + Ethiopia, fresh environment

Definition of done

• source("00-master.R") in a freshly scaffolded Rwanda project produces: app-data/shapes.rds, app-data/metadata.xlsx, app-data/pti-metadata.html, docs/index.html
• shiny::runApp("app.R") launches without error
• All website tutorial vignettes (Steps 0–6) reflect the new file names, helper functions, and workflow
• A freshly scaffolded project contains CLAUDE.md and .agents/skills/pti-data-prep/SKILL.md
• End-to-end test (M: API coordination — pti_patch_admin_sheet() vs generate_metadata_from_csv() (#7) #164) passes for Rwanda and Ethiopia in CI