Refactor validation into step-by-step, user-facing data checks with CSV support

[koito19960406]

Problem

Current validation (fct_validate_metadata.R, validators.R) has issues:

• Uses testthat (a dev testing framework) for runtime validation
• Outputs go to R console via rlang::inform/warn/abort, not to the Shiny UI
• All-or-nothing: either passes or fails with no step-by-step guidance
• No CSV support (only Excel .xlsx and RDS shapefiles)
• No ID mismatch detection between user data and shapefiles
• No preview of matched vs unmatched records
• No data type/range validation for indicator values

Context: metadata column burden

The current metadata sheet requires 14 columns. Analysis shows only 1 is truly user-provided (var_code = column names). The other 13 can be auto-generated with sensible defaults:

Minimum viable CSV: just adminXPcod + numeric indicator columns. The system should auto-generate the full metadata structure:

• var_name = var_code (column name)
• var_order = row number
• spatial_level = inferred from which adminXPcod is present
• pillar_group = 1, pillar_name = "Indicators"
• All fltr_* = FALSE, legend_revert_colours = FALSE

Vision

When users upload their own data (CSV or Excel) to supplement the global H3 database (#5), they need clear, step-by-step feedback:

Proposed validation steps

1. File format check: readable? required columns present? Auto-detect adminXPcod column, identify indicator columns (numeric).
2. ID matching: auto-detected ID column vs shapefile IDs. Report: X of Y IDs matched, Z unmatched (list them), W shapefile regions with no data (list them).
3. Data type validation: indicator columns numeric? NA percentage? Per-column summary (type, NA%, range).
4. Geometry compatibility (custom shapefile only): valid geometries? Proper admin hierarchy? Skip for off-the-shelf global shapes.
5. Cross-reference: admin levels in data match shapefile layers?
6. Dry-run PTI: auto-generate metadata, run one weight combo, confirm output.

Auto-generation function

New function generate_metadata_from_csv(df, shapefile):

• Detects adminXPcod columns -> determines spatial_level
• Remaining numeric columns -> var_code entries
• Builds the full 14-column metadata tibble with defaults
• Returns the list structure that mod_calc_pti2 expects

Architecture

• New R/fct_validate_user_data.R: pure functions, each returning list(status = "pass"|"warn"|"fail", message = "...", details = tibble(...))
• New R/fct_generate_metadata.R: auto-generate metadata from bare CSV
• New R/mod_validate_upload.R: Shiny module rendering checklist UI
• Refactor existing validators to return structured results (backward-compatible via return_structured parameter)
• New tests/testthat/test-validate-user-data.R

Key principle

Validation is decoupled from ingestion. Validate first, show all results, ingest only when checks pass (or user acknowledges warnings).

Acceptance criteria

• All 6 validation steps implemented as pure functions
• CSV upload supported alongside Excel
• Auto-generate metadata from bare CSV (only adminXPcod + numeric cols needed)
• Shiny UI shows step-by-step progress with pass/warn/fail per step
• ID mismatch report shows specific unmatched IDs
• Existing validate_metadata/validate_read_shp still work (backward compat)
• Unit tests for each validation step
• Integration test with sample Ukraine data

[koito19960406]

Status update from §12 sprint

The Phase 2.5 §12 bug-fix sprint (closed 2026-05-07) touched two of the symptoms this issue calls out, but the underlying architectural refactor remains pending.

What landed:

• PR fix(validation): validate_read_shp guards label construction on perfect shapefiles #64 fixed §12 row Remove or deprecate unused exported functions #4 — validate_read_shp no longer hits a str_detect(names(x), character(0)) vctrs size error on a "perfect" shapefile (every admin{N}Pcod field has a matching admin{N}_* slot). Surgical guard around the diagnostic-label construction; the broader behaviour is unchanged.
• The sprint also touched validate_metadata / validate_read_shp / validate_geometries indirectly via roxygen hygiene during Phase 3 (PR Phase 3 Batch 3: roxygen2 docs for data I/O & validation (11 fns) #49).

What's still pending under this issue:

The arch-01 / PR #64 narrative explicitly notes the bigger refactor target is out of scope and stays here:

Out of scope: the broader arch-01 refactor target — converting the runtime testthat::test_that machinery into ordinary validation calls — bigger scope, would obsolete capture_validator() entirely; left for a future PR.

That maps directly onto the first bullet of this issue's "Problem" section: "Uses testthat (a dev testing framework) for runtime validation." The acceptance criteria here (CSV support, auto-generated metadata from bare CSV, step-by-step UI feedback, structured pass/warn/fail results) are all still open.

Test pin name update: the existing pin in tests/testthat/test-validators.R was renamed by PR #64 from "validate_read_shp: round-trips through an .rds path (PINNED BUG)" to "validate_read_shp: perfect shapefile passes round-trip". The PR's narrative documents that the pin uses capture_validator() (which mocks testthat::test_that to a no-op) so the inner blocks never execute — a more rigorous pin would require the very test_that → ordinary validation refactor this issue is about.

Leaving #7 open for the architectural work.