Architecture Redesign — Master Tracking Issue

[EBukin]

devPTIpack Architecture Redesign

⚠️ Important caveat: The architecture documents referenced below (in .github/docs/) are proposed designs, not final mandates. They represent one well-researched path forward, but developers implementing these changes should treat them as guidance. Deviations are expected and welcome — discuss in the relevant issue or PR before taking a different direction. The truth is in the code and the tests, not in these documents.

Objective

Revise and refactor devPTIpack from a dual-generation golem package (with ~2000+ lines of dead legacy code, minimal documentation, and ~14% test coverage) into a clean, well-documented, thoroughly tested modern R package with a clear public API and extensible data ingestion.

Architecture Documents

All design documents live in .github/docs/:

Document	Content
arch-00-overview.md	Package summary, architecture layers, call chains, module generations
arch-01-cleanup.md	Function-by-function audit, 6 removal batches, permanent function registry
arch-02-docs.md	Documentation standards and implementation order
arch-02.01-testing-calc-pipeline.md	Detailed test specs for the calculation pipeline (~71 test cases)
arch-03-testing.md	Three-tier testing framework, per-function test map
arch-04-workspace.md	Workspace restructuring, vignettes plan, pkgdown structure
arch-05-hex-ingestion.md	New hex data ingestion pipeline (5 functions)

Sub-Issues (ordered by dependency)

• Architecture cleanup: remove legacy dead code #8 — Legacy Code Cleanup (6 batches, ~40 dead functions, ~2000 lines)
• Testing framework & pipeline orchestrator #10 — Testing Framework & Pipeline Orchestrator (3 tiers, ~160+ test cases)
• Roxygen2 documentation for all permanent functions #11 — Roxygen2 Documentation (~95 functions, priority-ordered)
• Workspace restructuring, vignettes & pkgdown site #12 — Workspace Restructuring, Vignettes & pkgdown (remove dev/, add vignettes, deploy site)
• Hexagonal data ingestion pipeline #13 — Hexagonal Data Ingestion Pipeline (5 new exported functions)

Relationship to Pre-Existing Issues

Issue	Relationship
#2 (debug artifacts)	Subsumed by #8 Batch 1
#3 (deprecate v1 modules)	Subsumed by #8 Batches 1–5
#4 (remove unused exports)	Subsumed by #8 Batch 1
#5 (H3 global database)	Upstream design; #13 implements the R package interface
#6 (unit test coverage)	Subsumed by #10
#7 (validation refactor)	Complementary; can proceed independently
#1 (data interfaces)	Long-term vision; #13 addresses the hex ingestion portion

Phasing & Dependencies

#8 Cleanup ──────┐
                 ├──► #11 Documentation ──► #12 Workspace/Vignettes/pkgdown
#10 Testing ─────┘
                        #13 Hex Ingestion (independent, can start anytime)

• Architecture cleanup: remove legacy dead code #8 and Testing framework & pipeline orchestrator #10 can proceed in parallel (tests verify modern code; cleanup removes legacy code)
• Roxygen2 documentation for all permanent functions #11 benefits from cleanup being done first (fewer functions to document)
• Workspace restructuring, vignettes & pkgdown site #12 requires docs + cleanup to be substantially complete
• Hexagonal data ingestion pipeline #13 is independent and can be developed on a feature branch at any time

End-State Goals

After all sub-issues are resolved:

1. Only modern pipeline remains — entry points: launch_pti(), launch_pti_onepage(), create_new_pti()
2. All ~95 permanent functions have complete roxygen2 documentation
3. Test coverage > 80% — Tier 1 (pure functions) + Tier 2 (module servers) automated in CI
4. Package website deployed on GitHub Pages with grouped vignettes
5. Hex ingestion pipeline provides developer-facing pre-deployment data prep from custom shapefiles + H3 API
6. R CMD check passes with 0 warnings, 0 errors, 0 notes

[EBukin]

@koito19960406 Here is the roadmap of the redesign. Please feel free to integrate relevant AI skills and claude.md type of instructions or agents into the project that you are using. I am happy to start using them too. Let us discuss these if needed. Let me konw if anything should be done differently.

[koito19960406]

Phase 2.5 §12 bug-fix sprint — closed 2026-05-07

The §12 bug-fix sprint (concurrent with the redesign tracks) wrapped up on 2026-05-07. All 14 §12 bugs from the bug ledger now have a contract-asserting test pinned in place of the original bug-asserting pin. The sprint did not extend the redesign roadmap — the master tracker stays open until Phase 4 (#12) and Phase 5 (#13) also land.

PRs landed (14 across 2026-05-06 → 2026-05-07)

§12 row	PR	Function	Class
#1	#56	complete_pti_labels	silent data corruption (popup suffix dropped)
#5	#57	get_adm_levels	silent corruption (lex sort vs numeric)
#6	#59	get_scores_data	silent corruption (1-row group → NA, not 0)
#7 + new row #14	#60	expand_adm_levels	>1-slot silent NULL + boundary-regex follow-up
#10	#61	get_vars_un_avbil	asymmetric availability
#2	#62	check_existing_groups	empty-input vctrs error
—	#58	(PLAN scaffold; no §12 fix)	docs-only sprint kickoff
#4	#64	validate_read_shp	empty-pattern str_detect
#8	#65	fct_internal_wt_to_exp	empty list → left_join error
#9	#66	get_var_choices	empty indicators → NULL-attr error
#11	#67	mod_fltr_sel_var2_srv	multi-var-pillar map_lgl length-N
#13	#68	mod_dwnld_file_server + launch_pti()	broken downloads + path-default API redesign (Arch-2 auto-materialize)
#3	#69	filter_admin_levels	name-vs-value asymmetry
#12	#70	gg_admin_list	undefined default (zam_bounds_simple not found)

Plus PR #71 (this comment was posted alongside it) — a pure-docs reconciliation correcting a PR #12 → PR #70 typo in PLAN.md §1.

Suite delta

testthat::test_local() summary-reporter PASS:

• Pre-sprint: 682 PASS / 0 fail / 1 skip
• Post-sprint: 710 PASS / 0 fail / 1 skip

Net +28 expectations, with most of the growth coming from two new defensive-input test files: test-mod-dwnld-file.R (+13) and test-gg-admin-list.R (+8). The remaining +7 are pin flips from expect_error(bug) to multi-assertion contract pins.

Out-of-scope items still tracked

Two cosmetic / refactor items were explicitly left out of scope and remain documented as follow-ups:

1. Orphaned " and ." connector text in the side-panel sentence when mtdtpdf_path is NULL. The connector logic in mod_map_dwnld_ui lives in three places (PTI / compare / explorer side panels) and an interim setdiff()-based hide attempt didn't reach all three cleanly. PR fix(download): launch_pti auto-materializes downloads on NULL paths (Arch-2) #68 narrative documents this. Candidate for a UI-cleanup follow-up; flagging for Eduard's Phase 4 vignette polish in case it folds in naturally.
2. Runtime testthat::test_that() refactor in validate_* helpers. The §12 sprint addressed the symptoms tracked in row Remove or deprecate unused exported functions #4 (the empty-pattern str_detect error inside validate_read_shp), but the underlying arch-01 refactor target — converting the runtime-testthat::test_that machinery into ordinary validation calls — is still pending. PR fix(validation): validate_read_shp guards label construction on perfect shapefiles #64 narrative explains why this was kept surgical. Tracked under Refactor validation into step-by-step, user-facing data checks with CSV support #7.

Next track

Phase 4 (#12, vignettes & pkgdown) is owned by @EBukin — a hand-off comment lands on that issue alongside this one. Phase 5 (#13, hex ingestion) remains independent and can start anytime on a parallel branch.

The koichi-arch-redesign integration branch is the right base for both Phase 4 and any further surgical fixes.