Generate federal spending pages from Public Accounts data - #275
Open
xrendan wants to merge 20 commits into
Open
Conversation
…d prose pipelines
Replace the hardcoded federal spending pages with [year]/[department] routes driven by data/federal, mirroring the provincial pattern: year toggle, CPI real/nominal toggle, line-item tables with CSV export, methodology page, and server-side French label substitution. Legacy department URLs redirect to the default year.
Dokploy Preview Deployment
|
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
canadaspends | 643249f | Jul 10 2026, 10:28 PM |
Restore list content dropped in extraction, fix figures that contradicted the generated Vol II data (Finance transfer share, ESDC individual-transfer share, DND entity framing), replace chart-conflicting literals with interpolated placeholders, and bring French in line. Each file carries an HTML-comment audit of figure decisions.
The exporter serialized subtotals on every parent node, but the shared chart sums trees with D3 hierarchy().sum(), which adds a node's own amount plus its descendants' — compounding to ~3.7x at the root (~$2.0T displayed instead of $548B) and unbalancing the two sides, since spending and revenue trees inflate by different depth-dependent factors. Strip parent amounts at serialization (matching the provincial contract) and harden the four sum accessors to ignore nodes with children, which is a no-op for provincial and budget data.
Split the inflation state: the nominal/real mode lives in a single app-level provider synced to localStorage, so the choice survives navigation between years, departments, and full reloads; each page provides its year's CPI multiplier. Scaling now covers the whole page — Sankey and miniSankey trees, entity bar lists, and line-item tables (including CSV export, with real-dollar column labels) — while percentages stay unscaled.
…ents Vote nodes now carry their appropriation description (Operating expenditures, Capital expenditures, ...) instead of bare vote numbers, with the number preserved on the node and same-named siblings disambiguated. Votes fan out to allotment children only when the allotments add information beyond generic shells, keeping sums exact and amounts leaf-only. Spec: docs/specs/mini-sankey-legibility.md.
Prose markdown now scripts the department page via {{section:...}}
tokens — the 14 hand-written FY2024 pages interleave narrative with
stats/charts in their original production order, and unmarked prose
keeps the default layout. The miniSankey switches from vote/allotment
trees to the official standard-object breakdown (Personnel, services,
acquisitions, with negative revenue leaves) sourced from the bilingual
open.canada.ca dmac-meso CSVs for all 12 years, reconciled against
allotment totals with documented deltas. Vote detail remains in the
line-item tables. Also completes the French UI catalog (539/539).
Spec: docs/specs/department-page-composition.md.
The 2020-2021 editions carry SSC under a gn-dg noise code that was mapped to the Governor General portfolio, inflating it to ~$3B; the standard-object table confirms the true GG figure is ~$25M. An organization override now routes SSC to Public Services and Procurement in all years, with a regression test, corrected prose for the four affected department-years, and full French coverage maintained.
Headline figures now equal the Consolidated Statement of Operations exactly: total expenses include net actuarial losses (FY2024 becomes $521.4B, deficit $61.9B — the old site's $513.9B excluded them, now reason-coded in the parity report), the deficit field is the published Annual operating deficit line with a Surplus stat-card branch for surplus years, net actuarial losses become a sign-normalized cost category under Obligations, and a single visible accounting-and- consolidation-adjustments leaf ties the thematic spending tree to the headline to the dollar, with per-year identity validations enforced at export. Spec: docs/specs/consolidated-statement-alignment.md.
The overview's ministry list and thematic Sankey are now sourced from Vol I Table 3.6 segment accruals (portfolios mapped/allocated to slugs, sub-ministry detail prorated by Vol II shares), so every number on the page — stat cards, chart, and list including the appended statement rows (net actuarial losses, provisions, and pre-2017 consolidated Crown corporations) — sums to the published total in all 12 years, and the accounting-basis-adjustments leaf is gone. Vintage-edition figures are scaled to the restated statement totals (residual ≤0.7%, guarded at 1%, factors documented); department pages stay on Vol II appropriations with the basis banner marking the seam. Spec: docs/specs/main-page-accrual-basis.md.
Department pages move entirely to the net standard-object presentation: stat card, entity list, federal share, and historical share all equal the chart's figures exactly (306 department-years bulk-verified), the votes/available/lapsed table is removed, and the line-items table is transfers-only. Each organization's transfer-payments object fans out into named transfer programs (scaled to the object total, top-N + Other) where the org unambiguously holds the portfolio's transfers. Allotment data remains internal-only (theme allocation weights and QA), enumerated in NOTES.md. Prose, methodology, parity report, and both French catalogs updated. Spec: docs/specs/drop-authorities-transfer-programs.md.
New pb workforce scraper pulls TBS Population of the Federal Public
Service figures (HTML 2015-2025, open-data CSV backfill for 2014, with
cross-checked overlap) into a committed reference file; the exporter
merges it with the standard-object Personnel totals into
data/federal/{year}/workforce.json for all 12 years. The overview page
gains a per-year workforce strip (headcount, personnel spending,
average personnel cost per employee, headcount trend chart) above the
existing latest-year demographics section, EN+FR.
Spec: docs/specs/workforce-headcount-salary.md.
pb workforce now also scrapes the TBS dimension tables: age bands and employment tenure for all 12 years (summing exactly to headcount, including the published Unknown band) and salary ranges for 2017+ (the employment-equity population subset TBS publishes, captioned as such — the same source the old hardcoded chart used). The demographics section now renders the SELECTED year's data with official French labels, degrading gracefully where a dimension isn't published; the hardcoded FederalWorkforce/TenureChart/SalaryDistributionChart components and 417 lines of embedded data are deleted. Spec: docs/specs/workforce-headcount-salary.md (extension).
8 confirmed findings (stale prose figures from the pre-accrual vintage, an unescaped HTML sink in prose rendering, undisclosed magnitude of the accrual/standard-object seam, and minor doc/i18n/lint gaps); 24 failed falsification attempts confirming the numerical pipeline ties to source.
Adds a figure-consistency lint over reviewed prose (wired into rake) and fixes the 80 stale literals it caught — pre-accrual totals, COVID- era figures, drifted entity values and shares — with a justified allowlist for source-quoted figures. Hardens renderProse (HTML escaping + https-only links, attack-fixture verified), discloses the accrual seam on department pages via a new accrualSpending field and note, flips the prose gate to opt-in, fixes the governor-general French label collision, stale reconciliation wording, NBSP artifacts, hreflang/title gaps, and cross-environment float-format non-determinism.
…ts-pipeline # Conflicts: # src/locales/en.po # src/locales/fr.po
Dokploy Preview Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the hardcoded federal spending pages with statically generated pages built from JSON derived from the Public Accounts of Canada, covering all fiscal years 2014–2025 with a year toggle — matching the provincial/municipal architecture. Spec:
docs/specs/federal-public-accounts-pipeline.md(plus follow-on specs indocs/specs/).etl/federal/, Ruby, copied from the public_accounts repo):pb exportgeneratesdata/federal/{year}/(summary, thematic sankey, reconciliation, per-department JSON) with hard validation — zero unmapped lines, parent=children within $1M, per-yearspending − revenue == published deficitidentity, byte-deterministic output. Pluspb translate/pb prosepipelines. 215 tests./federal/spending/[year]and[year]/[department]routes for every ministry (generateStaticParams, FY2025 default), year selector, page-wide CPI real/nominal toggle persisted across navigation, searchable transfer-payment tables with CSV export, methodology page, legacy URL redirects. The 14 hardcoded department folders, inline 800-line Sankey tree,useDepartments, andsankeyDepartmentMappingsare deleted.{{section:...}}tokens script the page so narrative interleaves with stats/charts in the original production order; figures only via interpolated placeholders; gated onreviewed: truefrontmatter (FY2024+FY2025 reviewed by @xrendan; 2014–2023 render stats-only pending review).regional-economic-developmentportfolio wherever the source data separates them (not 2016–2018, whose editions lack the granularity).The headline now equals the published Consolidated Statement of Operations exactly, and the spending Sankey sums to it to the dollar:
reconciliation.jsonand the methodology page document the Vol II bridge.docs/specs/federal-parity-report-2024.md(zero unexplained rows) and explained on/federal/spending/methodology.Data-quality corrections found along the way
Adversarial review
An adversarial review (
docs/specs/adversarial-review.md, findings indocs/specs/adversarial-review-findings.md) attempted 24 falsifications of the numerical pipeline — all failed (figures re-derived from raw CSVs and checked against canada.ca). Its 8 confirmed findings (stale prose figures from superseded bases, an unescaped HTML sink, undisclosed accrual/expenditure seam magnitude, minor i18n/doc gaps) are fixed, including a permanent figure-consistency lint over reviewed prose wired into the test suite.Testing
bundle exec rakeinetl/federal: 215 runs, 0 failures; export byte-deterministic; per-year statement-identity validations.pnpm build: passes (~14k prerendered static pages); tsc/eslint clean on touched files; French catalog 0 missing.Generated with Claude Code