🎉 Add harmonized PIP–WID income distributions and inequality trends - #6764
Draft
paarriagadap wants to merge 23 commits into
Draft
🎉 Add harmonized PIP–WID income distributions and inequality trends#6764paarriagadap wants to merge 23 commits into
paarriagadap wants to merge 23 commits into
Conversation
Ports the between/within-country inequality methodology from github.com/JoeHasell/prague-pip-wid into ETL, extended from a single year to the full 1990-2024 panel, so it re-runs on every PIP/WID data update: - wid/2026-08-25/wid_population: WID adult (20+) and total population (new npopul snapshot), the demographic yardstick for the comparison. - poverty_inequality/2026-08-25/harmonized_income_distributions: PIP and WID full distributions on an identical 109-bin structure, three derived bridging series (income basis, top adjustment, mean-rescaled WID), and the between/within MLD decomposition per year and series, with the source project's conventions preserved. - poverty_inequality/2026-08-25/inequality_trends_by_reference_year: Gini and Generalized Entropy metrics computed from the same bins, and rising/falling/stable counts, population shares and average changes for every reference year vs the latest year. Grapher steps carry only the summary tables (decomposition and reference-year aggregates as "World", the per-country metrics panel). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Quick links (staging server):
Login: chart-diff: ✅No charts for review.data-diff: ❌ 1 changed · 2 new · 1 identical — full report+ garden/poverty_inequality/2026-08-25/harmonized_income_distributions (new dataset: 5 table(s), 24 column(s))
+ garden/poverty_inequality/2026-08-25/inequality_trends_by_reference_year (new dataset: 3 table(s), 24 column(s))
~ garden/wid/2026-06-18/world_inequality_database
+ population (new table, 2 column(s))= 1 compared dataset(s) turned out identical; details in the full report Automatically updated datasets matching excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included. Run locally with Edited: 2026-08-28 17:29:43 UTC |
Contributor
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 293758d8d5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The Stata script now exports the wid command's response unchanged (country/variable/percentile/year/value plus age and pop); the reshape to one column per variable and the descriptive names move to the meadow step. Values are unchanged: the new long snapshot pivots to exactly the previous wide one (63,341 country-years, zero differing values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cdfcc0a8b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A few consumption-to-income adjusted distributions are non-monotone along the percentile axis (kept upstream, as in the source project), so the Gini must sort each row's bins by income before accumulating; the stable sort leaves monotone rows bit-identical. Affects 979 of 59,080 ginis, only in the two adjusted PIP series, all upward, at most +0.00075. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a6a863cfc0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Nothing charts these indicators yet, and the only consumer is the presentation repo, which reads the garden tables from the catalog directly. Adding grapher steps now would mint variable IDs nothing points at. The garden steps move up to be top-level DAG entries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The consumption->income model fits a separate (alpha, beta) per percentile, and those coefficients are not monotone in p: p1 is the only percentile whose beta falls below 1 (0.776 against 1.09-1.22 above it) and it has the worst fit of the hundred (R^2 0.28). On a base under $1 a smaller exponent gives a larger answer, so at the consumption levels found at the bottom of poor countries p1 implies a higher income than p2 does, and the fitted profile inverts. 866 decreasing steps across 495 country-years, up to 31% of the preceding bin. Corrected with isotonic regression, population-weighted, which projects the profile back onto the monotone cone by replacing a violating run with its weighted average. That preserves each country's mean exactly: the 2023 between share moves +0.001pp for pip_income_basis and the top-adjusted series, and wid_after_tax_rescaled is bit-identical because it targets the top-adjusted PIP mean and that mean does not move. A cumulative maximum was tried first and rejected -- it broadcasts p1's least-reliable estimate across the whole run, shifting the mean +0.11% and the within-country MLD -0.011, against +0.00% and -0.0004 here. Behind ENFORCE_MONOTONE_INCOME_BASIS, with an assert that monotonicity is actually restored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two documentation gaps in the wid_population snapshot. The main WID extraction pulls 992j (equal-split adults) while this pulls population unit i, which looks inconsistent until you know that the population dimension only exists for distributed series. WID's codes dictionary says so, and the API confirms it: population(i j) on npopul reports "1 population category" and returns only npopul992i/999i, while the same request on aptinc returns both 992i and 992j. The j says how income is split among adults, not how many adults there are, so aptinc992j / npopul992i is the right pairing. The run instructions also said `stata-se`, which on a machine with several Stata versions under /Applications can resolve to an older one whose licence has expired -- that failure reads as "Stata is unavailable" when it is really the wrong binary. Pinned to the full path, with a note that -b sends output to the .log rather than the terminal. Verified by re-running the .do against WID: 126,606 rows, identical to the committed snapshot in every value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Population was a standalone one-table dataset with its own snapshot script, meadow and garden steps. It is now the `population` table of wid/2026-06-18/world_inequality_database, so any garden step reads it from the same dataset as the rest of WID. The extraction is consolidated too: the npopul block moves into wid_indices.do, and its CSV joins the FILES map in the snapshot script, so one .do and one .py cover all six WID snapshots. The block records why unit i is correct -- the population dimension exists only for distributed series, so aptinc992j / npopul992i is the right pairing. The .dvc moves to 2026-06-18 keeping its md5. Snapshot storage is content-addressed, so it resolves to the blob already uploaded: no Stata re-run and no re-upload, and the data is unchanged. Verified: the population table is identical to the standalone one it replaces (55,581 rows, 249 countries, zero differing values, origins preserved), and every 2023 between share is unchanged to four decimals. Note for #6755, which also edits these four files: the meadow, garden, garden .meta.yml and wid_indices.do changes here are additive, so the conflict should resolve mechanically. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Those comments explained why population sat in its own version folder with its own scripts. It no longer does -- one .do, one .py, one version -- so they were explaining something that is not true any more. The note in wid_indices.do stays, since the block really is a quick call next to the hours-long extractions above, but it now says why it is last (it can be re-run alone if only population changes) rather than implying it lives elsewhere. The .dvc keeps date_accessed 2026-08-25 while its five siblings say 2026-06-18. That asymmetry is real -- population was fetched later -- and recording the true access date is worth more than a tidy-looking folder. Verified by extracting the population block from the merged .do and running it: md5 c114e8de1bfa41cf7774a5a53997f960, identical to the committed snapshot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment coverage was thin in the functions doing the actual arithmetic: the 1000-to-109 bin aggregation, the per-percentile OLS fit, the welfare lookup, the top-tail graft, the rescale and the MLD decomposition all had docstrings but no labels inside, so following them meant re-deriving the intent from the operations. Sixteen labels added, marking what each block does rather than restating it. Two carry something a reader could not recover from the code: - The graft divides WID's top by its own anchor bin before rescaling, so only the SHAPE crosses over and any per-country factor in WID's levels cancels. That is why the PPP-vintage difference cannot reach this series. - merge_asof(direction="nearest") in the welfare lookup has no distance limit, so a panel year can take the basis of a survey 32 years away. Marked NOTE with a pointer to the review that flagged it. Comments only: all eight series' 2023 between shares are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same treatment as the harmonized step. changes_by_country and aggregate_by_reference_year had docstrings but nothing inside, and both do something a reader would otherwise have to infer: - changes_by_country merges on (country, series) and deliberately NOT on year, so every earlier year is paired with the single latest one. That cross join is what makes `year` mean "the year compared FROM" in the output, which is easy to misread as an ordinary join. - the rising/falling/stable split classifies on the RELATIVE change, so the same band applies to a Gini near 0.3 and a GE(2) in the tens. - population shares use the covered population as the denominator, which shrinks going further back, not a fixed world total. Comments only: 408 / 59,080 / 86,088 rows unchanged, Gini range and the rising+falling+stable identity intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The out-of-sample caveat said "no Sub-Saharan Africa or South Asia". The income-group breakdown is sharper: of 88 dual country-years across 19 countries, 59 are high-income, 24 upper-middle, 5 lower-middle and ZERO low-income. 14 of the 19 countries are European; Poland and Romania alone supply 30 of the 88 observations. That matters because of where the transform is applied -- consumption-based country-years, which are overwhelmingly the poor ones. So it is estimated on rich European income surveys and extrapolated onto low-income African and South Asian consumption surveys, the exact populations the sample excludes. It also explains why Benin 2015 was the worst monotonicity case: no observation in the sample resembles it. Recorded in the fit function's docstring, with the country lists and the release the counts come from (they will move on the next PIP update), and in the verification report next to the existing caveat. Docstring only: pip_income_basis between share still 60.321194%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It does not. PIP's methodology handbook states no preference rule: it calls the choice country-driven and, where it leans at all, leans the other way -- "the consumption approach is arguably more directly connected to economic welfare". And PIP's own consolidated complete_series keeps BOTH welfare types for the dual country-years, 88 consumption rows and 88 income rows, rather than picking one. The behaviour is right and unchanged; only the justification was wrong. Income wins the tie because it is the TARGET basis of this harmonisation: pip_income_basis exists to put every country on an income footing, so where the World Bank observes income directly there is nothing to estimate and the fitted transform is skipped. Running the model where the real thing is available would only add error. Both places carried the claim -- the function docstring and the pip_welfare_basis metadata. The docstring now says explicitly that this is not PIP's preference, so the wrong reason does not get re-introduced. Docstring and metadata only: pip_income_basis between share still 60.321194%, and the basis split is unchanged at 3,714 consumption / 2,271 income country-years. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third attempt at this sentence, so worth being explicit about what was wrong. It first claimed PIP prefers income where both exist: false. I then claimed PIP states no preference rule: also false -- I had read only chapter 2 of the handbook, where the rule does not appear. It is in chapter 5, section 5.4, and it is unambiguous: "Due to its closer connection to welfare, whenever both income and consumption estimates are available for a given reference year, consumption estimates are preferred." So PIP prefers CONSUMPTION, the opposite of what we do. The behaviour still stands and is now justified as a deliberate divergence: consumption is the better basis for measuring poverty, which is PIP's purpose, while income is the basis that makes these series comparable with WID, which is ours. For the 88 dual country-years this series therefore carries PIP's observed income where PIP's own headline carries its consumption. PIP publishes both, so nothing is overridden. Also recorded, from the same section: "interpolations are never done between consumption and income aggregates". PIP declines to bridge the two concepts; the consumption -> income model here is exactly that bridge, which is a reason to keep reporting its fit and its thin sample rather than burying them. The docstring now cites the URL, so the next reader checks the source instead of re-deriving the rule. Docs only: pip_income_basis between share still 60.321194%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For the 88 country-years where PIP publishes both welfare types, this step labelled them income, set adjusted=False and skipped the consumption -> income transform. But the thousand-bins values for those country-years are CONSUMPTION, so consumption was carried into pip_income_basis as though it were already income -- roughly 18% too low. Verified rather than inferred from the docs: the bins carry no welfare_type, so compare each country-year's bin mean against the mean PIP's percentiles imply for each type. All 88 match consumption, median relative distance 0.03%, against 18.06% for income. The two types differ by a median of 19.5% (min 2.3%) for these country-years, so none of the 88 is a close call. This is PIP's own documented rule -- consumption is preferred where both exist -- applied by the thousand-bins series. Effect on the 2023 between share: pip_income_basis 60.3212% -> 60.0838% (-0.24pp), top-adjusted 55.4066% -> 55.2043% (-0.20pp), and wid_after_tax_rescaled -0.07pp through the rescale target. Raw pip and every WID series are untouched. The direction is what the economics predicts: 157 panel country-years now get transformed, so the grand mean rises (22.310 -> 22.326) and within-country inequality rises (0.2860 -> 0.2880), income being more unequal than consumption -- the SHARE falls because within grows proportionally more. Added sanity_check_welfare_basis, which asserts the label matches the bins on every run. Confirmed it fails on the old behaviour, naming the country-years, rather than being an assert that cannot fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For a non-survey year, PIP's lined-up values were produced by choosing a welfare concept via its published decision tree, so following that tree recovers the concept those values carry instead of approximating it. The step now implements it: survey year -> the surveyed concept, consumption if it has both gap, one concept spanning both sides -> that concept (consumption first) otherwise -> the nearest survey's concept Nearest-survey agreed for 5,945 of 5,985 country-years and moves the 2023 decomposition by 0.0000pp, so this is inert for a single reference year. It matters for reading the panel year by year, which is the direction this is heading, and the 40 disagreements are exactly the interpolated years such a reading would rest on. They divide into two shapes. A stray survey no longer flips its neighbours: Ukraine is consumption throughout except a single 1999 income survey, which previously made 1998 and 2000 income too and now leaves only 1999 itself. And a dual survey now counts as an endpoint for either concept, so Haiti 2007-2011 interpolates with income, spanned by income at 2001 and 2012, even though 2012 is reported as consumption. Adjusted country-years 3,871 -> 3,861. The bins-vs-label guard still passes on all 88 testable survey years; it cannot test interpolated years, since those are the ones PIP publishes only one concept for, so the tree being right is now load-bearing. nearest_survey and latest_survey are kept as modes for comparison. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three deliberate departures from the source project have accumulated, so claims about reproducing it no longer hold: - ENFORCE_MONOTONE_INCOME_BASIS said "set False to reproduce the source project's unadjusted output". False alone no longer does that, because the welfare basis changed too. - latest_survey was "kept for validation runs". It cannot validate against the source project, which preferred income where both concepts exist -- the mislabelling this step now corrects. No mode does that any more, on purpose. - the module docstring opened with "ports the verified data pipeline", which reads as though the output should match. It should not. The docstring now lists the three departures up front, with pointers to where each lives, and says plainly that exact reproduction is neither possible from a switch nor intended, since two of the three fix errors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The build_welfare_basis docstring had been edited three times in place and ended up arguing both sides: it explained that consumption wins the tie because that is what the data holds and what PIP's rule says, then two paragraphs later claimed the step deliberately diverges from PIP by carrying income for those same 88 country-years. The second half was dead text from before the fix, and it contradicted the first. Rewritten whole rather than patched again. It now states the rule, the two cases PIP settles (survey years with both concepts; years between surveys), the evidence for each, the history of the bug, and what the guard can and cannot test. Also corrected survey_year_used, which is user-facing metadata. It was titled "Survey year the welfare basis is taken from" and described as the survey "whose welfare type is applied" -- true under nearest-survey, but not under the decision tree, where an interpolated year can take the concept of the surveys bracketing it instead. It is now titled as what it is, the nearest survey year, with description_key spelling out when it does and does not identify where the concept came from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docstring read the rule too broadly. "Interpolations are never done between consumption and income aggregates" forbids drawing a line from a consumption point to an income point -- one calculation. It is not a refusal to change concept over time: the decision tree's extrapolation branch takes whichever concept its anchor survey carries, so a country's lined-up series does switch concepts between periods. 29 of the 171 countries here do, which is exactly why the basis is assigned per country-year rather than once per country -- a point the overbroad reading obscured. The inference built on it was also weaker than stated. What PIP never does is CONVERT one concept into the other: it selects between measured series, it does not estimate an income distribution from a consumption one. That is what this step's model does, so it sits outside PIP's methodology rather than extending it -- a better reason to keep its fit and its thin sample visible than the misread rule was. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both quotes were taken from a fetch summary rather than the page. Verified verbatim against the source, and now cited by section so the next reader can check rather than trust: - §5.2 "Interpolations are never done between consumption and income aggregates" -- and the docstring now says where it sits, which is what makes the narrow reading obvious: that section is about interpolating a survey MEAN between two surveys using national-accounts growth. - §5.4 "consumption estimates are preferred" where both exist. The summary had also dropped a sentence from §5.4 that matters here: "when both kinds of poverty estimates are available for the same years (but not for a particular reference year), interpolations and extrapolation are made using the consumption estimates". That confirms the preference carries past the reference year, which is the assumption behind the tree checking consumption before income. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checked against PIP's data rather than reasoned from the handbook. PIP lines up non-survey years by distribution-neutral growth, so a lined-up year is a scaled copy of its anchor survey and has that survey's Gini exactly. That identifies which survey PIP used. For the 46 country-years where one concept sits before and the other after -- the branch the published decision tree leaves ambiguous, since its wording is written for years outside the survey range -- PIP uses the NEAREST survey, which is what this step already did: 32 of the 33 identifiable cases match, the Gini agreeing to 1e-9 or better. The one exception was a tie. Kyrgyzstan 1999 lies one year from a 1998 income survey and one from a 2000 consumption survey; np.argmin returned the earlier, giving income, while PIP's 1999 distribution matches the 2000 consumption shape to 4.7e-10. Ties now go to consumption, matching both the observed behaviour and the stated preference in handbook 5.4, and removing a dependence on argmin's arbitrary tie handling. Kyrgyzstan 1999 flips to consumption; adjusted country-years 3,861 -> 3,863; the 2023 between share moves 60.083800% -> 60.083848%. Three Seychelles years (2007-2009) remain unidentifiable: they share one Gini but it matches no survey's shape, so PIP appears not to have used plain scaling there. Our label is the closer of the two either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Ports the between/within-country inequality methodology from Joe's prague-pip-wid research pipeline into ETL — extended from a single year (2023) to the full 1990–2024 panel — so the whole exercise re-runs on every PIP/WID data update. Three new chains, versioned 2026-08-25:
1.
wid/2026-08-25/wid_population(snapshot → meadow → garden)WID's own population counts (
npopul: adults 20+ and total, all countries, all years), from a new small Stata fetch (wid_population.do, one API call — minutes, unlike the hours-long main WID extraction). WID demography is the yardstick the comparison's weighting convention requires; it also covers historical entities (USSR, Czechoslovakia, Yugoslavia, …) absent from the income fetch.2.
poverty_inequality/2026-08-25/harmonized_income_distributions(garden)income_distributions(~5.9M rows, garden-only): PIP and WID full income distributions on an identical 109-bin structure (PIP's 1,000 bins nest exactly — zero approximation error), 8 series: PIP, WID pre/post-tax × per-adult/per-capita, plus three derived bridging series — PIP on an income basis (per-percentile consumption→income regression fitted on PIP's 88 dual country-years), that series top-adjusted with WID's post-tax tail shape above P95, and WID post-tax rescaled to the adjusted-PIP country means.inequality_decomposition(+ per-country table): exact between/within MLD decomposition of global inequality per year × series over the 211-country common sample. Conventions preserved from the source project: country weights are WID demography matched to each series' basis (including for PIP), zeros floored at $0.01/day inside the MLD only, decomposition identity asserted at 1e-9.consumption_income_modelandpip_welfare_basisaudit tables. One deliberate deviation from the source project: the welfare basis (income vs consumption) is assigned per country-year from the nearest national survey rather than one static type per country — 27 countries switch basis across the panel.3.
poverty_inequality/2026-08-25/inequality_trends_by_reference_year(garden)"Is inequality rising everywhere?" computed on identical definitions for both sources: Gini + Generalized Entropy GE(0)/GE(1)/GE(2) from the 109-bin distributions per country-year, then — for every reference year vs the latest common year (2024) — counts of countries with rising/falling/stable inequality (±5% relative band, a tunable constant), population shares, and average changes (weighted/unweighted). The reference year is the time axis, so charts read as "compared to year, inequality has risen in N countries".
Garden only — no grapher steps. Nothing charts these indicators yet, and the only consumer today is the presentation repo, which reads the garden tables directly from the catalog. Adding grapher steps now would mint variable IDs nothing points at; they can be added when a chart or MDim actually needs them.
Who uses this
The immediate consumer is the deck these steps were ported from, in
JoeHasell/prague-pip-wid#1:
all 22 of its chart slides now read these datasets instead of computing their own
numbers, which is what the port was for. That PR is also a live check on the
output — it reproduces the source project's figures closely enough to attribute
every difference (see below), and it surfaced two bugs in the original.
Adversarial data review
Ran
/adversarial-data-reviewover both garden datasets and the calculations behind every deck figure. Report:ai/adversarial-review-harmonized_income_distributions-2026-08-26.md. 20 invariants, 18 pass.Validated against the source's own published aggregates — the strongest check available:
share_top_1Fixed — non-monotone income profiles from the consumption→income transform. The model fits a separate
(alpha, beta)per percentile, so a monotone consumption profile can come out non-monotone: in 495 country-years across 37 countries the bottom bins inverted (p1p2 below p0p1, up to p4p5), by a median of 8% and at worst 31% — concentrated where the fit is weakest (R² 0.28 at p1). Rawpiphas zero inversions, and the artifact propagated intopip_income_basis_top_adjusted.Corrected with isotonic regression (pool-adjacent-violators, population-weighted) — the least-squares projection onto the monotone cone, which replaces a violating run with its weighted average and so preserves each country's mean exactly. Behind
ENFORCE_MONOTONE_INCOME_BASIS(setFalseto reproduce the source project's output), with an assert that monotonicity is actually restored.Root cause is the coefficients, not the data: p1 is the only percentile with
betabelow 1 (0.776 vs 1.085–1.224 above) and has the worst fit (R² 0.280), so on a sub-$1 base it implies a higher income than p2 does.A cumulative maximum was implemented first and replaced — it broadcasts p1's least-reliable estimate across the whole run, shifting the mean +0.11% and the within-MLD −0.011, against +0.00% and −0.00035 for isotonic.
Impact: decreasing steps 1,732 → 0; 2023 between share +0.001pp (
pip_income_basis) and +0.001pp (top-adjusted);wid_after_tax_rescaledbit-identical, since it targets the top-adjusted PIP mean and that mean did not move. Per-country Ginis move by a median of 4.7e-06.This also makes the earlier Lorenz income-sort (
a6a863cfc0) load-bearing rather than cosmetic — without it these inverted bins would have biased the Gini, by ~2e-5.Reported, not fixed — two items for follow-up:
shareandavgdo not reconcile for 11 country-years (Chile 2009 publishes a 7.43% top-0.1% share where its own averages and populations imply 6.63%; also El Salvador, Peru, Argentina, Brazil, Ecuador — all Latin American). Corroborated independently: the six largest gaps between our reconstructed Gini and WID's published Gini across all 7,385 country-years are exactly these country-years. Source-side; we reproduce it faithfully. Nocorrections.ymlentry — the contradiction proves an error exists amongshare,avgand the fractile population without identifying which, so per the skill's guard there is nothing to override. Worth raising with WID. Impact on any Gini < 0.003.Two minor notes:
shareis stored to 4 decimals so it sums to 1 only within ~0.0016 (derive fromavg × popwhere exactness matters); and WID's separate "after tax disposable" series gives Czechia a 0.81 Gini for 1996, which is implausible — nothing here uses that series.Validation
xlcusp— whichwid_indices.dodoes viamax(year), and which the source project pins to 2023. The pin overstates income wherever a currency has since depreciated in real terms: Argentina 5.7×, Venezuela 16.2×, Sudan 5.6×, Turkey 2.8×. Sanity-checked against World Bank GNI per capita, where mean post-tax national income per capita should not greatly exceed GNI pc — the 2023 pin puts Argentina at 4.03× GNI pc, this pipeline at 0.71×, and across 194 countries yields 60 implausible cases against our 19 (all tax havens, identical either way).make checkand the full unit-test suite (793 passed) are green;etl diffconfirms all-new datasets.The zero-income floor is the softest assumption here
The MLD is undefined at zero income, so
ZERO_INCOME_REPLACEMENTfloors thosebins at $0.01/day inside the decomposition only — the distributions keep their
true zeros, and only α ≤ 0 needs a floor at all (the Theil index takes
0·log 0 = 0; Gini and GE(2) handle zeros directly). That much is mechanical.
What makes it worth flagging is that the zeros are structural, not incidental: in
WID's pre-tax series the bottom five percentiles are exactly zero in 185 of 211
countries, covering 4.3% of the sample population. So the constant does real
work. Across floors from $0.001 to $1.00/day the pre-tax between-country share
moves 20.1% → 25.6%, and dropping the zero bins instead gives 28.0% — a wider band
than the ~3pp the source project documents. The PIP-side series have no zero bins
and are completely unaffected; WID post-tax nearly so (0.05% of population).
Two further notes for whoever reviews the method:
of daily rather than annual units changes the answer: the same "$0.01" in annual
units would report a within component of 1.274 instead of 1.018. A floor set to a
fraction of each country's own mean is scale-invariant and, at 1e-4 × mean,
reproduces the current numbers almost exactly — a drop-in if we want it.
the between-country share stays between +41 and +49 points under every choice.
Left open
distributiongarden table (this step's main input) has no entry in the WID.meta.yml— the new steps are metadata-self-sufficient, but adding one upstream would benefit all consumers; best coordinated with 🎉 Add WID wealth composition of households and NPISH #6755, which touches that file.wid/2026-06-18; repoint if a WID refresh lands first.CI is green including the staging build. The deliberate decisions it stopped
questioning — extrapolations included, WID-demography weights, the floor above,
garden-only bin and audit tables — have not had a human read yet.
🤖 Generated with Claude Code