📊 Add long-run US immigration data (DHS) by edomt · Pull Request #6498 · owid/etl

edomt · 2026-07-22T09:25:08Z

Written by Claude Fable 5 — @edomt at the wheel.

Part of https://github.com/owid/owid-issues/issues/2467 (long-term migration data for the United States).

Adds two new datasets:

1. Legal permanent residents (flow) — `us_lawful_permanent_residents`

The number of people obtaining legal permanent resident status, from the DHS Yearbook of Immigration Statistics:

Annual totals, 1820–2024 (2024 yearbook), plus a rate per 1,000 US population.
By country of origin, by decade, 1820s–2010s (2020 yearbook, the last edition with this breakdown).
By region of origin, by decade — rebuilt from the country rows using OWID region definitions.

Notes:

Country names kept as published (Russia, Czechoslovakia, Yugoslavia, Korea…), except "Canada and Newfoundland" → Canada. Combined rows (Austria-Hungary, Norway-Sweden) are used until the separate countries fully reconcile with them; the separate countries are used from then on.
Canada and Mexico are dropped from the country table before the 1910s (overland arrivals were not fully counted until 1908) but still count toward regional totals.
Sanity checks assert the rebuilt regional totals match the source's aggregates exactly, and that the two yearbook editions agree on decade totals within 1%.

2. Foreign-born population (stock) — `us_foreign_born_population`

The foreign-born population and its share of the total population, 1850–2024:

1850–2000 (decennial): Census Bureau working paper "Historical Census Statistics on the Foreign-Born Population of the United States" (Gibson & Jung 2006), parsed from Table 1 of the PDF.
2005–2024 (annual): American Community Survey 1-year estimates (table B05002), via the Census Bureau API. No 2020 value (not released due to the pandemic).

Also broken down by country of birth (133 entities) and by region of birth (OWID regions):

1850–2000 (decennial): Table 4 of the same working paper, now snapshotted as the XLS files the Census Bureau publishes alongside the PDF (the earlier PDF snapshot is replaced). No 1940/1950 (only published for the White population back then).
2005–2024 (annual): ACS table B05006 via the API, storing each year's raw variable codes and labels (they shift between years).
Countries are kept as the sources report them (Czechoslovakia stays combined even today, Korea covers both Koreas, China excludes Hong Kong and Taiwan since 1980). The United Kingdom series uses the source's Great Britain row before 1930, and Turkey combines the source's "Turkey in Europe" and "Turkey in Asia" rows.
Region aggregates are rebuilt on OWID definitions from the sources' own aggregate rows, with an exact-partition assert against the total in every year.

Sanity checks pin parsed values and computed shares to figures printed in the working paper (e.g. 1890 = 14.8%, 1970 = 4.7%).

🤖 Generated with Claude Code

owidbot · 2026-07-22T09:26:13Z

Quick links (staging server):

Site Dev	Site Preview	Admin	Wizard	Docs	Docs Preview

Login: ssh owid@staging-site-data-add-longrun-us

chart-diff: ❌

0/9 reviewed charts
Modified: 0/0
New: 0/9
Rejected: 0
Data changes: 0
Metadata changes: 0

data-diff: ❌ 1 changed · 2 new · 6 identical — full report

~ garden/regions/2023-01-01/regions
    ~ regions.code dim (removed data)
        - Removed values: 14 / 436 (3.21%)
            EI_AFR
            EI_APAC
            EI_CIS
            (+11 more)
    ~ regions.defined_by (changed data)
        - Removed values: 14 / 436 (3.21%)
            EI_AFR: ei
            EI_APAC: ei
            EI_CIS: ei
            (+11 more)
    ~ regions.is_historical (changed data)
        - Removed values: 14 / 436 (3.21%)
            EI_AFR: False
            EI_APAC: False
            EI_CIS: False
            (+11 more)
    ~ regions.members (changed data)
        - Removed values: 14 / 436 (3.21%)
            EI_AFR: ["AGO", "BDI", "BEN", "BFA", "BWA", "CAF", "CIV", "CMR", "COD", "COG", "COM", "CPV", "DJI", "DZA", "EGY", "ERI", "ESH", "ETH", "GAB", "GHA", "GIN", "GM…
            EI_APAC: ["AFG", "ASM", "AUS", "BGD", "BRN", "BTN", "CCK", "CHN", "COK", "CXR", "FJI", "FSM", "GUM", "HKG", "IDN", "IND", "JPN", "KHM", "KIR", "KOR", "LAO", "L…
            EI_CIS: ["ARM", "AZE", "BLR", "KAZ", "KGZ", "MDA", "RUS", "TJK", "TKM", "UZB"]
            (+11 more)
    ~ regions.name (changed data)
        - Removed values: 14 / 436 (3.21%)
            EI_AFR: Africa (EI)
            EI_APAC: Asia Pacific (EI)
            EI_CIS: CIS (EI)
            (+11 more)
    ~ regions.region_type (changed data)
        - Removed values: 14 / 436 (3.21%)
            EI_AFR: aggregate
            EI_APAC: aggregate
            EI_CIS: aggregate
            (+11 more)
    ~ regions.short_name (changed data)
        - Removed values: 14 / 436 (3.21%)
            EI_AFR: Africa (EI)
            EI_APAC: Asia Pacific (EI)
            EI_CIS: CIS (EI)
            (+11 more)
+ garden/migration/2026-07-22/us_foreign_born_population (new dataset: 4 table(s), 5 column(s))
+ garden/migration/2026-07-22/us_lawful_permanent_residents (new dataset: 3 table(s), 4 column(s))

= 6 compared dataset(s) turned out identical; details in the full report

Edited: 2026-08-10 12:10:43 UTC
Execution time: 43.58 seconds

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt · 2026-07-22T10:24:29Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ad8cd542b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…e rows are partial The separate Austria and Hungary rows do not cover all years in the 1860s, 1890s, and 1900s (Table 2, footnote 2), so preferring them over the combined row dropped the unallocated remainder (~846k people in total) from the by-country table. Switch from the combined row to the separate rows only from the first decade after which the separate rows always add up to the combined row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt · 2026-07-22T10:40:37Z

@codex review

…nces Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chatgpt-codex-connector · 2026-07-22T10:53:42Z

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 2d0f3f1981

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ngrun-us

…ion bullets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…description bullets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…, style fixes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Move the status-grants note below the counting-changes text and keep the description key order aligned with that presentation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…aveats Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt · 2026-07-22T13:31:13Z

@codex review

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chatgpt-codex-connector · 2026-07-22T13:41:22Z

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 253e55bbaa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ngrun-us

…+ ACS) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…mplate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This reverts commit 18ca818.

In both by-country tables, people not attributed to a listed country now appear as e.g. "Asia (not specified)" when the source gives their region, with plain "Not specified" kept for the truly unrecorded. The stock dataset's by-country selection is rewritten as an exact-cover walk of the source's hierarchy, so remainders of rows whose sub-rows do not add up (Yugoslavia, Portugal) are captured and every year is asserted to add up to the source's total. Also adds entity annotations for Korea, Czechoslovakia, and Yugoslavia, and corrects the China/Hong Kong bullet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt · 2026-07-23T07:47:33Z

@codex review

chatgpt-codex-connector · 2026-07-23T07:52:18Z

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 0b62b3bbef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ngrun-us

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

New share_by_origin_group table (Africa, Asia, Oceania, Europe, Mexico, Canada, Rest of America, Not specified), as requested for the stacked long-run chart. Also removes the dead pre-walker selection functions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt · 2026-07-23T09:26:21Z

@codex review

chatgpt-codex-connector · 2026-07-23T09:36:29Z

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 3ad239cb10

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ngrun-us

📊 Add long-run US immigration data (DHS)

3fe091a

📊🤖 Add long-run US lawful permanent residents dataset (DHS Yearbook)

ad8cd54

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chatgpt-codex-connector Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread etl/steps/data/garden/migration/2026-07-22/us_lawful_permanent_residents.py Outdated

edomt and others added 2 commits July 22, 2026 12:36

🔨🤖 Fold Other America into the Not specified series

758d0e6

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt and others added 3 commits July 22, 2026 12:40

💄🤖 Reorder description bullets; drop decade and regional-totals sente…

033bf66

…nces Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Trim yearbook description to its first sentence

c3a044e

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Drop country-of-origin vs last-residence bullet

2d0f3f1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt and others added 11 commits July 22, 2026 12:54

Merge branch 'master' of https://github.com/owid/etl into data-add-lo…

58336fd

…ngrun-us

💄🤖 Expand counting-changes into sub-points; reorder and trim descript…

ac4092d

…ion bullets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Move overland-counting caveat into the counting-changes sub-points

a987b22

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Drop redundant Canada/Mexico bullet from by-country indicator

1013294

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Restructure borders bullet into sub-points; bold the 1908 cutoff

0a722d9

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Drop temporary visitors from what-this-counts (not true before 1868)

b272676

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Rename indicators, set subtitles and origin-chart footnotes, trim …

f21c49f

…description bullets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Reword overland-counting caveat to 'until 1908'

ebaa424

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Source-based titles, IRCA bullet, drop subtitles and display names…

2821580

…, style fixes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Drop redundant title_public and title_variant fields

f7369f1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Add status-grants bullet to origin indicators

0adb351

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt self-assigned this Jul 22, 2026

Reorder LPR metadata text

838b0ca

Move the status-grants note below the counting-changes text and keep the description key order aligned with that presentation.

edomt marked this pull request as ready for review July 22, 2026 12:12

edomt and others added 4 commits July 22, 2026 14:39

💄🤖 Set chart footnotes on all flow indicators

3182a95

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Say decade in origin-chart footnotes

5d0c037

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

📊🤖 Add US foreign-born population dataset (Census Bureau + ACS)

a319577

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Expand foreign-born metadata: definition, denominator, stitching c…

a2cd6e3

…aveats Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

🔨🤖 Keep ACS snapshot as raw API output, rename columns in meadow

253e55b

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt and others added 4 commits July 22, 2026 15:33

💄🤖 Reword undercount bullet

42e13a9

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Replace ACS promotional blurb with a factual description

96b7090

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Reword census working-paper description from OWID's point of view

13e3edb

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

🔨🤖 Set dataset owner

808d70d

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt and others added 9 commits July 22, 2026 15:44

Merge branch 'master' of https://github.com/owid/etl into data-add-lo…

52e5de9

…ngrun-us

📊🤖 Add country-of-birth breakdown to US foreign-born dataset (Census …

4e4e2c9

…+ ACS) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

🐛🤖 Count European Turkey in Asia, per OWID region definitions

0f6c7cd

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Move snapshot-not-arrivals sentence from WYSK to chart footnote

d57b6d4

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Split missing-years sentence into its own bullet, reuse sources te…

9d5b216

…mplate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Lowercase white per AP style

94dee06

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

🔨🤖 Show the source's combined Korea figure as South Korea

18ca818

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Revert "🔨🤖 Show the source's combined Korea figure as South Korea"

0b62b3b

This reverts commit 18ca818.

edomt and others added 8 commits July 23, 2026 09:53

💄🤖 Write WYSK as markdown prose instead of top-level bullets

65202b6

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge branch 'master' of https://github.com/owid/etl into data-add-lo…

494e6dd

…ngrun-us

💄🤖 Reword flow chart footnotes: residency granted vs arrival

73582f6

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Rename indicators to people granted permanent residency

25c0bb8

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Drop the amnesty WYSK paragraph; the footnote covers it

e4bf0ee

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Retitle origin-group share to by place of birth

0070645

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

💄🤖 Keep the undercount paragraph last in every WYSK

3ad239c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

edomt added 2 commits July 23, 2026 11:39

Merge branch 'master' of https://github.com/owid/etl into data-add-lo…

9f0dbaf

…ngrun-us

🔨 Wake up staging server

0a2c353

Navigation Menu

Uh oh!

Uh oh!

📊 Add long-run US immigration data (DHS) - #6498

📊 Add long-run US immigration data (DHS)#6498
edomt wants to merge 51 commits into
masterfrom
data-add-longrun-us

edomt commented Jul 22, 2026 •

edited

Loading

Uh oh!

owidbot commented Jul 22, 2026 •

edited

Loading

Uh oh!

edomt commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

edomt commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 22, 2026

Uh oh!

edomt commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 22, 2026

Uh oh!

edomt commented Jul 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 23, 2026

Uh oh!

edomt commented Jul 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

edomt commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Legal permanent residents (flow) — us_lawful_permanent_residents

2. Foreign-born population (stock) — us_foreign_born_population

Uh oh!

owidbot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edomt commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

edomt commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 22, 2026

Uh oh!

edomt commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 22, 2026

Uh oh!

edomt commented Jul 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 23, 2026

Uh oh!

edomt commented Jul 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

edomt commented Jul 22, 2026 •

edited

Loading

1. Legal permanent residents (flow) — `us_lawful_permanent_residents`

2. Foreign-born population (stock) — `us_foreign_born_population`

owidbot commented Jul 22, 2026 •

edited

Loading