Download PubMed abstracts into a version-history DuckDB and export them as JSON and Parquet

[gaurav]

Adds pubmed2db: a uv/click tool that downloads every PubMed abstract, loads it into a DuckDB database that keeps full version history, and exports the latest version of each abstract to JSON (DocumentMetadataAPI field names, for Node Annotator / ElasticSearch) and Parquet (PubMed field names, for downloadable queries).

The eventual goal is to replace the PubMed download in Babel (createcompendia/publications.py).

The pipeline

Four independent commands, plus two that read or chain them:

• download — bulk baseline/update fetching via cthoyt/pubmed-downloader, with each file's published .md5 checksum recorded in a source_file registry. A new or changed checksum is what marks a file for (re)loading; verification is on by default and hashes only files that are new or whose published checksum moved (--no-verify opts out), and discards a file that fails twice rather than leaving it for load to pick up.
• journals — refreshes the journal dimension from the NLM Catalog (J_Entrez), joined on nlm_catalog_id. Re-fetched on every run, since the catalog changes and the file is small.
• load — parses the XML and inserts it, keeping every version. Rows carry their source_file provenance and a file_order_key that reproduces PubMed's chronological ordering, so the latest_article view can pick the newest non-deleted version of each PMID (honouring <DeleteCitation>).
• export — sharded NDJSON (DocumentMetadataAPI names, empty strings rather than nulls, pub_month as a 3-letter abbreviation) or one Parquet file per table (latest version by default, --all for full history). Both formats sweep files left by a previous export — shards a wider run wrote, or a Parquet file whose table left the schema — so a consumer globbing the directory can't read two exports at once.
• status reports the state of all of it read-only; update chains download → journals → load for scheduled runs, and treats a failed journal refresh as non-fatal so an NLM outage can't discard a completed download.

Design decisions

• The database uses PubMed's own field names. DocumentMetadataAPI names are applied only in the JSON export, so the store stays faithful to the source and the API contract lives in one place.
• Full version history, not upsert. Every file's rows are tagged and kept; latest_article derives the current view. This makes a re-load idempotent and a corrected file cheap to apply.
• Publication dates keep full fidelity. We drive the XML iteration ourselves rather than using the upstream process pipeline, because it collapses partial and MedlineDate-only dates into a datetime.date. Raw Year/Month/Day/MedlineDate are stored, and the JSON export recovers a year from MedlineDate shapes where the parsed fields are empty.
• Nothing is skipped silently. The parser drops three kinds of record — ones the extractor rejects, ones it returns None for (an empty <ArticleTitle>, a missing <MedlineJournalInfo>), and <PubmedBookArticle> citations, which the DTD allows in these files but we don't parse. None of them become rows, so no downstream count is short; source_file.n_failed and n_book_records make them queryable and status reports the total.
• No citation graph. One real article carries ~444 references, so at corpus scale reference_citation would have been the largest table here, for data no consumer asked for. The extraction is parked (uncalled) with re-enabling instructions rather than deleted.
• Step ordering is enforced from database state, not a run flag. load errors when nothing is downloaded, export errors when nothing is loaded and warns about unloaded files or an empty journal table. Deriving readiness from the registry's watermarks means a check can't disagree with the data it guards.
• Columnar bulk load. Each file's rows go in as an Arrow table via INSERT ... SELECT rather than row-by-row executemany, which took ~20 minutes per file. The benchmark measures ~5–6 s per file, but a corpus-scale load has since been observed at ~91 s per file, so treat the benchmark figure as a floor until Load is running ~15x slower than slurm/README.md claims — re-measure #11 re-measures it.
• MD5 is treated as low-priority insurance. HTTP downloads are reliable and PubMed files are immutable, so verification only hashes files that are new or whose published checksum moved — re-hashing an unchanged corpus costs tens of GiB of I/O and catches nothing.
• schema.sql only ever adds, and runs on every connect: CREATE TABLE IF NOT EXISTS plus explicit ALTER TABLE ... ADD COLUMN IF NOT EXISTS migrations for anything added after the first release. Nothing drops a table or column, which is why a schema change wants a rebuild rather than load --force (below).

Upstream bugs worked around

Three pubmed-downloader (≤ 0.0.14) bugs, each tracked in FUTURE.md with a pinning test so the workaround fails loudly once upstream fixes it. The dependency is pinned <0.1 because we also call private APIs (_extract_article, _ensure_urls).

• Journal parsing raises. catalog.process_journal_overview()'s Journal model requires start_year/end_year, which the real J_Entrez.txt does not carry, so we parse the overview file ourselves. Filed upstream as Make catalog.Journal.start_year / end_year optional so it works with J_Entrez.txt cthoyt/pubmed-downloader#16
• References are never found. _extract_article looks for .//ReferenceList/Reference under MedlineCitation, but PubMed nests <ReferenceList> under <PubmedData>, so Article.cites_pubmed_ids is always empty on real data. Harmless here — we store no citation graph — but a working counter-example is kept for whenever it is reported.
• Article IDs are over-collected. That same .// descent attributes every cited reference's DOI/PMID to the citing article, which would have been silently wrong rows in article_id proportional to reference count. We use the direct PubmedData/ArticleIdList/ArticleId path instead.

Operations

• Everything runs from the repo with uv run pubmed2db …; nothing is installed.
• Group-level --threads / --temp-dir cap DuckDB's thread pool and set its spill directory, since DuckDB otherwise sizes its pool from the node's cores rather than the Slurm allocation. All four group options also read an environment variable (PUBMED2DB_THREADS, PUBMED2DB_DUCKDB_TEMP_DIR, PUBMED2DB_DATA_DIR, PUBMED2DB_DB), which --help documents.
• slurm/README.md sizes both jobs: load ~16 GB (memory is bounded by the largest single file) and export ~256 GB (whole-corpus snapshot + sort). The export figure comes from the full-scale run below; the load figure comes from the benchmark, and a real run has since exceeded it (Load is running ~15x slower than slurm/README.md claims — re-measure #11).
• Re-running after a gap can't duplicate data, and the README explains why. Two cases cost real time: a new baseline year is a full re-parse that stores a second version of every PMID, and applying a parser or schema change to already-loaded data. For the latter the README now gives the rule — rebuild from a fresh database unless you know a parsing change is all you need. Both cost one full load and neither re-downloads, but since schema.sql only ever adds, a table or column removed from it keeps its rows through any number of load --force runs, and a forced reload leaves wrong rows in place for files it re-parses identically. Only the rebuild is guaranteed to leave the shape the schema describes.
• export publishes in place, not atomically — a run that dies partway leaves a half-written dataset in --out. Export into a fresh directory and swap it if consumers read the output while exports run (export publishes in place, so a failed run leaves a half-written dataset #24).
• Documentation is split by audience: README.md for running the pipeline, AGENTS.md for a map of the source and the decisions that aren't visible from it, FUTURE.md for deferred work. Anything a docstring or a schema comment can carry lives there instead.

Verification

• 84 tests pass with no network: parse fidelity, latest-version and deletion logic, idempotent and MD5-triggered reloads, journal parsing, the JSON field contract, Parquet filtering, DuckDB tuning options, schema migration, and the CLI end to end.
• Full-scale run on RENCI's cluster: the complete corpus exported as 40,901,984 documents across 16 NDJSON shards in 23m13s (≈30k documents/s), peak RSS 201.1 GiB under --mem=256G --cpus-per-task 8.
• Single-file live run: MD5-verified download → 16,664 articles + 56 deletions → 41,923 journals → 16,664 documents across even NDJSON shards (zero nulls, journal names and abbreviations resolved, months normalized) and 15 consistent Parquet files.
• Two fields a validation run reported as blank are not a parser gap. PMID 10137601's issue ("Suppl") and PMID 28972331's article_title ("[Not Available].") were fetched from Entrez and run through parse → load → export: both come out correct at every stage. They are in the fixtures and pinned, because both shapes — a non-numeric issue whose volume repeats it, and a title that is a literal placeholder with the real one in <VernacularTitle> — are what a future change might reasonably decide to drop.

Follow-on work

Issues opened for what this PR deliberately leaves out:

• Consider dropping the global sort in the JSON export #8 — drop the JSON export's global ORDER BY (measured: the sort is ~0.8 GiB and 75% of wall time on a 2M-row sample). Being done in Let DuckDB write the JSON export, and stop sorting it (~3x faster) #12.
• Load is running ~15x slower than slurm/README.md claims — re-measure #11 — a corpus-scale load ran ~15x slower than slurm/README.md documents, and above its stated memory ceiling; the load figures need re-measuring against a large database rather than a short benchmark.
• Reloading a file replaced on disk needs --force: needs_load never looks at the filesystem #21 — needs_load never looks at the filesystem, so a file replaced on disk is only picked up by load --force.
• Prune a superseded baseline year: files and source_file rows are never removed #22 — nothing prunes a superseded baseline year; status reports the condition and the README gives the rm, but 51 GB of baseline plus 13 GB of updates per year wants a command.
• Consider a smoke test for documented paths (three README/CLAUDE.md claims had drifted) #23 — a smoke test for documented paths, after three drifted doc claims.
• export publishes in place, so a failed run leaves a half-written dataset #24 — neither export publishes atomically.
• Stream the XML with iterparse: parse_file holds the whole DOM (~1.2 GiB/file) #25 — parse_file holds the whole DOM (~1.2 GiB of the ~2.2 GiB per-file peak); iterparse measures at 0.02 GiB.
• Overlap parsing with insertion in load_files (~2s of every ~6s per file is idle) #26 — parse and insert never overlap, though DuckDB releases the GIL: ~40 minutes of a full load.
• Decide what to do with <PubmedBookArticle> records, which parse_file silently ignores #27 — decide whether to parse <PubmedBookArticle> records; the per-file count this PR records answers whether it matters.
• Confirm the pub_year backfill at corpus scale after the next rebuild #28 — confirm the pub_year backfill at corpus scale (verified on one baseline file: 3,625 of 30,000 records, 127 distinct MedlineDate shapes, all recovering a year).
• Run the Parquet export at corpus scale and record its cost #29 — run the Parquet export at corpus scale; only the JSON path has full-corpus numbers.

[Copilot]

Pull request overview

Initial implementation of pubmed2db, a Python CLI tool that downloads PubMed baseline/update XMLs, loads them into a full-version-history DuckDB schema, and exports the latest non-deleted article set to NDJSON (DocumentMetadataAPI fields) and Parquet.

Changes:

• Added end-to-end pipeline modules (download → parse → load → export → status) plus a Click CLI wrapper.
• Introduced a normalized DuckDB schema with a latest_article view for newest-version selection and deletion handling.
• Added comprehensive pytest coverage with XML/journal fixtures, plus docs and Slurm run guidance.

Reviewed changes

Copilot reviewed 26 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/pubmed2db/__init__.py	Defines package version and module docstring.
src/pubmed2db/cli.py	Click CLI implementing download, journals, load, export, update, status.
src/pubmed2db/db.py	DuckDB connection + schema init + source-file registry helpers.
src/pubmed2db/download.py	PubMed sync via pubmed-downloader with .md5 sidecar tracking and optional verification.
src/pubmed2db/export.py	JSON (DocumentMetadataAPI fields, empty-string-not-null) and Parquet exports.
src/pubmed2db/load.py	Normalized loading with full history, idempotent reloads, and journal overview parsing/loading.
src/pubmed2db/parse.py	Self-driven XML iteration using _extract_article, plus raw PubDate + DeleteCitation capture.
src/pubmed2db/schema.sql	Full normalized schema + latest_article view implementing newest-version selection with deletions.
src/pubmed2db/status.py	Read-only pipeline readiness/state summarization derived from DB state.
src/pubmed2db/util.py	Shared helpers for peak RSS and duration formatting.
tests/conftest.py	Shared fixtures (gzipping XML fixtures, DuckDB connections, staged download layout).
tests/test_cli.py	CLI end-to-end tests (export, status, error messaging, directory scan, gzip export).
tests/test_db_download.py	Tests for filename ordering, registry upsert behavior, and MD5 parsing/helpers.
tests/test_export.py	Tests for month normalization, JSON field contract, sharding, and Parquet latest/all behavior.
tests/test_journals.py	Tests for journal overview parsing and journal load behavior.
tests/test_load.py	Tests for full history retention, latest selection, deletion behavior, and MD5-triggered reloads.
tests/test_parse.py	Tests for raw-date fidelity, rich extraction fields, and DeleteCitation capture.
tests/__init__.py	Marks tests as a package.
tests/fixtures/pubmed25n0001.xml	Baseline fixture XML used for parsing/load/export tests.
tests/fixtures/pubmed25n0002.xml	Update fixture XML including a revised article and a DeleteCitation.
tests/fixtures/J_Entrez_sample.txt	Sample NLM journal overview fixture for journal parser tests.
scripts/benchmark_load.py	Script to benchmark parsing vs insertion throughput and peak RSS.
slurm/README.md	Operational notes for running loads on Slurm (memory/time sizing, monitoring).
README.md	Project documentation: purpose, pipeline, usage examples, and development notes.
CLAUDE.md	Architecture/design orientation document for the repository.
FUTURE.md	Deferred work items and known limitations.
pyproject.toml	Project metadata, dependencies, and pytest configuration.
.gitignore	Ignores downloaded data while keeping directory skeleton.
data/.gitkeep	Keeps data/ directory present in the repository.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 30 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/pubmed2db/parse.py:144

• PubmedArticleSet can also contain PubmedBookArticle records, including Bookshelf records with a <BookDocument><Abstract>. This loop silently ignores those PMIDs (and does not increment n_failed), so a successful load/export is missing PubMed abstracts despite the PR's all-abstracts contract. Add an explicit book-record parsing path, or narrow and document the supported corpus while reporting skipped book records.

    for element in root.findall("PubmedArticle"):

src/pubmed2db/download.py:116

• changed only reflects registry membership/checksum, not whether ensure() fetched bytes. If a registered file was removed locally while its published checksum stayed the same, ensure() downloads it again but --verify skips hashing; a pending file can then be loaded from an unverified transfer. Track local existence before ensure() and hash whenever bytes were newly downloaded or the checksum changed.

        if verify and published_md5 is not None and changed: