Raised by Copilot as a suppressed comment on #1 (src/pubmed2db/parse.py).
parse_file iterates root.findall("PubmedArticle"). The DTD the files declare
(pubmed_250101.dtd) is:
<!ELEMENT PubmedArticleSet ((PubmedArticle | PubmedBookArticle)+, DeleteCitation?) >
<!ELEMENT PubmedBookArticle (BookDocument, PubmedBookData?)>
so Bookshelf citations (GeneReviews, StatPearls and friends — real PMIDs, many with a <BookDocument><Abstract>) can appear in the same files, and we skip them without a log line or a counter. That contradicts the "every PubMed abstract" framing of this tool.
How common are they? Unknown at corpus scale. Neither real file on hand contains one — pubmed26n0005.xml.gz (30,000 records) and pubmed26n1334.xml.gz (4,989 records) both have zero — so this may be a small tail or may be concentrated in particular files. #1 now logs a warning naming the count per file, so the next full load answers this from the logs before any parsing work is decided.
The decision, once the count is known:
- If the count is material, add a book-record path. It is not a small change:
_extract_article upstream handles MedlineCitation only, and BookDocument differs in the fields that matter to us — BookTitle/ArticleTitle, no MedlineJournalInfo (so no nlm_catalog_id and no journal join), PubDate under Book, contributors split between AuthorList and InvestigatorList. That means either a second extraction path or explicit nulls, plus a decision about what the JSON export's journal_name/journal_abbrev should carry for a book chapter.
- If it is negligible, narrow the documented contract in
README.md/CLAUDE.md to "journal citations", and keep the warning so the assumption stays falsifiable.
Either way the count comes first; that is the cheap half and it is already in.
Raised by Copilot as a suppressed comment on #1 (
src/pubmed2db/parse.py).parse_fileiteratesroot.findall("PubmedArticle"). The DTD the files declare(
pubmed_250101.dtd) is:so Bookshelf citations (GeneReviews, StatPearls and friends — real PMIDs, many with a
<BookDocument><Abstract>) can appear in the same files, and we skip them without a log line or a counter. That contradicts the "every PubMed abstract" framing of this tool.How common are they? Unknown at corpus scale. Neither real file on hand contains one —
pubmed26n0005.xml.gz(30,000 records) andpubmed26n1334.xml.gz(4,989 records) both have zero — so this may be a small tail or may be concentrated in particular files. #1 now logs a warning naming the count per file, so the next full load answers this from the logs before any parsing work is decided.The decision, once the count is known:
_extract_articleupstream handlesMedlineCitationonly, andBookDocumentdiffers in the fields that matter to us —BookTitle/ArticleTitle, noMedlineJournalInfo(so nonlm_catalog_idand no journal join),PubDateunderBook, contributors split betweenAuthorListandInvestigatorList. That means either a second extraction path or explicit nulls, plus a decision about what the JSON export'sjournal_name/journal_abbrevshould carry for a book chapter.README.md/CLAUDE.mdto "journal citations", and keep the warning so the assumption stays falsifiable.Either way the count comes first; that is the cheap half and it is already in.