Skip to content

Decide what to do with <PubmedBookArticle> records, which parse_file silently ignores #27

Description

@gaurav

Raised by Copilot as a suppressed comment on #1 (src/pubmed2db/parse.py).

parse_file iterates root.findall("PubmedArticle"). The DTD the files declare
(pubmed_250101.dtd) is:

<!ELEMENT PubmedArticleSet ((PubmedArticle | PubmedBookArticle)+, DeleteCitation?) >
<!ELEMENT PubmedBookArticle (BookDocument, PubmedBookData?)>

so Bookshelf citations (GeneReviews, StatPearls and friends — real PMIDs, many with a <BookDocument><Abstract>) can appear in the same files, and we skip them without a log line or a counter. That contradicts the "every PubMed abstract" framing of this tool.

How common are they? Unknown at corpus scale. Neither real file on hand contains one — pubmed26n0005.xml.gz (30,000 records) and pubmed26n1334.xml.gz (4,989 records) both have zero — so this may be a small tail or may be concentrated in particular files. #1 now logs a warning naming the count per file, so the next full load answers this from the logs before any parsing work is decided.

The decision, once the count is known:

  • If the count is material, add a book-record path. It is not a small change: _extract_article upstream handles MedlineCitation only, and BookDocument differs in the fields that matter to us — BookTitle/ArticleTitle, no MedlineJournalInfo (so no nlm_catalog_id and no journal join), PubDate under Book, contributors split between AuthorList and InvestigatorList. That means either a second extraction path or explicit nulls, plus a decision about what the JSON export's journal_name/journal_abbrev should carry for a book chapter.
  • If it is negligible, narrow the documented contract in README.md/CLAUDE.md to "journal citations", and keep the warning so the assumption stays falsifiable.

Either way the count comes first; that is the cheap half and it is already in.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions