Skip to content

Confirm the pub_year backfill at corpus scale after the next rebuild #28

Description

@gaurav

Important

Milestone 2026aug20: the rebuild this issue is waiting for is the one about to run.

This says "confirm after the next rebuild" — that is now. The backfill is on main and verified on one baseline file (3,625 of 30,000 records carry a MedlineDate, 127 distinct shapes, all recovering a year); what is unverified is the whole corpus, and only a corpus export produces that evidence.

It is a query against the finished export rather than anything to remember during the run, so nothing here changes how the run is launched — but the export has to be kept around long enough to ask. Worth pairing with #24's export-to-a-new-directory suggestion.

Note the companion gap: pub_year recovers from MedlineDate, but pub_month does not preserve approximate months on main (#14). Both are worth checking in the same pass over the export, since they come from the same records.


Deferred from #1, which added the backfill.

Records whose <PubDate> carries only a <MedlineDate> (e.g. 1994 Jun-Jul, 1998 Winter) have no <Year> element, so pub_year exported blank for them. The JSON export now recovers a year from the MedlineDate string when the parsed fields are empty (export._year_from_medline_date).

Verified so far, on one real baseline file (pubmed26n0005.xml.gz): 3,625 of 30,000 records carry a MedlineDate, across 127 distinct shapes, and all of them recover a year.

What is still unverified: the same thing across the whole corpus. The check is cheap once a full export exists — validate's core-fields sample should drop from 20 mismatches to ~2 (the two residual ones, which #1 investigated separately and could not reproduce against current code).

So: after the next full rebuild and export, run validate and record the core-fields number here. If it is not ~2, the MedlineDate shapes in the tail differ from the ones the sample file covers, and _year_from_medline_date needs the failing shapes added.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions