You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Milestone 2026aug20: the rebuild this issue is waiting for is the one about to run.
This says "confirm after the next rebuild" — that is now. The backfill is on main and verified on one baseline file (3,625 of 30,000 records carry a MedlineDate, 127 distinct shapes, all recovering a year); what is unverified is the whole corpus, and only a corpus export produces that evidence.
It is a query against the finished export rather than anything to remember during the run, so nothing here changes how the run is launched — but the export has to be kept around long enough to ask. Worth pairing with #24's export-to-a-new-directory suggestion.
Note the companion gap: pub_year recovers from MedlineDate, but pub_month does not preserve approximate months on main (#14). Both are worth checking in the same pass over the export, since they come from the same records.
Records whose <PubDate> carries only a <MedlineDate> (e.g. 1994 Jun-Jul, 1998 Winter) have no <Year> element, so pub_year exported blank for them. The JSON export now recovers a year from the MedlineDate string when the parsed fields are empty (export._year_from_medline_date).
Verified so far, on one real baseline file (pubmed26n0005.xml.gz): 3,625 of 30,000 records carry a MedlineDate, across 127 distinct shapes, and all of them recover a year.
What is still unverified: the same thing across the whole corpus. The check is cheap once a full export exists — validate's core-fields sample should drop from 20 mismatches to ~2 (the two residual ones, which #1 investigated separately and could not reproduce against current code).
So: after the next full rebuild and export, run validate and record the core-fields number here. If it is not ~2, the MedlineDate shapes in the tail differ from the ones the sample file covers, and _year_from_medline_date needs the failing shapes added.
Important
Milestone 2026aug20: the rebuild this issue is waiting for is the one about to run.
This says "confirm after the next rebuild" — that is now. The backfill is on
mainand verified on one baseline file (3,625 of 30,000 records carry aMedlineDate, 127 distinct shapes, all recovering a year); what is unverified is the whole corpus, and only a corpus export produces that evidence.It is a query against the finished export rather than anything to remember during the run, so nothing here changes how the run is launched — but the export has to be kept around long enough to ask. Worth pairing with #24's export-to-a-new-directory suggestion.
Note the companion gap:
pub_yearrecovers fromMedlineDate, butpub_monthdoes not preserve approximate months onmain(#14). Both are worth checking in the same pass over the export, since they come from the same records.Deferred from #1, which added the backfill.
Records whose
<PubDate>carries only a<MedlineDate>(e.g.1994 Jun-Jul,1998 Winter) have no<Year>element, sopub_yearexported blank for them. The JSON export now recovers a year from theMedlineDatestring when the parsed fields are empty (export._year_from_medline_date).Verified so far, on one real baseline file (
pubmed26n0005.xml.gz): 3,625 of 30,000 records carry aMedlineDate, across 127 distinct shapes, and all of them recover a year.What is still unverified: the same thing across the whole corpus. The check is cheap once a full export exists —
validate'score-fieldssample should drop from 20 mismatches to ~2 (the two residual ones, which #1 investigated separately and could not reproduce against current code).So: after the next full rebuild and export, run
validateand record thecore-fieldsnumber here. If it is not ~2, theMedlineDateshapes in the tail differ from the ones the sample file covers, and_year_from_medline_dateneeds the failing shapes added.