Conversation
When a player has more than one event in the same Wikipedia lineup cell (e.g. a first yellow at 36' and a second yellow at 88'), the scraped `events` string concatenates them with no comma between them. The bookings and substitutions builders only split on commas via separate_rows(), so these collapsed into a single row: the first event's minute was kept, the second event's minute was discarded, and the flags from both events were merged onto one row (e.g. yellow_card=1 AND second_yellow_card=1 on the 36' row, with the actual sending-off minute of 88' lost). Two one-off hand-patches already existed for individual instances of this, confirming the bug but not generalizing it. Insert a comma after an event's minute whenever it is immediately followed by another event keyword, so each card / substitution becomes its own row. Effect on the data: - bookings: 3178 -> 3263 rows (+85 first-cards recovered) - yellow_card total: 3092 -> 3162; red_card, second_yellow_card and sending_off totals unchanged (no fabricated cards or dismissals) - sending-off minutes now point at the dismissal card, not the earlier first yellow - substitutions: no change in the current data (no remaining multi-event sub cells), but the same fix is applied defensively Regenerated data/bookings.RData, data-csv/bookings.csv, the bookings table in data-sqlite/worldcup.db, and data-json/worldcup.json. Only the bookings table changed; all other tables are byte/row-identical.
mslovich
added a commit
to mslovich/worldcup-malloy
that referenced
this pull request
Jun 18, 2026
Patch bookings.parquet with corrected data (jfjelstul/worldcup#12)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #11.
The bug
The
bookingstable is documented as having one row per booking, but when a player has more than one event in the same Wikipedia lineup cell — e.g. a yellow card followed by the second yellow or red card that results in a sending-off — the scrapedeventsstring concatenates them with no comma between them ("yellow card 40' red card 42'"). The bookings and substitutions builders only split events on commas viaseparate_rows(events, sep = ","), so these collapse into a single row: the first event's minute is kept, the second event's minute is discarded, and the flags from both events are merged onto one row.This is exactly the inconsistency reported in #11. Ray Wilkins' dismissal vs Morocco (1986) was stored as a single row flagged both
yellow_card = 1andred_card = 1at 40', rather than a yellow at 40' and the sending-off at 42'.Two one-off hand-patches already existed in the builder (Khalil Ghanim's booking, one substitution), confirming the bug but not generalizing it.
The fix
Before the comma-based
separate_rows(), insert a comma after an event's minute whenever it is immediately followed by another event keyword, so each card / substitution becomes its own row. Applied in both the bookings and substitutions builders; it also subsumes the two existing hand-patches.Effect on the data
Ray Wilkins (England vs Morocco, 1986) — #11:
Totals:
red_card,second_yellow_card,sending_offtotals unchanged — no cards or dismissals fabricated; the recovered rows are the previously-dropped first cardsWhy substitutions is fixed in code but not regenerated
The same fix is applied to the substitutions builder because it shares the identical root cause, but no substitutions data files are changed in this PR. The current scraped data contains no remaining multi-event substitution cells (the only such case was already covered by an existing hand-patch), so running the patched builder produces output byte-identical to the committed
substitutionsdata (verified: 10,222 rows in, 10,222 rows out, no diff). The committed substitutions files therefore already match the fixed code — there is nothing to rebuild. The change is included as a defensive, consistent fix so the bug cannot reappear in substitutions if the source data is ever re-scraped or extended.Regenerated artifacts
data/bookings.RData,data-csv/bookings.csv, thebookingstable indata-sqlite/worldcup.db, anddata-json/worldcup.json. Only thebookingstable changed; every other table (includingsubstitutions) is byte/row-identical.