feat(scryfall): ingest curated Japanese collector printings#66
Conversation
Enrich the bulk ingest with Japanese-exclusive-art and Kamigawa full-art printings pulled from the Scryfall search API. These link to existing English Cards by name, so no new Card rows or oracle/legality drift — only Printings gain `lang` and `printed_name`. Key changes: - Add `lang` + `printed_name` to Printing model and map them from the Scryfall card payload (both feed the version hash) - Add `ingestCollectorPrintings` step running after bulk upsert and before the checkpoint commit, paginating search and deduping by scryfallId - Add JP_COLLECTOR_QUERIES as the curated source of search queries - Treat a 404 search response as an empty result, not an error - Fix token upsert target table: card_token -> card_tokens (@@Map'd plural; singular threw 42P01 in prod)
|
Warning Review limit reached
More reviews will be available in 30 minutes and 14 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (12)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
A search-API outage in the JP collector-printing enrichment step threw before the checkpoint commit, stranding the checkpoint and forcing the next cron to re-download the full ~500MB bulk. Make enrichment best-effort and harden its filtering and tests. Key changes: - ingest: wrap ingestCollectorPrintings in try/catch + logWarn so the checkpoint commits even when JP enrichment rejects - filter: extract language-agnostic isPaperPlayable guard and apply it in the JP search path to keep digital-only/token-layout printings out - steps: add inter-query courtesy delay mirroring the inter-page delay - tests: cover the printing UPDATE-path SQL (lang/printed_name + version guard) and the checkpoint-commits-on-JP-failure path; assert query count against JP_COLLECTOR_QUERIES.length - runbook: document the one-time printingsUpdated spike on first deploy
1 participant
Description
Adds a post-bulk enrichment step that pulls curated Japanese collector printings from the Scryfall
/cards/searchAPI and upserts them as Printings against the already-ingested English Cards. Japanese printings keep their Englishname(the newprinted_namecolumn holds the Japanese title), so each links to its existing Card by name — no new Card rows, no oracle/legality drift.The set of queries lives in
lib/scryfall/jp-collector-queries.tsas the single source of truth. Theart:japanese-exclusive-arttag isolates exclusive-art printings across every set with no per-set maintenance; NEO is a named supplement because its soft-glow / ukiyo-e treatments reuse the English art and fall outside that tag.Also fixes a latent bug: token upsert targeted
card_token(singular) but theCardTokenmodel is@@map'd tocard_tokens— a singular target throws42P01in prod. Now guarded by a test asserting the table name.Fixes: #62
Checklist
pnpm test) and lint is cleanrevalidate/dynamic/unstable_cache— used'use cache'+cacheLife/cacheTag<Link>imported fromapp/_components/link.tsx, notnext/linkScreenshots / notes
graph TD Q[jp-collector-queries.ts] --> ICP[ingestCollectorPrintings 'use step'] ICP --> FSS[fetchScryfallSearch<br/>paginate + 404 = empty] FSS --> SC[Scryfall /cards/search<br/>unique=prints] ICP --> DEDUPE[dedupe by scryfallId] DEDUPE --> NAMES[resolve cardId by English name] NAMES --> DIFF[buildPrintings → diffPrintings] DIFF --> WRITE[applyPrintingWrites<br/>lang + printed_name] WF[scryfallIngestWorkflow] -->|after bulk upsert,<br/>before checkpoint| ICPSchema
prisma/schema.prisma,prisma/migrations/.../migration.sql— addlang(NOT NULL DEFAULT 'en') and nullableprinted_nametoPrinting.Mapping
lib/scryfall/schema.ts— accept optionalprinted_namefrom Scryfall payloads.lib/scryfall/map.ts— maplangandprinted_nameintoPrintingCreateData(both feed the content-hashversion).lib/scryfall/jp-collector-queries.ts— new curated query list (source of truth).Workflow
workflows/scryfall/steps.ts— newingestCollectorPrintingsstep +fetchScryfallSearchpaginator;lang/printed_namethreaded through the printing upsert SQL; fixcard_token→card_tokensin token upsert.workflows/scryfall/ingest.ts— invoke the step after bulk upsert, fold its stats in; lock release / staging cleanup still run viafinally.Testing
pnpm test. New coverage:lang/printed_namedefaults, JP mapping, and version sensitivity to each.ingestCollectorPrintings: happy path (links JP printings to English Cards),next_pagepagination, skip printings with no matching Card, and 404-as-empty (no Card lookup).card_tokens, notcard_token.upsertandcommitScryfallCheckpoint.