[epic] interactive-lookup-audio — Phase 1 — Make it interactive: click-to-lookup pipeline, word cards, per-word audio

[julianken]

North star

The reading-pedagogy law (DESIGN.md) makes the word card the product's sole surface for a word's sound and meaning — the conversation never teaches inline, so if the card is thin, the product cannot teach at all. Today a click yields transliteration, a wire-carried gloss, and morphology; there is no audio, no hover tier, no dictionary, and no lookup pipeline behind it. This epic turns every L2 word into a full pedagogical object: instant verified contextual gloss, morphological breakdown, and real per-word audio that always speaks the full vocalized form — proven on both instances.

Scope

In

• Three-tier /enrich lookup pipeline: Tier 0 deterministic NLP (lemma/POS/morphology from each pack's analyzer behind the shared analyze(text, lang) → Token[] interface) → Tier 1 open-data dictionary candidate senses → Tier 2 LLM as constrained sense selector only when genuinely ambiguous. Cache keys: glosses on (lemma, sense_id, dump_version); contextual glosses on (lemma, sense_id, context_hash).
• Generation-time sense_id self-annotation so tokens carry glossRef/audioRef cache keys and a click is a cache hit — sub-100 ms perceived target on the common path, no LLM round-trip.
• The three-rung disclosure stack per the v2 plans: hover/focus quick-peek (translit + gloss + audio) → word-card popover on click (vocalized headword, contextual meaning, audio, "I know this", "add to list") → "more →" margin detail (ar: root+pattern morphology, POS/feats, word family; zh: character/tone breakdown). All click/hover-only — the reading line never changes.
• Real per-word TTS replacing seam 7's canned stubs in BOTH packs: audio always speaks the FULL vocalized form even when the display is bare; playable for GREEN-verified words only; synthesized from verified forms only (ar: the verified full vocalization, never an engine auto-diacritizer; zh: verified surface-pinyin driving phoneme-level control, pinned voice versions).
• TTS provider selection as a child-issue decision: the hard requirement is word-timing marks (the karaoke highlight in polish-scale disqualifies timing-less providers); the provider named in the older docs (Azure) is among Retire old-stack terms (Cloudflare Pages / Neon / Azure / split-origin subdomains) across docs #51's stale-vs-planned terms — re-validate, don't assume.
• Audio storage + CDN with content-hash keys sha256(text+lang+voice+model+rate); background pre-warm for newly-introduced L2 words.
• Dictionary licensing + attribution footer from day one (CC-BY-SA sources; zh: CC-CEDICT-class; the ar Tier-1 source is TBD — the third source named in architecture.md §5, JMdict, belongs to a retired-language example).
• Gloss confidence/coverage gating whether a word is safe to render in L2 at all: dictionary as candidate generator, LLM as constrained selector, never free-generating an L2 form.
• Learner's own words get the same clickable treatment via a lighter user-turn token schema; "I know this" fires the vocab-advance learning event and a word-click is a soft negative — both emitted over the existing learning-event channel.

Out

• Accounts, live composer, datastore, and durable persistence of the events this epic emits — app-shell (the two epics interleave; these surfaces render over replay/corpus turns until it lands).
• What the signals do — word_state + FSRS, status-wash changes, blend movement — learner-model-blend.
• Correction of learner turns — writing-correction (this epic only makes their tokens clickable).
• Karaoke word-timing highlight and full-sentence playback — polish-scale (this epic only requires the chosen provider to emit timing marks).
• Live-LLM generation enablement and blend quality — generation-quality (the sense_id self-annotation changes the generation schema; coordinate there — it must work replay-first).
• Deeper verification legs, human-calibration loops, corpus growth — content-verification-depth.
• ASR / speaking practice — an explicit product non-goal — ASR / speaking practice is out of product (language-agent is a reading + writing tutor; speech is a different modality with its own verification problem), recorded here as a decided boundary, not an unowned orphan; revisitable post-release only as a deliberate scope expansion.

Current state (honest)

All verified on main today:

• A click-a-word breakdown panel already exists in both packs: src/packages/lang-ar/src/react/ArWordToken.tsx + ArBreakdownPanel.tsx (translit, gloss, root/pattern morphology) and src/packages/lang-zh/src/react/ZhWordToken.tsx + ZhBreakdownPanel.tsx (per-character rows, bound-morpheme gloss suppression). Its data comes from the gate-verified wire annotations, not a lookup pipeline. No hover quick-peek, no audio, no "I know this" / "add to list", no margin "more →" rung.
• The annotation schemas already carry sense-shaped fields the self-annotation can key on: ar lemma + gloss (src/packages/lang-ar/src/annotations.ts); zh senseId/gloss plus the two load-bearing pinyin layers citationPinyin/surfacePinyin (src/packages/lang-zh/src/annotations.ts) — surface-pinyin is the layer that must drive TTS.
• Seam 7 (TtsDriver, src/packages/contract/src/ports.ts) already types synthesize on VerifiedToken[], making "TTS from unverified text" unrepresentable — but both packs ship canned stubs (ar-tts-stub/v1 in src/packages/lang-ar/src/pack.ts; zh-tts-stub/v1 in src/packages/lang-zh/src/pack.ts).
• No /enrich route exists. The server's whole surface is GET /api/stream (?fixture= / ?turn=), POST /api/turns, and POST/GET /api/learning-events (src/apps/server/src/http/server.ts).
• The learning-event return channel is live end to end — contract LearningEvent (src/packages/contract/src/scaffold.ts) + the fire-and-forget web sink (src/apps/web/src/shell/learning-events.ts) — but nothing stores the events durably (no datastore exists; that is app-shell's).
• Both verification gates are real and offline-at-runtime (ar: dediac-roundtrip + calima-msa-r13 analyzer-membership; zh: deterministic legs only), so "GREEN-verified" is a real, checkable property for audio eligibility.
• Vocabulary is a sliver: 60 ar entries (src/packages/lang-ar/assets/seed-vocab.json), 58 zh tuples (src/packages/lang-zh/assets/seed-tuples.json). NB: opening the zh token universe (e.g. a second reading per surface form arriving with dictionary-scale senses, or learner-writing lookups) is exactly walking-skeleton follow-up: zh neural gate legs hot (g2pW + AS-HanLP) + sidecar re-measure #22's wake trigger — the deterministic-only zh sidecar stops being sufficient at that point.
• The proof infrastructure the done-criteria lean on already runs: the [fake, ar, zh] conformance TCK (src/packages/lang-conformance, scripts/conformance-leg.sh) and the Playwright e2e suite (src/apps/web/e2e/).
• One DESIGN.md tension to reconcile when audio ships: the word-card-popover spec lists audio among the card's contents, while the "Pronunciation surfaces" note still says audio playback is deferred with the speaking feature. This epic un-defers playback, so DESIGN.md (and tokens.json where tokens change) must be reconciled in the shipping PRs.
• No audio storage/CDN exists; infra is the Phase 0 walking skeleton (Cloud Run + Worker apex router). R2/CDN is named in plans only.

Suggested work items

Suggestions, not tickets — implementation issues are authored (and plan-review-gated) at epic kickoff; nothing below is committed scope.

• /enrich endpoint + the three-tier pipeline (Tier 0 pack analyzer → Tier 1 dictionary senses → Tier 2 constrained LLM selector), with both cache keys and the never-free-generate constraint under test.
• Dictionary source selection + ingestion per pack (zh: CC-CEDICT-class; ar: TBD — evaluate Kaikki/Wiktionary-class sources), dump versioning, and the CC-BY-SA attribution footer on both instances.
• Generation-time sense_id self-annotation in the structured-output schema + replay fixtures (coordinate with generation-quality; must work replay-first — no live key required).
• Quick-peek hover/focus tier (keyboard-accessible per DESIGN.md a11y; touch skips straight to the card).
• Word-card upgrade: the existing breakdown panels grow into the full card (vocalized headword, contextual meaning, audio, "I know this", "add to list") plus the "more →" margin-detail rung. Figma frames first, per the design-SoT lockstep policy (see the two Figma GAPS rows).
• TTS provider decision spike: word-timing-marks requirement, per-word cost, voice-version pinning; re-validate the architecture.md §5 Azure recommendation against Retire old-stack terms (Cloudflare Pages / Neon / Azure / split-origin subdomains) across docs #51's stale-vs-planned judgment pass.
• Real TtsDriver in both packs: ar synthesizes the verified full vocalization (never an auto-diacritizer); zh drives phoneme-level control from verified surface-pinyin with pinned voices; GREEN-only playability enforced.
• Audio blob storage + CDN with sha256(text+lang+voice+model+rate) keys; background pre-warm on newly-introduced words; ops handoff notes to platform-ops-hardening.
• Gloss confidence/coverage gate wiring (a low-confidence gloss blocks rendering that word in L2 at all).
• Lighter user-turn token schema so learner-authored words are clickable; wire "I know this" (vocab advance) and word-click (soft negative) through the existing /api/learning-events channel with stable token IDs.
• Conformance TCK + Playwright coverage: the disclosure stack and the audio seam proven on ar (RTL + ḥarakāt) and zh (no-space segmentation, ruby); no live provider calls on PR-blocking paths (record/replay for TTS, mirroring the LLM convention).
• zh scope check against walking-skeleton follow-up: zh neural gate legs hot (g2pW + AS-HanLP) + sidecar re-measure #22's wake trigger: decide at kickoff whether Phase-1 dictionary senses stay inside the curation-gated token universe or the neural legs must go hot first.
• DESIGN.md reconcile: un-defer audio playback in "Pronunciation surfaces"; spec the quick-peek/card/margin states in §0 + tokens.json in the same PRs.

Dependencies

• Depends on no epic to start (links live in the master epic table). Interleaves with app-shell: durable persistence of the learning events rides there; until the composer lands, every surface here renders over replay/corpus turns — expected, not a defect.
• Coordinates with generation-quality on the sense_id self-annotation (it touches the generation-time structured-output schema and the replay fixtures).
• learner-model-blend consumes the events this epic emits; agree the event shape before that epic starts.
• May fire walking-skeleton follow-up: zh neural gate legs hot (g2pW + AS-HanLP) + sidecar re-measure #22's wake trigger (zh token-universe opening) depending on how dictionary senses are scoped — resolve at kickoff, not discovered after.
• External / operator: a TTS provider account + key (provisioned like the LLM key, via GCP Secret Manager); dictionary dump acquisition and license review (CC-BY-SA obligations).

Done criteria

• Clicking any GREEN-verified L2 word in either live instance opens the word card with a vocalized headword, a contextual (sense-selected) gloss, a morphological breakdown, and playable audio — and meaning/pronunciation appear nowhere inline (reading-pedagogy law upheld; the boundary gates stay green).
• The common click path is a cache hit via self-annotated refs — measured with a real number against the sub-100 ms perceived target (record the honest measurement even if the target is missed and re-negotiated).
• Audio speaks the FULL vocalized form when the display is bare: demonstrated on an ar bare-skeleton token (audio fully vocalized) and a zh token (audio driven by verified surface-pinyin). No synthesis path accepts unverified text (type-level plus a test).
• Both packs' seam-7 stubs are gone; the real drivers pass the conformance TCK on the [fake, ar, zh] matrix, and the Playwright e2e suite covers all three disclosure rungs on both instances.
• Tier 2 never emits an L2 form absent from Tier 1's candidate set (tested); low-confidence glosses block L2 rendering of the word.
• The attribution footer renders on both instances wherever dictionary data surfaces.
• "I know this" and word-click events arrive at /api/learning-events keyed to stable token IDs (durable storage explicitly not required here — that is app-shell).
• DESIGN.md / tokens.json reconciled for every surface this epic ships (card states, quick-peek, audio un-deferral), with Figma frames per the lockstep policy.

References

• docs/plans/architecture.md — §8 Phase 1 (the phase definition), §1–§2 (hot-path shape: per-word enrichment is lazy-on-click from cache, never on the streaming path), §5 (build-vs-buy: dictionary tiering, the TTS word-timing-marks requirement), §7 risks 4–6 (latency, wrong-sense glosses, code-switch TTS cliff).
• DESIGN.md — the reading-pedagogy law, word-card-popover / word-card-detail, elevation (the popover is the only shadowed element), a11y (focus-visible peek; touch goes straight to the card).
• docs/plans/v2-design-plan.md — the "I know this" confirm as the vocab-axis advance event.
• docs/plans/v2-journeys-plan.md — J1-S7/S8/S9 (the three disclosure rungs), US-8.2 (audio matches the vocalized form, GREEN-only).
• Open issues this epic interacts with (not owned or duplicated here): walking-skeleton follow-up: zh neural gate legs hot (g2pW + AS-HanLP) + sidecar re-measure #22 (zh neural legs — its wake trigger can be fired by this epic's dictionary scope) and Retire old-stack terms (Cloudflare Pages / Neon / Azure / split-origin subdomains) across docs #51 (Azure is a stale-vs-planned term — the TTS provider child issue lands with or after its judgment pass).
• GAPS.md rows: "zh neural gate legs hot (g2pW + AS-HanLP)" (the wake trigger), "Figma SoT catch-up" and "Figma MCP write access" (new UI surfaces owe frames under the lockstep policy).

---

Skeleton epic — part of the release spine (see the master epic). Drafted by an agent session 2026-07-04 and fresh-context reviewed before posting.

[julianken]

AGENT: Part of the release spine — master epic #61. This is a skeleton epic: work items are suggestions; implementation issues are authored (and plan-review-gated) at kickoff.