Skip to content

[integrations] Enrichment worker: in-place LLM metadata for un-enriched thoughts + Readwise stamping - #454

Open
eazene wants to merge 18 commits into
NateBJones-Projects:mainfrom
eazene:contrib/eazene/enrichment-worker
Open

[integrations] Enrichment worker: in-place LLM metadata for un-enriched thoughts + Readwise stamping#454
eazene wants to merge 18 commits into
NateBJones-Projects:mainfrom
eazene:contrib/eazene/enrichment-worker

Conversation

@eazene

@eazene eazene commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • New enrichment-worker Edge Function: drains thoughts with enrichment_status NULL/'pending'/'fallback'/'skipped' through the canonical extractMetadata cascade, in place, via an atomic FOR UPDATE SKIP LOCKED claim RPC with a 10-min lease
  • Merge policy (unit-tested, 15 tests): top-level source_type immutable; Readwise rows keep type='reference' (classifier opinion → metadata.classified_type); native tags/topics/people/action_items preserved whole with LLM additions capped to the room under 12 (fail-closed coercion for non-string list shapes); classifier fallback writes tracking keys only — never fallbackMetadata content
  • Circuit breaker: provider outages / budget exhaustion abort the tick, clear claims, and consume zero row attempts (incl. single-row transport-class failures); synthesis artifacts (generator-keyed metadata) are excluded from the queue and derived-thought writers stamp enrichment_status='exempt' + provenance
  • readwise-capture stamps enrichment_status='pending' + fire-and-forgets a single-id enrich; import-readwise.py stamps the same
  • extractMetadata gains additive _enrichment_error reasons so callers can distinguish run-level from row-level failures
  • 15-min pg_cron tick (x-brain-key header auth — key deliberately not in the URL); backfill.sh drain loop with 5-strike outage cooldown; auditor reports un-enriched: N (M stuck-at-max) in its hygiene line (auditor repo copy also reconciled with live source and its install docs fixed to the MCP_ACCESS_KEY header pattern)

Review

  • Every task implemented+reviewed via subagent-driven development (spec + quality gates), then a whole-branch review, then two independent external model reviews (Codex, Grok) whose findings (synthesis-row protection, native-tag preservation, single-row attempt burn, upsert_thought portability) are all fixed in this branch
  • Deployed and operating live prior to PR: worker + claim RPC + cron + webhook verified end-to-end; a real multi-provider outage exercised the circuit breaker and cooldown paths in production

Test plan

  • deno test --allow-read — 15/15 (merge policy incl. preservation/coercion edge cases + vendored-_shared drift guard)
  • Claim RPC: double-claim under lease returns disjoint rows
  • Live: seeded Readwise-shaped row enriches with native metadata intact, type pinned reference
  • Live: circuit breaker validated during a real provider outage (zero attempts consumed)
  • Secret scan across all changed files: clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH

eazene and others added 17 commits July 15, 2026 14:36
@eazene
[integrations] enrichment-worker: queue view + atomic claim RPC
6f804cd
@eazene @claude
[integrations] enrichment-worker: merge policy (TDD) + extractMetadat…
6fc05dd 
…a error reasons

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] enrichment-worker: Edge Function (claim loop + circuit…
26b4e8c 
… breaker)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] readwise: stamp enrichment_status=pending + fire-and-f…
4d9ae96 
…orget enrich

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] enrichment-worker: 15-min pg_cron tick
08c5a4b 
Key is passed via the x-brain-key header rather than a ?key= URL param
so the secret does not persist in cron.job, cron.job_run_details, or
request URL logs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene
[integrations] enrichment-worker: correct security comment (header ke…
e2c703a 
…eps key out of URL logs only)
@eazene @claude
[integrations] enrichment-worker: backfill drain loop
d4a4b5c 
Drain thoughts_needing_enrichment by repeatedly invoking the worker.
Stops cleanly only when remaining==0; treats remaining<0 (count-query
sentinel) or non-numeric as transient and retries; exits 1 on circuit
break; retries on curl failure. Progress line every 10 iterations.
No set -x — the access key is never echoed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[recipes] editorial-policy: reconcile auditor repo copy with live source
b100bda 
The repo copy predated the live Tier-1 hygiene block. Mirror the deployed
source exactly: import the canonical derived-thought-writer, add the
lint_hygiene_summary fetch + hygiene/llm_error surface, switch to
claude-haiku-4-5 with JSON-fence stripping, and consolidate auth onto
MCP_ACCESS_KEY. Also vendor the _shared/derived-thought-writer.ts the
function bundles so the repo is deployable in isolation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[recipes] editorial-policy: auditor reports un-enriched + stuck counts
0b2cf0a 
Fold the Task 1 enrichment-backlog views into the auditor's Tier-1 hygiene
surface. fetchHygiene() now head-counts thoughts_needing_enrichment and
thoughts_enrichment_stuck (before the LLM pass, cheap, always computed) and
merges needs_enrichment / enrichment_stuck (-1 on null) into the hygiene
object. The mechanical hygiene line gains, in the same middot style:
un-enriched: N (M stuck-at-max). Deployed as auditor v14; verified live the
rendered line reads 'un-enriched: 8540 (1 stuck-at-max)'.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] enrichment-worker: backfill survives transient circuit…
5d8035e 
… breaks (5-strike cooldown)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] enrichment-worker: contribution docs
aba6662 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[recipes] editorial-policy: align auditor auth docs with MCP_ACCESS_K…
3517a74 
…EY + fix '?' render

Finding 1: auditor code authenticates on MCP_ACCESS_KEY (x-brain-key header)
but the recipe docs still told installers to use the retired AUDITOR_ACCESS_KEY
via ?key= URL param — a fresh install would 401 forever. Update README env
list, credential tracker, smoke-test + troubleshooting invocations, and
schedule.sql to the header form; refresh the required-env header comment in
index.ts.

Finding 2: enrichment head-counts are stored as -1 on failure (never null),
so the `?? "?"` render fallback never fired — failures rendered as -1. Add a
showN() helper that renders '?' for negative values on the two enrichment
fields, and correct the stale comment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] enrichment-worker: trim unused deps from deno.json
825f5d2 
deno.json was copied from enhanced-mcp and shipped @hono/mcp,
@modelcontextprotocol/sdk, hono, and zod — none of which the worker imports.
The @supabase/supabase-js map entry was also dead: index.ts imports the full
npm:@supabase/supabase-js@2 specifier, so the bare-key pin never resolved.
Empty the imports map. Tests unchanged (11/11).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] enrichment-worker: preserve native lists; guard single…
86096b8 
…-row transport failures

Fix B (native list preservation): buildCompletePatch now keeps the native
base list WHOLE (never capped/dropped) and appends LLM extras only into the
room left under the cap of 12. Adds coerceStringList() to handle string,
string[], and Readwise {name}/{tag} object shapes, failing closed (raw base
kept verbatim, no merge) on any unrecognizable value. Applied uniformly to
topics/tags/people/action_items. TDD: 4 new tests + updated cap test.

Fix C (single-row transport failures): a lone claim has no circuit probe, so
a transport-class fallback (fatal_provider_error / transient_failures_exhausted
/ all_providers_failed) on the only row now trips the breaker — clears the
claim, burns zero attempts, reports circuit_broken — instead of consuming an
attempt. Multi-row batches keep per-row fallback semantics (probe proves
provider health; cron batches provide the stuck-escalation path).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] enrichment-worker: exclude synthesis artifacts from th…
54beadf 
…e queue (schema v2)

Fix A (defense-in-depth): both the thoughts_needing_enrichment view predicate
and the claim_thoughts_for_enrichment RPC predicate now add
`AND NOT (metadata ? 'generator')`. Synthesis artifacts (audit reports,
briefings, wiki pages) carry a generator key and must never be re-classified;
writers also stamp status 'exempt'. This is the schema-independent belt.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[recipes] editorial-policy: stamp derived thoughts exempt + portable …
a2f6cbe 
…upsert

Fix A (writer + auditor): storeDerivedThought now forces
metadata.enrichment_status='exempt' (belt) and stamps derivation_layer='derived'
+ derivation_method='synthesis' on the row (suspenders, via a best-effort
follow-up UPDATE so a missing provenance column can't fail a synthesis run).
The auditor's audit_report metadata also carries enrichment_status='exempt'
explicitly.

Fix D (portability): the writer attempts the optional 3-arg
upsert_thought(content,payload,embedding); on PGRST202 / 42883 it falls back to
the canonical 2-arg upsert_thought and sets the embedding in the follow-up
UPDATE.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@eazene @claude
[integrations] enrichment-worker: document synthesis-artifact exempti…
8de2ccc 
…on + backfill

Adds a "Synthesis artifacts (never enrich these)" README section explaining the
three-layer exemption (exempt status, derived provenance columns, generator-key
predicate) and a documented-not-run enumerated-or-predicate backfill UPDATE for
pre-existing synthesis rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
@github-actions github-actions Bot added integration Contribution: MCP extension or capture source recipe Contribution: step-by-step recipe labels Jul 15, 2026
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown

OB1 PR Gate

✅ Folder structure — All files are in allowed directories
✅ Required files — README.md and metadata.json found in all contribution folders
✅ Metadata valid — All metadata.json files passed JSON Schema validation
✅ No credentials — No API keys, tokens, or secrets detected
✅ SQL safety — No destructive SQL or core table modifications
✅ Category artifacts — Required file types present for each category
✅ PR format — Title follows [category] Description format
✅ No binary blobs — No oversized or binary files
❌ README completeness — Incomplete READMEs:

  • recipes/readwise-import/README.md: missing sections: Step-by-step instructions

✅ Contribution dependencies — All declared skill and primitive dependencies exist and are linked in README
✅ LLM clarity review — Covered by Claude PR Review workflow
✅ Remote MCP pattern — No local MCP server patterns detected — uses remote MCP correctly
❌ Tool audit link — Missing tool audit guide link:

  • integrations/enrichment-worker/README.md: missing link to MCP Tool Audit & Optimization Guide. Required for extensions and integrations that expose MCP tools.
  • integrations/readwise-capture/README.md: missing link to MCP Tool Audit & Optimization Guide. Required for extensions and integrations that expose MCP tools.
    See the extension template for the recommended format.
    ✅ Scope check — All changes are within the contribution folder(s)
    ✅ Internal links — All relative links in READMEs resolve to existing files

Result: 13/15 checks passed. Please fix the issues above and push again.

Post-Merge Tasks

These don't block merge — they're reminders for admins after this PR lands.

@eazene @claude
[integrations] enrichment-worker: guard against legacy non-object met…
20a7fdb 
…adata

Legacy rows storing metadata as a JSON string/array were array-ified by the
claim RPC's jsonb || (scalars concat as single-element arrays) and then
spread into numeric-keyed objects by the merge (2026-07-15 incident: 48
rows mangled, all repaired; 50 string-shaped rows normalized). Guard both
layers: the RPC wraps non-object metadata verbatim under legacy_metadata,
and asObjectMetadata() does the same in every merge builder. 17/17 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxP7M99m4EF5Ve7rn3SNKH
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

integration Contribution: MCP extension or capture source recipe Contribution: step-by-step recipe

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@eazene