[integrations] Fix smart-ingest non-dry-run commit bugs - #473
Open
symy1610 wants to merge 2 commits into
Open
Conversation
…rojects#379, NateBJones-Projects#468) Two bugs blocked every real (non-dry-run) smart-ingest write: 1. UUID/bigint mismatch (NateBJones-Projects#379). upsert_thought's v_id is UUID (schemas/enhanced-thoughts/schema.sql), serialized as a JSON string, but extractThoughtId() only accepted numbers via Number.isFinite(). Every successful insert was read back as "no thought_id" and marked failed, even though the row landed in thoughts. ingestion_items.matched_thought_id / result_thought_id were also declared bigint, so they couldn't hold a UUID even if the client accepted one. append_thought_evidence(bigint, ...) had the same problem — comparing a bigint parameter against thoughts.id (uuid) can't succeed. Fix: extractThoughtId() now accepts UUID strings (numbers still accepted for bigint-id deployments), matched_thought_id/result_thought_id are now uuid columns (idempotent migration included, safe because the bug meant these were always NULL in practice), and append_thought_evidence takes uuid. 2. OpenRouter extraction failures on instruction-shaped content (NateBJones-Projects#468). response_format: json_object is an OpenAI-shaped parameter that the default OpenRouter model (an Anthropic model) silently ignores, so a model could still respond with prose instead of JSON — reproduced with HTTP 500 "OpenRouter returned invalid JSON" on a real ingest. Fix: retry once with an explicit "return only JSON" correction before failing, per the issue's suggested fix NateBJones-Projects#1. Left a comment explaining why response_format isn't a reliable guard on its own. Verified: deno check and deno lint pass on integrations/smart-ingest (the one pre-existing lint warning and fmt diff are unrelated to this change, confirmed present on origin/main). Not deployed/tested against a live Supabase instance as part of this change — recommend running against a real instance per CONTRIBUTING.md before merge. Tested: reproduced both failures against a live OB1 deployment prior to this fix (HTTP 200 with added_count: 0 / failed_count: 1 for NateBJones-Projects#379; HTTP 500 extraction_failed for NateBJones-Projects#468); this PR fixes the root causes identified in that repro and in the linked issues, but has not yet been re-verified end to end against a live instance since the fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Hey @sydney-moutia — welcome to Open Brain Source! 👋 Thanks for submitting your first PR. The automated review will run shortly and check things like metadata, folder structure, and README completeness. If anything needs fixing, the review comment will tell you exactly what. Once the automated checks pass, a human admin will review for quality and clarity. Expect a response within a few days. If you have questions, check out CONTRIBUTING.md or open an issue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the two bugs that block every real (non-dry-run)
smart-ingestwrite, tracked in #379 and #468.#379 — UUID/bigint mismatch on
upsert_thoughtreturn.upsert_thought'sv_idisUUID(schemas/enhanced-thoughts/schema.sql), serialized as a JSON string.extractThoughtId()inintegrations/smart-ingest/index.tsonly accepted numbers (Number.isFinite), so every successful write was read back as "no thought_id" and the item markedfailed— even though the row had already landed inthoughts.ingestion_items.matched_thought_id/result_thought_idwere also declaredbigint, so they couldn't hold a UUID even if the client accepted one, andappend_thought_evidence(bigint, ...)compared abigintparameter againstthoughts.id(uuid), which can't succeed.Fix:
extractThoughtId()accepts UUID strings (numeric ids still accepted, for bigint-id forks).ingestion_items.matched_thought_id/result_thought_idare nowuuidcolumns, with an idempotent migration for existing installs (safe — the bug meant these were alwaysNULLin practice).append_thought_evidencenow takesp_thought_id uuid; the oldbigintoverload is dropped rather than left dangling.#468 — OpenRouter extraction fails on instruction-shaped content.
response_format: { type: "json_object" }is an OpenAI-shaped parameter; the default OpenRouter model (an Anthropic model) silently ignores it, so it can still respond with prose instead of JSON — reproduced live as an HTTP 500"OpenRouter returned invalid JSON".Fix: per the issue's suggested fix #1, retry once with an explicit "return only JSON" correction before failing. Left a comment on
response_formatexplaining it isn't a reliable guard on its own.What it requires
smart-ingesttoday: Supabase + pgvector,enhanced-thoughtsschema applied (forupsert_thought), OpenRouter/OpenAI/Anthropic key.schemas/smart-ingest/schema.sqlon an existing install.Testing
Repro'd both failures against a live OB1 deployment before this fix:
dry_run) POST → HTTP 200,added_count: 0, failed_count: 1;ingestion_items.error_message = 'upsert_thought returned no thought_id'.{"error":"Extraction failed","reason":"extraction_failed"};ingestion_jobs.error_message = 'OpenRouter non-transient failure (no fallback): OpenRouter returned invalid JSON'.For this change itself:
deno checkanddeno lintpass onintegrations/smart-ingest(the one pre-existing lint warning in_shared/helpers.tsand thedeno fmtdiff onindex.tsboth predate this PR — confirmed present onorigin/mainbefore these edits, unrelated to this change). I was not able to redeploy and re-verify end-to-end against a live Supabase instance as part of this PR (no test project available in this environment) — flagging that per CONTRIBUTING.md's testing guidance, and happy to verify against a real instance if a maintainer can point me at one, or if the original reporter (who offered to send a PR in #379) wants to confirm independently.Closes #379, and fixes the OpenRouter side of #468 for
smart-ingestspecifically (the issue also covers the general "no guard on non-JSON responses" observation, which this PR's retry addresses for this call site).