[Noobb-byte]

Summary

Implements historical backfill capability and persistent ingestion gap detection for Lens:

• Persistent Gap Tracking: Added the IngestionGap Prisma model to store ingestion holes per (network, source, pair) with statuses 'open', 'filled', and 'permanent'.
• Idempotent Gap Detection & Merging: Implemented cadence-based hole detection (detectGaps) and merging logic (recordGaps) that combines overlapping or adjacent intervals into single continuous spans without duplicate records. Handles pair inversions (e.g. XLM/USDC and USDC/XLM) and splits partially filled gaps.
• Historical Backfill Service & CLI: Added src/backfill/backfillService.ts and scripts/backfill.ts (npm run backfill) to refetch trade history for history-capable sources (SDEX, AMM) via Horizon. All refetched data writes through the same idempotent upsertPricePoints path as live ingestion, preventing duplicate points on re-runs.
• Permanent Gaps (Non-history Sources): Sources without historical trade logs (soroswap_amm, aquarius_amm) automatically record or transition gaps to 'permanent' and exit cleanly without failing noisily.
• Incomplete Window Declarations: Time-window query endpoints (/prices/history, /candles/*, /price/*/*history, /price/twap/*, /price/vwap/*) declare incomplete: true along with overlapping gaps: [...] whenever the requested window intersects open or permanent gaps. Windows that are clean or contain only filled gaps omit these fields.
• Specs & Documentation: Updated OpenAPI specifications (openapi.yaml and openapi.json), REST response schemas, and README.md with gap cadences, CLI command usage, and incomplete response formats.

Related issue

closes #241

Type of change

• Bug fix
• New feature
• Refactor
• Docs
• Tests
• CI / tooling

Checklist

• I have read CONTRIBUTING.md
• npx tsc --noEmit passes
• npm run build passes
• I added / updated tests where relevant
• I updated docs where relevant

[drips-wave]

@Noobb-byte Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Miracle656]

Not merging yet — and the reason is a bug in main, not in your branch. I have filed it as #620.

What happened

Your branch passed cleanly against main as of this morning: clean merge, tsc clean, 1,616 tests. After merging current main it fails on one test — yours, and the one that matters most:

proves idempotency on re-run: second run inserts 0 points and changes nothing
  AssertionError: expected 1 to be +0

It passes in isolation and fails in a full run, which is what made it worth chasing rather than dismissing as flake.

The cause

prisma/schema.prisma states the invariant the identity index rests on:

sources with none … get a key synthesized deterministically from the row's content, never from a clock

synthesizeEventId hashes p.timestamp.toISOString() among other fields. #394, merged a couple of hours ago, added this to upsertPricePoints:

const observedAt = p.observedAt ?? p.timestamp ?? new Date()
const timestamp  = p.timestamp ?? observedAt
eventId: p.eventId || synthesizeEventId(network, { ...p, timestamp })

A point with neither timestamp nor observedAt now gets the wall clock folded into its event id. Two runs over identical data produce different ids, the (network, source, eventId) index does not match, skipDuplicates does not skip, and the row is inserted twice.

Your backfill is simply the first thing to re-ingest identical data deliberately, so it is the first thing to notice. Please do not try to work around it here — #620 is where it gets fixed, and a workaround in the backfill would hide a problem that affects ordinary ingestion too.

Two things I did resolve, so you do not have to redo them

• src/db.ts conflicted between your deterministic id derivation and feat: separate observation and source time (#240) #394's observedAt/sourceTime handling. I unioned them: id: p.id ?? (p.eventId ? \${network}:${p.source}:${p.eventId}` : undefined)` alongside the new fields. Deriving the row id from the event identity so a re-run collides on the primary key is a good second line of defence behind the unique index — worth keeping.
• src/backfill/backfillService.ts imported ../ingesters/toid, which moved to ../ingest/toid when Refactor/ingest consolidation 225 #255 consolidated the ingesters this afternoon.

What I would like

Rebase once #620 lands, and keep that idempotency test exactly as it is. It is the most valuable assertion in the PR: it states the contract in a sentence, it proved itself by catching a regression from a different PR entirely, and a weaker version — asserting only that the second run "succeeded" — would have passed while silently duplicating rows.

One note for the rebase: run the full npm test, not just the backfill file. This failure is invisible in isolation.

[Miracle656]

Merged into main (28af638). You were right to wait rather than work around it — a8f33c9 landed this morning and unblocked this properly.

Why the block mattered. The backfill refetches points through upsertPricePoints, and until this morning the synthesized event id borrowed new Date() when a point had no timestamp. Re-running a backfill would therefore have produced a different identity for the same observation each time, slipping past the unique index as a fresh row. Had you worked around it locally, the workaround would have hidden the bug rather than fixed it, and the duplicates would have shown up in production. Thanks for holding.

One fix on merge, and it was your harness rather than your code.

Your own idempotency test — "second run inserts 0 points and changes nothing" — was failing. The cause is in the createMany fake:

const exists = storedPoints.some(
  p => p.id === id && p.timestamp.getTime() === item.timestamp.getTime()
)

That models the composite primary key @@id([id, timestamp]) only. The schema also declares:

@@unique([network, source, eventId], map: "price_point_identity")

and createMany({ skipDuplicates: true }) skips a row violating either constraint. A re-run whose timestamp falls back to now lands on a different primary key while the event id — the thing that actually identifies the observation — stays the same, so the real database absorbs it and the fake did not.

The fake now checks both constraints, and all 13 backfill tests pass. The feature was idempotent the whole time; the test was measuring the wrong index.

Worth taking from that: when a fake stands in for a database, the constraints it simulates are part of the contract under test. A fake that models the primary key but not the uniqueness rule your feature depends on will fail you in exactly the case the feature exists for.

One thing I did not change, and filed instead: #625. The backfill modules read prisma and pgPool off globalThis, which src/db.ts publishes unconditionally. I understand why — importing ../db from gaps.ts breaks six suites that mock ../config partially, because db.ts evaluates config.db.url at load. I tried two ways round it during the merge and reverted both (the second broke 33 suites), so it needs a deliberate fix rather than an in-flight one. The issue records what I tried and three options. It is not urgent, but it is worth doing before anything else starts reading those globals.

tsc clean, 1,791 tests pass.

https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB