Repository navigation
upsertPricePoints can synthesize an event id from the wall clock, breaking dedupe #620
Description
Background
prisma/schema.prisma states the invariant the price-point identity index depends on:
sources with none — AMM/Soroswap pool snapshots, Aquarius, anything added later — get a key synthesized deterministically from the row's content, never from a clock.
synthesizeEventId honours that by hashing the row's content, including the timestamp:
const content = [ network, p.source, p.pairKey, p.poolId ?? '', String(p.ledger),
p.timestamp.toISOString(), String(p.price), ... ]#394 introduced a fallback in upsertPricePoints (src/db.ts):
const observedAt = p.observedAt ?? p.timestamp ?? new Date()
const timestamp = p.timestamp ?? observedAt
...
eventId: p.eventId || synthesizeEventId(network, { ...p, timestamp })When a point arrives with neither timestamp nor observedAt, timestamp becomes new Date() — so the synthesized event id now incorporates the wall clock. Two ingests of byte-identical data produce different event ids, the (network, source, eventId) identity index does not match, and skipDuplicates does not skip. The row is inserted twice.
That is the exact failure the identity work in #237 was built to make impossible, and it inflates volume and skews VWAP — silently, because nothing errors.
How it was found
Lens #385 (historical backfill) has a test that is the clearest statement of the contract:
proves idempotency on re-run: second run inserts 0 points and changes nothing
AssertionError: expected 1 to be +0
It passed on that branch against main as of this morning and fails after merging current main. The branch is not at fault — its test is the thing that noticed.
Why it is subtle
Before #394, a point with no timestamp would have thrown on p.timestamp.toISOString(). #394 turned a loud crash into silent non-determinism, which is the worse of the two.
What to build
- Decide what a point with no
timestampmeans. If every real ingester always sets one, the fallback should be an error rather than a clock read — a crash is recoverable, a duplicated money row is not. - If a clock fallback must stay, it must not reach
synthesizeEventId. The id has to be derived only from fields that are stable across runs, and a row with no stable temporal anchor cannot be given a deterministic identity at all — it should be rejected rather than assigned a random one. - Either way,
observedAtitself may keep thenew Date()default; it is "when Lens saw it" and is not part of the identity.
Acceptance criteria
- A point ingested twice with identical content produces the identical
eventId, with notimestampand noobservedAtsupplied - A test that fails against current
mainand passes after - The behaviour for a point with no temporal anchor is explicit — documented, and either rejected or given a non-clock identity
-
src/__tests__/backfill.test.ts's idempotency case passes in a fullnpm testrun, not only in isolation
Notes
Found while reviewing #385, which is blocked on it. That PR should not be asked to fix this — it is a regression in main.
scripts/migrations/2026-10-17-price-point-identity.sql and the @@unique in prisma/schema.prisma are the surrounding context.
Activity
sparkysolutions10-pixel commentedon Oct 25, 2026
Hi @Miracle656,
I would like to take this on. I reviewed the regression in upsertPricePoints (src/db.ts) where fallback
ew Date() introduces clock drift into synthesizeEventId and breaks idempotency deduplication.
Implementation plan:
Could you please assign this issue to me?
fix(db): prevent clock fallback in synthesizeEventId to maintain idem…
Merge upstream/main into feat/241-backfill-gap-detection