Skip to content

upsertPricePoints can synthesize an event id from the wall clock, breaking dedupe #620

Closed

Description

@Miracle656

Background

prisma/schema.prisma states the invariant the price-point identity index depends on:

sources with none — AMM/Soroswap pool snapshots, Aquarius, anything added later — get a key synthesized deterministically from the row's content, never from a clock.

synthesizeEventId honours that by hashing the row's content, including the timestamp:

const content = [ network, p.source, p.pairKey, p.poolId ?? '', String(p.ledger),
                  p.timestamp.toISOString(), String(p.price), ... ]

#394 introduced a fallback in upsertPricePoints (src/db.ts):

const observedAt = p.observedAt ?? p.timestamp ?? new Date()
const timestamp  = p.timestamp ?? observedAt
...
eventId: p.eventId || synthesizeEventId(network, { ...p, timestamp })

When a point arrives with neither timestamp nor observedAt, timestamp becomes new Date() — so the synthesized event id now incorporates the wall clock. Two ingests of byte-identical data produce different event ids, the (network, source, eventId) identity index does not match, and skipDuplicates does not skip. The row is inserted twice.

That is the exact failure the identity work in #237 was built to make impossible, and it inflates volume and skews VWAP — silently, because nothing errors.

How it was found

Lens #385 (historical backfill) has a test that is the clearest statement of the contract:

proves idempotency on re-run: second run inserts 0 points and changes nothing
  AssertionError: expected 1 to be +0

It passed on that branch against main as of this morning and fails after merging current main. The branch is not at fault — its test is the thing that noticed.

Why it is subtle

Before #394, a point with no timestamp would have thrown on p.timestamp.toISOString(). #394 turned a loud crash into silent non-determinism, which is the worse of the two.

What to build

  • Decide what a point with no timestamp means. If every real ingester always sets one, the fallback should be an error rather than a clock read — a crash is recoverable, a duplicated money row is not.
  • If a clock fallback must stay, it must not reach synthesizeEventId. The id has to be derived only from fields that are stable across runs, and a row with no stable temporal anchor cannot be given a deterministic identity at all — it should be rejected rather than assigned a random one.
  • Either way, observedAt itself may keep the new Date() default; it is "when Lens saw it" and is not part of the identity.

Acceptance criteria

  • A point ingested twice with identical content produces the identical eventId, with no timestamp and no observedAt supplied
  • A test that fails against current main and passes after
  • The behaviour for a point with no temporal anchor is explicit — documented, and either rejected or given a non-clock identity
  • src/__tests__/backfill.test.ts's idempotency case passes in a full npm test run, not only in isolation

Notes

Found while reviewing #385, which is blocked on it. That PR should not be asked to fix this — it is a regression in main.

scripts/migrations/2026-10-17-price-point-identity.sql and the @@unique in prisma/schema.prisma are the surrounding context.

Activity

  1. sparkysolutions10-pixel commented on Oct 25, 2026

    @sparkysolutions10-pixel
    Contributor

    Hi @Miracle656,

    I would like to take this on. I reviewed the regression in upsertPricePoints (src/db.ts) where fallback
    ew Date() introduces clock drift into synthesizeEventId and breaks idempotency deduplication.

    Implementation plan:

    1. Update synthesizeEventId to accept optional imestamp?: Date and derive identity strictly from deterministic content fields without incorporating wall-clock time when imestamp is omitted.
    2. In upsertPricePoints, pass p.timestamp directly to synthesizeEventId so points lacking a temporal anchor produce identical hashes across replays.
    3. Add a dedicated regression test in src/tests/pricePointIdentity.test.ts verifying that replaying points with neither imestamp nor observedAt dedupes cleanly.
    4. Ensure full test suite passes with zero regressions.

    Could you please assign this issue to me?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions