[Riri-dev1]

fix(ingest): add rolling-median plausibility check and quarantine table (#238)

Summary

Fixes unvalidated price tick ingest where outlier rejection previously only occurred at read-time for TWAP/VWAP. Unvalidated ticks from thin pools could pollute raw price tables and trigger false alerts across /candles, /price, /screener, and webhooks.

Key Changes

1. Quarantine Table (Prisma Model):

   • Added QuarantinedPricePoint model mapped to quarantined_price_points in prisma/schema.prisma.
   • Preserves full tick payload (network, pairKey, price, baseVolume, counterVolume, source, poolId, ledger, timestamp, eventId) along with quarantine diagnostics (referenceMedian, rejectionReason, rejectionBound, quarantinedAt).
   • Indexes lead with network to maintain query efficiency and cross-chain isolation ((network, pairKey, timestamp) and (network, pairKey, quarantinedAt)).

2. Ingest-Time Plausibility Check:

   • Implemented partitionPricePointsByPlausibility and getRecentMedianPrice in src/ingest/plausibility.ts.
   • Incoming ticks are evaluated against a rolling median of recent price points before being written to price_points.
   • Configurable via environment variables with sensible defaults:
     • INGEST_PLAUSIBILITY_THRESHOLD (default: 0.20 = 20% max deviation)
     • INGEST_PLAUSIBILITY_WINDOW_MINUTES (default: 60)
     • INGEST_PLAUSIBILITY_MIN_SAMPLES (default: 1)
   • Ticks exceeding the threshold bound are diverted to quarantined_price_points and omitted from price_points.
   • Ingest venue runners (sdex, amm, soroswap, runner) only advance last-price caches and trigger webhook/WebSocket updates for accepted ticks, preventing false alerts.

3. Metrics per Source:

   • Added Prometheus counters in src/metrics.ts tagged by ['source', 'network']:
     • price_points_accepted_total
     • price_points_quarantined_total (aliased to price_points_rejected_total)
   • Incremented during ingestion for both accepted and quarantined ticks.

4. Tests:

   • Followed test-first methodology: proved defect via automated test in src/__tests__/ingestPlausibility.test.ts.
   • Covers 40% rapid upward swings, 40% downward crashes, quarantine payload integrity, exclusion from raw tables, last-price cache isolation, and metric counter tracking.

Related issue

Closes #238

Type of change

• Bug fix
• New feature
• Refactor
• Docs
• Tests
• CI / tooling

Checklist

• I have read CONTRIBUTING.md
• npx tsc --noEmit passes
• npm run build passes
• I added / updated tests where relevant
• I updated docs where relevant

[drips-wave]

@Riri-dev1 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Miracle656]

Merged — full CI green, including the migration drift & deploy job once I approved the fork run.

I like that this fails open: no reference median means isPlausible: true, so a cold window never silently drops real data, and rejects land in quarantined_price_points rather than being discarded. That makes a bad threshold recoverable instead of lossy. The migration is purely additive (CREATE TABLE plus two indexes, no ALTER on existing tables), which is what let it go in ahead of a deploy.

One operational note, not a code issue. The shared Postgres is in read-only mode right now (25006, cannot execute INSERT in a read-only transaction) and the Lens Render service is suspended, so this migration is committed but not applied. Lens boots with migrate deploy, so when the service is un-suspended the migration has to apply before ingest writes — on a read-only database it will not, and quarantinedPricePoint.createMany would then throw against a table that does not exist. Worth confirming the database is writable before bringing Lens back up.

The ?.createMany optional-chain guard does protect against a stale client, though not against a missing table.