Skip to content

data-quality.md

Latest commit

 

History

History
153 lines (121 loc) · 16.5 KB

data-quality.md

File metadata and controls

153 lines (121 loc) · 16.5 KB
Raw

Lens Data Quality Guarantees

This document formally states the data quality guarantees provided by Lens: freshness, completeness, precision, venue inclusion, and the explicit system behaviors when any guarantee is violated. Every numerical bound documented below links directly to the code or configuration that enforces it.

Freshness Guarantees

Lens continuously aggregates order book trades and AMM liquidity pool snapshots across Stellar networks. The following freshness bounds are enforced:

Live Pricing Staleness Bound

A price response is considered fresh when its newest underlying trade observation occurred within 300 seconds (5 minutes). When the newest observation age exceeds 300 seconds, the response explicitly sets stale: true.

  • Enforcing Code: src/aggregator/vwap.ts (STALENESS_THRESHOLD_SECONDS)
  • Violation Behavior: The engine returns the last observed price alongside stale: true and lastTradeAgeSeconds: <age> (exceeding 300 seconds). Confidence rating degrades to 'low'.

Confidence Rating Freshness Tiers

Every pricing response carries a categorical confidence rating derived from observation age and venue diversity:

  • High Confidence: Requires an observation age under 30 seconds across at least 2 distinct sources with positive 24h volume.
  • Medium Confidence: Requires an observation age under 300 seconds.
  • Low Confidence: Assigned when observation age equals or exceeds 300 seconds.
  • Unknown Confidence: Assigned when no trades have ever been recorded (lastTradeAgeSeconds === null).
  • Enforcing Code: src/aggregator/vwap.ts

Venue Ingestion Liveness Window

Venue adapters report network liveness on /venues/registry. A venue is marked 'live' if a price point has been ingested within the last 300000 ms (5 minutes).

  • Enforcing Code: src/routes/venues.ts (LIVE_WINDOW_MS)
  • Violation Behavior: If no price points are observed within 300000 ms, venue liveness transitions to 'stale'. Disabled venues report 'disabled'.

Ingestion Polling Intervals

  • SDEX Order Book & AMM Trades: Horizon ingestion polls every 5000 ms with a batch page size of 200 records for SDEX and 200 records for AMM pools.
  • Soroswap AMM Reserves: Contract reserves poll via Soroban RPC every 60000 ms.
  • Enforcing Code: src/config.ts (pollIntervalMs, soroswapPollMs)

Cache Expiration & Network Timeouts

  • Redis Price Cache TTL: Aggregate prices are cached in Redis with a TTL of 10 seconds.
  • Outbound HTTP Timeout: External HTTP requests enforce a deadline timeout of 10000 ms.
  • Enforcing Code: src/config.ts (priceTtl, timeoutMs)
  • Violation Behavior: Outbound calls abort upon reaching 10000 ms; ingester logs errors and retries on subsequent poll ticks.

Snapshot and Partition Retention

Completeness Guarantees

Multi-Venue Liquidity Aggregation

High confidence requires price observations from at least 2 distinct liquidity sources (e.g. SDEX and AMM pools). When only 1 source is active, confidence is limited to 'medium'.

Statistical Outlier Rejection Thresholds

TWAP and VWAP computations apply statistical filtering to reject manipulation spikes:

  • Minimum Sample Count: Outlier filtering requires at least 4 sample points.
  • Interquartile Range (IQR): Rejects samples beyond Q1 - 1.5 * IQR or Q3 + 1.5 * IQR.
  • Modified Z-Score: Rejects samples with absolute score exceeding 3.5, using a consistency factor of 0.6745.
  • Enforcing Code: src/pricing/twap.ts
  • Violation Behavior: If fewer than 4 samples exist, filtering is bypassed (outlierRejected is zero). If 0 samples exist in the window, endpoints return zero TWAP and sample count.

Request Window & Parameter Bounds

  • TWAP Windows: Window ranges from 1 to 1440 minutes (default 60 minutes). Sample intervals range from 1 to 3600 seconds (default 60 seconds).
  • VWAP Windows: Window ranges from 1 to 1440 minutes (default 60 minutes).
  • Price History Limits: Bucket count is capped at 1000 buckets (default 100 buckets).
  • Screener Pair Limits: Pair count defaults to 20 and is clamped between 1 and 100.
  • Spread Observations: Venues must quote at least 2 observations in the window to report a spread.
  • Violation Behavior: Inputs outside allowed ranges trigger HTTP 400 Bad Request with validation details.

Precision Guarantees

Database Storage Precision

  • Prices and VWAPs: Stored as PostgreSQL DECIMAL(36, 18) with 18 decimal digits of fractional scale.
  • Stellar Native Reserves & Volumes: Stored as PostgreSQL DECIMAL(36, 7) matching Stellar's native 7 decimal place stroop scale (1 stroop = 0.0000001 XLM).
  • Enforcing Code: prisma/schema.prisma

API Boundary Serialization

  • Standard Default Mode: Returns numbers as IEEE-754 double precision floating-point numbers (~15-17 significant digits).
  • Exact String Representation Mode: When negotiated via X-Decimal-Format: string, Accept: application/json; decimals=strings, or ?decimals=strings, all decimal quantities serialise as exact strings without floating-point truncation.
  • Enforcing Code: prisma/schema.prisma, src/types.ts
  • Violation Behavior: Consumers that require exact 18-decimal precision must negotiate string format; unnegotiated requests receive standard JSON numbers for backward compatibility.

Venue Inclusion Guarantees

Lens integrates 4 registered venue adapters:

Venue Identifier Kind Testnet Mainnet Included In Enforcing Code
Stellar Decentralized Exchange (SDEX) sdex orderbook Enabled Enabled VWAP, Best Route, Depth, Book, Volumes, Spreads src/ingest/venues/sdex.ts
Stellar Native Liquidity Pools (AMM) stellar-amm amm Enabled Enabled Spot, Pools, Best Route, Depth, Book, Volumes src/ingest/venues/amm.ts
Soroswap AMM soroswap amm Enabled Enabled Price aggregation, multi-venue pools src/ingest/venues/soroswap.ts
Aquarius AMM aquarius amm Disabled Enabled Mainnet liquidity pool aggregation src/ingest/venues/aquarius.ts
  • Reflector Oracle: Used optionally for oracle benchmark comparison (/compare/:asset) and GraphQL oracle resolution. Disabled by default unless configured.
  • Enforcing Code: src/ingest/venues/index.ts, src/config.ts

Violation Semantics Summary

When a data quality guarantee is breached, Lens behaves deterministically:

  • Stale Observation (age >= 300 s): Returns the last known price marked with stale: true and downgraded confidence (low), preserving availability while warning callers.
  • Missing Ingestion / Empty Window (trades = 0): Returns confidence: "unknown" with lastTradeAgeSeconds: null, falling back through VWAP 1h -> VWAP 24h -> AMM Spot -> 0.
  • Unwatched Pair: Returns HTTP 404 Not Found with { "error": "Pair <A>/<B> not watched on <network>" }.
  • Invalid Query Parameters: Returns HTTP 400 Bad Request with structured Zod validation errors.
  • Outbound Failure / Ingestion Silence (>= 300000 ms): Venue liveness degrades from 'live' to 'stale'.
  • Rate Limit Exceeded: Returns HTTP 429 Too Many Requests.

Summary of Enforced Bounds

Parameter Bound Units Enforcing Code Description
staleness_threshold 300 seconds src/aggregator/vwap.ts:4 Ingestion age after which a price is flagged stale
high_confidence_max_age 30 seconds src/aggregator/vwap.ts:5 Max trade age for high confidence rating
medium_confidence_max_age 300 seconds src/aggregator/vwap.ts:6 Max trade age for medium confidence rating
venue_liveness_window 300000 ms src/routes/venues.ts:7 Silence interval after which a venue is marked stale
indexer_poll_interval 5000 ms src/config.ts:266 Polling interval for SDEX / Horizon trades
sdex_page_size 200 rows src/config.ts:267 Maximum trade records fetched per Horizon request
amm_page_size 200 rows src/config.ts:268 Maximum pool snapshot records fetched per page
soroswap_poll_interval 60000 ms src/config.ts:178 Polling interval for Soroswap smart contract reserves
price_cache_ttl 10 seconds src/config.ts:271 Redis cache TTL for pricing responses
http_timeout_ms 10000 ms src/config.ts:274 Outbound deadline timeout for external network requests
snapshot_retention_days 30 days src/jobs/snapshotRetention.ts:12 Retention window for historical pool snapshots
snapshot_retention_batch_size 1000 rows src/jobs/snapshotRetention.ts:14 Row limit for batch snapshot pruning statements
price_points_retention_days 30 days src/partitioning/pricePoints.ts:5 Retention window before daily partition drops
min_outlier_samples 4 samples src/pricing/twap.ts:288 Minimum samples required for statistical outlier rejection
iqr_multiplier 1.5 factor src/pricing/twap.ts:289 Interquartile range multiplier for outlier fence
modified_z_score_threshold 3.5 factor src/pricing/twap.ts:290 Modified Z-Score deviation threshold
modified_z_score_factor 0.6745 factor src/pricing/twap.ts:291 Consistency adjustment factor for MAD normal distribution
twap_window_min 1 minutes src/routes/price.ts:8 Minimum allowed TWAP calculation window
twap_window_max 1440 minutes src/routes/price.ts:9 Maximum allowed TWAP calculation window (24h)
twap_window_default 60 minutes src/routes/price.ts:10 Default TWAP window
twap_sample_interval_min 1 seconds src/routes/price.ts:11 Minimum sample interval for TWAP calculation
twap_sample_interval_max 3600 seconds src/routes/price.ts:12 Maximum sample interval for TWAP calculation
twap_sample_interval_default 60 seconds src/routes/price.ts:13 Default sample interval for TWAP calculation
vwap_window_min 1 minutes src/routes/price.ts:15 Minimum allowed VWAP calculation window
vwap_window_max 1440 minutes src/routes/price.ts:16 Maximum allowed VWAP calculation window (24h)
vwap_window_default 60 minutes src/routes/price.ts:17 Default VWAP window
history_default_limit 100 buckets src/api/rest.ts:24 Default bucket limit for historical aggregates
history_max_limit 1000 buckets src/api/rest.ts:25 Upper ceiling for historical aggregates
screener_default_limit 20 pairs src/services/screener.ts:16 Default limit of pairs returned by screener
screener_max_limit 100 pairs src/services/screener.ts:17 Maximum clamped limit of pairs returned by screener
spread_min_observations 2 observations src/routes/spreads.ts:20 Minimum observations required before a venue spread is reported
registered_venues_count 6 adapters src/ingest/venues/index.ts:23 Total number of registered venue adapters
database_price_precision 36 digits prisma/schema.prisma:20 Total numeric precision in PostgreSQL decimal columns
database_price_scale 18 decimals prisma/schema.prisma:20 Decimal places stored for pricing in PostgreSQL
stellar_native_scale 7 decimals prisma/schema.prisma:21 Native Stellar stroop scale (one stroop equals ten to the negative seven XLM)