Skip to content

data-layout.md

Latest commit

 

History

History
167 lines (132 loc) · 5.61 KB

data-layout.md

File metadata and controls

167 lines (132 loc) · 5.61 KB
Raw

Data Layout Reference

This document defines the current filesystem/data-root artifacts for raw capture and implemented normalization.

Root Paths

The three data-root contexts are distinct:

  • Checkout commands use runtime.data_root from the checkout configuration; the committed example uses ./data. The repo-local data/ directory is ignored and supports local experimentation or a mount.
  • Direct, config-consuming installed CLI commands use runtime.data_root from the selected /etc/CryptoTrader/<instance>.yaml.
  • The installed service-worker does not take its storage root from that YAML value. systemd StateDirectory=market-recorder/%i passes its effective --data-root as /var/lib/market-recorder/<instance>.

For the default production instance, the generated YAML and systemd state directory normally name the same /var/lib/market-recorder/production path. That agreement does not make a YAML edit a worker-storage migration: changing runtime.data_root and restarting does not move or retarget installed worker storage.

Do not commit real market data.

Current Top-Level Layout

<data-root>/
  raw/
  normalized/
  normalized_chunks/
  ledger/
  manifests/
  logs/
  .tmp/
Directory Current role
raw/ Append-only, source-of-truth raw envelopes.
normalized/ Canonical trusted normalized Parquet tables.
normalized_chunks/ Provisional partition-clean normalization chunks.
ledger/ SQLite normalization coverage and publication state.
manifests/ Runtime-health and normalization-run metadata.
logs/ Logical logging boundary; the installed recorder emits stdout and stderr to journald rather than data-root log files.
.tmp/ Internal normalization staging and the normalization lock.

The current runtime creates and uses raw/, normalized/, normalized_chunks/, ledger/, normalization-manifest directories, and .tmp/ staging paths. The installed service writes runtime logs to journald.

Raw Data Layout

Raw data is the source of truth and is append-only.

raw/<source>/<transport>/<source_symbol>/<stream>/date=YYYY-MM-DD/hour=HH/
  part-<segment_start_utc>-<run_id>.jsonl.zst.open
  part-<segment_start_utc>-<segment_end_utc>-<run_id>.jsonl.zst

The implemented path builder sanitizes components into filesystem-safe names. Examples include markPrice@1s as markPrice_1s, depth@100ms as depth_100ms, and BTC/USD as BTC_USD.

Current source routes include:

raw/pyth/sse/MULTI/price_stream/date=YYYY-MM-DD/hour=HH/
raw/aster/ws/BTCUSDT/aggTrade/date=YYYY-MM-DD/hour=HH/
raw/aster/ws/BTCUSDT/bookTicker/date=YYYY-MM-DD/hour=HH/
raw/aster/ws/BTCUSDT/markPrice_1s/date=YYYY-MM-DD/hour=HH/
raw/aster/ws/BTCUSDT/depth20_100ms/date=YYYY-MM-DD/hour=HH/
raw/aster/ws/BTCUSDT/depth_100ms/date=YYYY-MM-DD/hour=HH/
raw/aster/rest/BTCUSDT/depth_snapshot_1000/date=YYYY-MM-DD/hour=HH/
raw/tradingview/webhook/ALL/alert/date=YYYY-MM-DD/hour=HH/

Raw File Lifecycle

.jsonl.zst.open   active writer-owned segment
.jsonl.zst        sealed segment for validation and reads
  • Raw files contain one JSON object per Zstandard-compressed line.
  • Writers create active .jsonl.zst.open files and never write directly to a sealed-looking path.
  • Sealing closes the Zstandard frame and atomically renames the active path to a sealed .jsonl.zst path.
  • Validators and readers treat sealed files as authoritative and skip or refuse active files.
  • The raw layer does not rewrite sealed files during normal operation.

Each route keeps an independent segment series. segment_start_utc identifies the route-local bucket; segment_end_utc records the actual seal time. A run ID prevents overwrites across process restarts. Rotation is resolved per source and logical stream from the configured age and size policy.

Normalization Layout

The installer, ops/install/install.sh, provisions the derived-data roots for an installed instance:

/var/lib/market-recorder/<instance>/
  normalized/
  normalized_chunks/
  ledger/
    normalization.sqlite
  manifests/
    normalization_runs/
    chunk_runs/
    compaction_runs/
  .tmp/
    chunking/
    compaction/
    normalization.lock

The systemd unit passes its installed state directory to service-worker; it uses this layout rather than provisioning it. The installer creates the generic derived roots; the normalizer creates table and partition directories at runtime.

Canonical normalized output paths are:

normalized/<table>/source=<source>/symbol=<canonical_symbol>/year=YYYY/month=MM/day=DD/data.parquet

Implemented normalized tables:

pyth_prices
trades
book_ticker
mark_price
l2_partial_depth
l2_diff_depth
l2_depth_snapshots

Normalization reads sealed .jsonl.zst files only and does not mutate raw files. normalized_chunks/ is provisional internal state; normalized/ is the canonical trusted dataset root. The compactor alone publishes canonical data.parquet files. ledger/normalization.sqlite records input discovery, coverage, compaction, and publication state. Staging occurs under .tmp/, whose normalization.lock prevents overlapping non-dry-run writers for one output root.

Manifest Layout

Runtime health manifests are emitted by src/market_recorder/service.py:

manifests/runtime/health-<run_id>.json

Normalization manifests are handled by src/market_recorder/normalize/manifests.py and are stored under:

manifests/normalization_runs/<normalization_run_id>.json

The installer also provisions manifests/chunk_runs/ and manifests/compaction_runs/ for the corresponding normalization metadata.