fix(streaming,cdc): compare mixed tz-awareness in UTC; flag implausible inferred epoch event times

[kevincostner17]

Summary

This PR fixes two event-time bugs:

• Streaming crash: StreamingCleaner raised TypeError: Cannot compare tz-naive and tz-aware timestamps when a datetime column changed tz-awareness between batches.
• Wrong epoch reading: cdc_profile read small numbers such as row numbers 1..100 as epoch seconds, then reported a 56-year-old batch as passing with no warning.

Root cause

• Streaming: ColumnState._update_datetime (streaming/_state.py) folded each batch's datetime min/max into the running state with plain min()/max()/<. The time-series watermarks (TimeSeriesProcessor._late_mask) did the same. The first batch fixed whether the stored value was tz-naive or tz-aware, so a later batch of the other kind raised on comparison. That includes a naive batch followed by strings with mixed UTC offsets, which parse to UTC.
• CDC: infer_epoch_unit picks a unit from the median magnitude alone, so any number below 1e11 is read as seconds. cdc_profile then computed freshness, lateness and ordering from those instants as if they were real.

Behaviour change

• Streaming comparisons in UTC:
  • Streaming datetime statistics and watermarks now compare in UTC, reading naive values as UTC. This matches the rule tz-aware vs naive datetimes crash contracts, semantic checks, domain checks and cdc_profile #233/fix(timeseries): epoch and mixed-offset timestamps, MAD zero-scale, stable anomaly columns, cdc tz #360/fix(semantic): duplicate row labels, non-string column labels, tz-aware date checks #363 already use; the scalar helper reused is fieldcheck._as_utc.
  • Stored values keep their arrival zone, so streams whose batches are all naive or all tz-aware produce the same output.
  • In time-series mode, a batch whose timestamp or event-time column changes awareness gets a warning, and state_ marks that column tz_mixed.
• Implausible epochs:
  • When the unit is inferred and the median event time is before 1990-01-01, cdc_profile reports an event_time_implausible error. It skips freshness (freshness_seconds is None), lateness and ordering. Key, duplicate, operation and missing-value checks still run.
  • Time-series cleaning warns about the same kind of column but still parses it.
  • An explicit event_time_unit=/timestamp_unit= still forces the unit and skips the check.
  • Real epoch columns in s/ms/us/ns dated 1990 or later behave exactly as before.

Default-output changes

• cdc_profile, for a numeric event-time column whose unit is inferred and whose median is before 1990-01-01 (row numbers, counters, zeros, negatives, small floats, or real epoch values from 1970–1989):
  • New defect kind event_time_implausible at level error, with details keys inferred_unit, median and plausible_from. The DefectKind literal gains "event_time_implausible".
  • passed goes True → False, and n_errors rises by 1.
  • freshness_seconds changes from a float to None, and summary() omits the freshness line.
  • For that column, late/out_of_order/stale defects are no longer reported, and replay_risk counts late rows as 0.
  • The freshness and ordering trust penalties are 0.0.
  • to_frame(), to_dict() and to_json() gain the new defect row.
• fd.clean_timeseries / time-series StreamingCleaner, for a numeric timestamp or event-time column with an inferred unit and a median before 1990-01-01: a new report.warnings entry. Parsing, rows and actions are unchanged.
• StreamingCleaner with batches of mixed tz-awareness:
  • It no longer raises TypeError. This input used to crash.
  • In time-series mode, each batch that changes awareness adds a report.warnings entry.
  • state_["columns"][col]["datetime"] gains "tz_mixed": True, only for columns that saw both kinds.
• Genuine pre-1990 numeric epoch data: event times stored as integer epochs before 1990-01-01 (e.g. archival data from 1985) now fail cdc_profile with event_time_implausible unless event_time_unit= is passed explicitly. datetime64 columns are unaffected.
• Exit codes: unchanged. The streaming CLI exits on the trust gate only, and no CLI calls cdc_profile.
• Unchanged: streams whose batches are all naive or all tz-aware, and epoch columns dated 1990 or later.

Tests

• tests/test_streaming_state.py:
  • naive→aware and aware→naive min/max/ordering
  • UTC comparison against a non-UTC zone
  • no tz_mixed key when awareness never changes
• tests/test_streaming_timeseries.py:
  • naive→aware and aware→naive batches through StreamingCleaner: late-row counts, the watermark in its arrival zone, the awareness warning, and state_["columns"][col]["datetime"]["tz_mixed"]
  • UTC watermark comparisons for naive→Asia/Kolkata and Asia/Kolkata→naive
  • a naive batch followed by strings with mixed offsets
  • the row-number timestamp warning and its explicit timestamp_unit override
  • no warning for real epochs in s/ms/us/ns
  • the warning for a row-number event-time column
• tests/test_cdc_profile_fixes.py:
  • row numbers 1..100 give event_time_implausible, passed=False and no freshness
  • time checks are skipped while key, duplicate and missing checks still run, with a nullable Int64 column, with and without a watermark
  • the 1990-01-01 boundary in all four units
  • zero, negative and float offsets
  • a real epoch-seconds column is unchanged
  • an explicit event_time_unit="s" still reads small numbers as epochs

Verification

• ruff check .: passed
• mypy src/freshdata: no issues in 205 source files
• pytest -m "not online and not large":
  • Python 3.12, pandas 2.3.3: 6284 passed, 14 skipped
  • Python 3.9, pandas 1.5.3: 6255 passed, 18 skipped

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 16d217b2-2a10-4a45-b323-17e3345f80f9

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)