Skip to content

Two-station data prep: support real deployment data (gaps, multi-deployment, upstream covariates, per-day fitting) #475

Description

@mdodrill-usgs

The current two-station (VFTS) data prep pipeline (metab_bayes_2s(),
prepdata_bayes_2s(), mm_validate_data_2station()) was built to match the
shape of the reference dataset used in the original methodology - one
continuous deployment, regular timesteps, no gaps, downstream-only
covariates, always fit jointly. Several review comments on PR #469
surfaced real gaps between that assumption and what real-world monitoring
data actually looks like, plus one architectural question that reframes
several of them. Collecting these as one follow-up scope, since they're
facets of the same underlying question: what does two-station data prep
need to support real deployment data, not just the reference dataset?

Known facets (not a commitment to any particular solution yet):

  • User-facing lag/shift step: moving the upstream/downstream lag
    application out of internal-only logic (currently inside
    prepdata_bayes_2s(), reaching into the previous date's raw rows) into an
    exported preprocessing helper. This turns out to be the key piece several
    of the items below depend on - see next item.
  • Per-day fitting: currently, two-station always fits every date
    jointly in one Stan call, which was assumed to be a Stan-model
    limitation. It isn't - it's a consequence of the lag-shift happening
    inline. If the lag-shift were a separate preprocessing step (above), each
    date's modeled rows would be self-contained, and per-day fitting (like
    one-station's bayes_1ply()) should be possible. This would also likely be
    a prerequisite for pool_K600 support, which needs the same per-day/
    hierarchical structure one-station already has.
  • Gap tolerance / interpolation: two-station currently has no per-day
    timestep-regularity check at all (one-station has one, though see prepdata_bayes()'s "modal timestep" check doesn't do what its name/comment claim #474
    for a bug in it). The published methodology linearly interpolated
    single-timestep gaps and dropped days with larger gaps; informal team
    discussion suggested a 4-8 timestep (1-2hr) tolerance. Building this means
    new interpolation logic in the shared mm_is_valid_day()/
    mm_filter_valid_days() validation layer (currently strict accept-or-
    reject-whole-day only, no data mutation) - not a two-station-only change.
  • Upstream/averaged covariates: light/depth/temp.water are currently
    hardcoded to downstream-only. Real setups may measure these upstream, or
    the team may prefer averaging paired upstream/downstream values.
  • Multi-deployment / phase-shifted timesteps: concatenated deployments
    with internally-regular but mutually-misaligned timesteps (e.g. one
    deployment logging at :01/:16/:31/:46, the next at :04/:19/:34/:49)
    aren't currently distinguished from a real data gap.

Also worth folding into this pass, lower priority: a pre-existing
one-station quirk noticed while auditing failed-run handling -
bayes_allply()'s comment claims a failed run's mcmc object is preserved,
but the code path that would attach it is skipped on failure, so it isn't.
Not urgent, but worth reconciling if/when this area gets touched again.

These are related but distinct enough that they likely need scoping as
separate pieces of work once picked up, not one big PR. Filed to capture
the discussion and guide a future design pass, not to commit to a specific
approach now.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status
    In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions