You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current two-station (VFTS) data prep pipeline (metab_bayes_2s(),
prepdata_bayes_2s(), mm_validate_data_2station()) was built to match the
shape of the reference dataset used in the original methodology - one
continuous deployment, regular timesteps, no gaps, downstream-only
covariates, always fit jointly. Several review comments on PR #469
surfaced real gaps between that assumption and what real-world monitoring
data actually looks like, plus one architectural question that reframes
several of them. Collecting these as one follow-up scope, since they're
facets of the same underlying question: what does two-station data prep
need to support real deployment data, not just the reference dataset?
Known facets (not a commitment to any particular solution yet):
User-facing lag/shift step: moving the upstream/downstream lag
application out of internal-only logic (currently inside
prepdata_bayes_2s(), reaching into the previous date's raw rows) into an
exported preprocessing helper. This turns out to be the key piece several
of the items below depend on - see next item.
Per-day fitting: currently, two-station always fits every date
jointly in one Stan call, which was assumed to be a Stan-model
limitation. It isn't - it's a consequence of the lag-shift happening
inline. If the lag-shift were a separate preprocessing step (above), each
date's modeled rows would be self-contained, and per-day fitting (like
one-station's bayes_1ply()) should be possible. This would also likely be
a prerequisite for pool_K600 support, which needs the same per-day/
hierarchical structure one-station already has.
Gap tolerance / interpolation: two-station currently has no per-day
timestep-regularity check at all (one-station has one, though see prepdata_bayes()'s "modal timestep" check doesn't do what its name/comment claim #474
for a bug in it). The published methodology linearly interpolated
single-timestep gaps and dropped days with larger gaps; informal team
discussion suggested a 4-8 timestep (1-2hr) tolerance. Building this means
new interpolation logic in the shared mm_is_valid_day()/
mm_filter_valid_days() validation layer (currently strict accept-or-
reject-whole-day only, no data mutation) - not a two-station-only change.
Upstream/averaged covariates: light/depth/temp.water are currently
hardcoded to downstream-only. Real setups may measure these upstream, or
the team may prefer averaging paired upstream/downstream values.
Multi-deployment / phase-shifted timesteps: concatenated deployments
with internally-regular but mutually-misaligned timesteps (e.g. one
deployment logging at :01/:16/:31/:46, the next at :04/:19/:34/:49)
aren't currently distinguished from a real data gap.
Also worth folding into this pass, lower priority: a pre-existing
one-station quirk noticed while auditing failed-run handling -
bayes_allply()'s comment claims a failed run's mcmc object is preserved,
but the code path that would attach it is skipped on failure, so it isn't.
Not urgent, but worth reconciling if/when this area gets touched again.
These are related but distinct enough that they likely need scoping as
separate pieces of work once picked up, not one big PR. Filed to capture
the discussion and guide a future design pass, not to commit to a specific
approach now.
The current two-station (VFTS) data prep pipeline (metab_bayes_2s(),
prepdata_bayes_2s(), mm_validate_data_2station()) was built to match the
shape of the reference dataset used in the original methodology - one
continuous deployment, regular timesteps, no gaps, downstream-only
covariates, always fit jointly. Several review comments on PR #469
surfaced real gaps between that assumption and what real-world monitoring
data actually looks like, plus one architectural question that reframes
several of them. Collecting these as one follow-up scope, since they're
facets of the same underlying question: what does two-station data prep
need to support real deployment data, not just the reference dataset?
Known facets (not a commitment to any particular solution yet):
application out of internal-only logic (currently inside
prepdata_bayes_2s(), reaching into the previous date's raw rows) into an
exported preprocessing helper. This turns out to be the key piece several
of the items below depend on - see next item.
jointly in one Stan call, which was assumed to be a Stan-model
limitation. It isn't - it's a consequence of the lag-shift happening
inline. If the lag-shift were a separate preprocessing step (above), each
date's modeled rows would be self-contained, and per-day fitting (like
one-station's
bayes_1ply()) should be possible. This would also likely bea prerequisite for
pool_K600support, which needs the same per-day/hierarchical structure one-station already has.
timestep-regularity check at all (one-station has one, though see prepdata_bayes()'s "modal timestep" check doesn't do what its name/comment claim #474
for a bug in it). The published methodology linearly interpolated
single-timestep gaps and dropped days with larger gaps; informal team
discussion suggested a 4-8 timestep (1-2hr) tolerance. Building this means
new interpolation logic in the shared mm_is_valid_day()/
mm_filter_valid_days() validation layer (currently strict accept-or-
reject-whole-day only, no data mutation) - not a two-station-only change.
hardcoded to downstream-only. Real setups may measure these upstream, or
the team may prefer averaging paired upstream/downstream values.
with internally-regular but mutually-misaligned timesteps (e.g. one
deployment logging at :01/:16/:31/:46, the next at :04/:19/:34/:49)
aren't currently distinguished from a real data gap.
Also worth folding into this pass, lower priority: a pre-existing
one-station quirk noticed while auditing failed-run handling -
bayes_allply()'s comment claims a failed run's mcmc object is preserved,
but the code path that would attach it is skipped on failure, so it isn't.
Not urgent, but worth reconciling if/when this area gets touched again.
These are related but distinct enough that they likely need scoping as
separate pieces of work once picked up, not one big PR. Filed to capture
the discussion and guide a future design pass, not to commit to a specific
approach now.