Add log-triangular (ltriangle) distribution - #169
joethorley wants to merge 24 commits into
Conversation
Add a log-triangular distribution where log(concentration) follows a symmetric triangular distribution with mode `locationlog` and half-width `scalelog`. Wired in following the `llogis` pattern: - R/ltriangle.R: ssd_pltriangle/qltriangle/rltriangle/eltriangle, the base symmetric-triangular helpers (p/q/rtriangle_ssd), the concentration-scale helpers (pltriangle_ssd/qltriangle_ssd) used in model averaging, and sltriangle starting values. - src/TMB/ll_ltriangle.hpp: TMB negative log-likelihood (CondExp-based CDF, soft floor on the density) registered in ssdtools_TMBExports.cpp. - dist_data: ltriangle row (bcanz=FALSE, tails=FALSE, npars=2, valid=TRUE, bound=FALSE) - bounded support, no tails, no bounded parameter. - ssd_pmulti/qmulti/rmulti and params.R documentation extended. - tests/testthat/test-ltriangle.R mirrors test-llogis.R (test_dist plus snapshots); test-dists.R lists updated; affected snapshots regenerated. - distributions.Rmd gains a log-triangular section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…triangle Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nded-support tests and docs Follow-up to #169. - sltriangle() falls back to a small positive half-width when the data have no spread on the log scale, avoiding a -Inf starting value. - Add tests that ssd_hc()/predict() are finite and monotonic across the full 1-99% range for the bounded-support distribution. - Note in the distributions vignette that ltriangle is experimental and not part of the default BCANZ set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to #169. The soft density floor in the log-triangular likelihood was an absolute 1e-8, which is not scale-consistent across data magnitudes. It is now 1e-8 * scalelog (a fraction of the peak density argument) so the soft barrier behaves the same regardless of the magnitude of the data. Fits where the support covers the data are unchanged (the floor never binds), so no existing snapshots change. Adds a scale-invariance test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…distribution # Conflicts: # tests/testthat/test-ltriangle.R
Fits the log-triangular to the public endosulfan acute toxicity dataset (Hose and Van den Brink 2004) from fitdistrplus, the kind of data the US EPA SSD Toolbox fits a (log-)triangular distribution to. Asserts the MLE parameters and HC5 (skips when fitdistrplus is not installed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Claude Code Review
Claude Code Review is paused for this repository. To reconnect it, an admin of this repository's GitHub organization (or the account owner, for personal repositories) who can also manage your Claude organization's Code Review settings needs to re-link GitHub in Code Review settings. This is a one-time step.
Tip: disable this comment in your organization's Code Review settings.
PR #169 review findings —
|
| Check | Result | Attribution |
|---|---|---|
devtools::document() sync |
PASS (with caveat) | Local artifact, not a PR defect |
devtools::check() |
1 ERROR, 1 NOTE | Both pre-existing/environmental |
devtools::test() |
FAIL 3 / PASS 1358 / SKIP 39 | Pre-existing on dev — not this PR |
| Snapshot stability | PASS | No ltriangle snapshot rewritten |
air format --check . |
PASS | Clean, exit 0 (air.toml present) |
1.1 Docs in sync — PASS with caveat
document() rewrote only NAMESPACE, and only as a formatting change: the committed file uses
multi-line importFrom(pkg, a, b) blocks; my locally installed roxygen2 8.0.0 collapses them to
one-per-line importFrom(pkg,a). No export()/S3method() line changed, and all four ltriangle
exports are present and correct:
export(ssd_eltriangle) export(ssd_pltriangle) export(ssd_qltriangle) export(ssd_rltriangle)
Discarded with git checkout -- NAMESPACE man/. No action for the author — this is a roxygen2
version skew on my machine. (Worth noting DESCRIPTION carries no RoxygenNote field, so there is
nothing pinning the expected roxygen version.)
1.2 devtools::check() — 1 ERROR, 1 NOTE
checking whether package 'ssdtools' can be installed ... [57s] OK— the TMB backend including
ll_ltriangle.hppcompiles cleanly. No compiler warnings surfaced.checking examples ... [20s] OK,re-building of vignette outputs ... [20s/22s] OK
(so the newdistributions.Rmdsection knits).- ERROR —
checking tests: the 3test-censor.Rsnapshot failures below. - NOTE — hidden files:
Found the following hidden files and directories: .git. Artifact of
checking from a git worktree. Not a PR issue.
1.3 devtools::test() — 3 failures, all pre-existing
FAILURE 'test-censor.R:47:3' Snapshot of `path` has changed.
FAILURE 'test-censor.R:54:3' Snapshot of `path` has changed.
FAILURE 'test-censor.R:61:3' Snapshot of `path` has changed.
[ FAIL 3 | WARN 0 | SKIP 39 | PASS 1358 ]
Root cause: the committed snapshot header is
Chemical,Species,Conc,Group,Units,Medium,right; the regenerated one is
Chemical,Species,Conc,Group,Units,right. My installed ssddata 1.0.0 has
ccme_boron columns Chemical,Species,Conc,Group,Units — no Medium column.
Attribution confirmed: I built a second worktree at upstream/dev (ff47b9ce) and ran the
censor tests there — identical 3 failures, [ FAIL 3 | PASS 30 ]. This is an ssddata version
mismatch in my environment, not caused by PR #169. (CI installs ssddata from
open-AIMS/ssddata, which presumably still has Medium.) No action for the author.
1.4 Snapshot stability — PASS
Running the suite rewrote no ltriangle-related snapshot. _snaps/ltriangle.md,
_snaps/estimates.md, _snaps/multi.md and _snaps/data/dist_data.csv were all stable — so the
snapshots committed in the PR are deterministic and current. The only churn was the 3 censor
.new.csv files (above) and deletion of .png snapshots, which happens because plot tests are
skip_on_os-skipped on Linux and testthat prunes them as unused. All discarded with
git checkout -- tests/testthat/_snaps.
1.5 Air — PASS
air format --check . exits 0 with no output. The PR is fully Air-formatted.
2. Phase 2A — code-reading findings
Overall the PR follows the llogis pattern closely and faithfully; the wiring is consistent,
alphabetically ordered, and complete. Divergences found are concentrated in the two places where
bounded support genuinely differs from llogis.
F1 — R/pqr.R:116-124 · quantile endpoints ignore bounded support (non-blocking, but new class of bug)
.qd() hard-codes the extreme quantiles before ever dispatching to the distribution's own function:
if (p == 0) { if (!.lgt) return(0); return(-Inf) }
if (p == 1) { return(Inf) }
do.call(fun, args = args)This is correct for every pre-existing distribution (all have support unbounded above, and →0
below). ltriangle is the first distribution in the package with bounded support, so it is the
first to violate the assumption. Measured on locationlog=0, scalelog=sqrt(6):
| returned | correct (= its own limit) | |
|---|---|---|
ssd_qltriangle(0) |
0 |
exp(-sqrt(6)) = 0.086338 |
ssd_qltriangle(1) |
Inf |
exp(+sqrt(6)) = 11.582435 |
ssd_qltriangle(1e-12) |
0.08633793 |
— (converges correctly) |
So the function is discontinuous at its own endpoints: it converges to 0.0863 as p→0 then
jumps to 0 at exactly p=0. Note this is a defect in shared infrastructure, not in
R/ltriangle.R — flagging it as something the PR exposes rather than introduces. Practical
blast radius is small (ssd_hc/ssd_hp don't evaluate at p=0/1), which is why nothing else caught
it. Fix would be to let qdist defer to the distribution when it has finite support.
F2 — src/TMB/ll_ltriangle.hpp:86-93 · the density floor is an attractor, not a barrier (BLOCKING)
Detailed in §3 with numerical evidence. Summary: the "soft floor" makes the negative log-likelihood
diverge to −∞ as scalelog → 0 once observations fall outside the candidate support, so it
rewards exactly the configuration it was meant to penalise.
F3 — src/TMB/ll_ltriangle.hpp:95-102 · censored branch has no floor at all (blocking-adjacent)
The uncensored branch is floored; the censored branch is not:
nll -= weight(i)*log(pright-pleft); // no guardIf a censored interval lies entirely outside the candidate support, ptri returns the same value at
both ends, pright-pleft == 0, and log(0) → -Inf → non-finite objective. Reproduced: fitting
interval-censored data with an interval far below the cluster fails with
L-BFGS-B needs finite values of 'fn'. This asymmetry is inherited structurally from ll_llogis.hpp
(which also has no guard) — but for llogis the CDF is strictly increasing everywhere, so the
difference is never exactly 0. Bounded support makes it reachable.
F4 — dist_data flags — all correct ✅
ltriangle, bcanz=FALSE, tails=FALSE, npars=2L, valid=TRUE, bound=FALSE.
bound=FALSEis genuinely correct, and the prompt's suspicion is resolved:R/data.R:52
documentsboundas "Whether one or more parameters have boundaries", consumed via
is_bounds()(R/helpers.R:133) to apply L-BFGS-B box constraints.ltriangle's parameters are
locationlog(unconstrained) andlog_scalelog(unconstrained, exp-transformed). Parameter
bounds ≠ support bounds. Correct as written.tails=FALSEis defensible (matchesinvpareto, the other non-tailed distribution) and has
no computational effect —tailsis used only as a filter inssd_dists()(R/dists.R:52-53).
Minor wording tension:R/data.R:49glossestailsas "has both tails", and a symmetric
triangular does have two (finite) tails. Reading it as "has both infinite tails" is what makes
FALSEright. Not worth changing, possibly worth a doc clarification.
F5 — Model-averaging wiring — correct ✅
ssd_pmulti/ssd_qmulti/ssd_rmulti each gain ltriangle.weight=0, ltriangle.locationlog=0,
ltriangle.scalelog=3, alphabetically placed between lnorm_lnorm and weibull, matching the
existing convention exactly. R/params.R:199-201 documents all three. pltriangle_ssd /
qltriangle_ssd mirror pllogis_ssd / qllogis_ssd precisely.
Verified end-to-end: averaging lnorm + llogis + ltriangle on ccme_boron works, and ltriangle
takes weight 0.706 (best AIC, log_lik = -116, delta = 0).
F6 — R/ltriangle.R:116-145 · base helpers — correct, minor divergence in the good direction ✅
ptriangle_ssdCDF branches verified against closed form.qtriangle_ssd:z = sqrt(2p) - 1forp<=0.5,1 - sqrt(2(1-p))above — exactly van
Straalen's inverse.rtriangle_ssdusesrunif(n) + runif(n) - 1: the sum of two U(0,1) is triangular on [0,2] with
mode 1, shifted to [-1,1] mode 0. Variance2 × 1/12 = 1/6, soVar = scale²/6— matches the
SD = scale/√6identity. Clean and correct.- Minor divergence:
ptriangle_ssd/qtriangle_ssdreturnrep(NaN, length(q))onscale <= 0
whereplogis_ssd/qlogis_ssdreturn scalarNaN. The ltriangle version is more correct
(length-preserving). No action; noting only because it is a deliberate departure from the template.
F7 — R/ltriangle.R:100-106 · starting value — sound ✅
halfwidth <- max(abs(logx - location)) guarantees the initial support covers the data, with a
1.1× cushion; the <= 0 fallback to 0.1 handles zero-spread data and is covered by a test. This is
a real improvement over blindly mirroring sllogis. (It does not, however, prevent the optimiser
from later leaving the covering region — see §3.)
F8 — TMB gradient at the mode — acceptable, worth a comment (nit)
absdev = CondExpGe(dev, 0, dev, -dev) gives |dev|, so the density has a slope sign-flip at the
mode and the log-likelihood is piecewise-smooth with kinks at each observation. This is inherent
to the triangular family, not a defect — L-BFGS-B tolerates it and every fit I ran converged. The
kink set has measure zero in the parameter space. Worth one comment in the header acknowledging the
non-smoothness so a future reader doesn't assume smooth-optimiser guarantees.
F9 — Tests — stronger than the llogis template ✅
test-ltriangle.R (178 lines) does more than mirror structure. Genuinely valuable additions:
test_dist("ltriangle")+ the standard p/q/r snapshots (template parity);- scale-invariance test —
Conc * 1000shiftslocationlogbylog(1000)and leaves
scalelogunchanged (this is the test that justifies the relative floor); - monotonic finite HC across
1:99/100, and finitepredict()across the full range; - zero-log-spread starting-value edge case;
- two external-reference fits: endosulfan (fitdistrplus, Hose & Van den Brink 2004) and a
US EPA SSD Toolbox permethrin reproduction (EPA/600/R-18/116 Table 2) with provenance and a public
domain note. Reproducing the EPA Toolbox'sa/bsupport endpoints in log10 is a nice check.
Gap: no test covers the failure modes in §3 — no test fits data where the optimum wants to leave
the covering region, and none covers censored data (the left < right TMB branch is untested for
this distribution).
F10 — Vignette — accurate, missing provenance (nit)
distributions.Rmd pdf (s-|y-μ|)/s² and the two-branch cdf are both correct and match the
implementation. Framing is honest — explicitly says "experimental", "not part of the default BCANZ
set", "only fitted when explicitly requested".
Flagged as requested: the section cites only Wikipedia. No origin reference for the
distribution in an SSD context — van Straalen (2002) and Stephan et al. (1985) are the
natural citations, and the US EPA SSD Toolbox is already cited in the test comments but not in the
user-facing docs. Recommend adding them.
3. Phase 2B — analytic verification battery
3.1 Result: 14/14 PASS
API introspected first: ssd_qltriangle(p, locationlog=0, scalelog=3, lower.tail, log.p);
probe ssd_qltriangle(0.5, locationlog=0, scalelog=sqrt(6)) returned exactly 1.00000000, so these
functions work on the concentration scale and comparisons were taken through log().
| Check | Actual | Expected | Result |
|---|---|---|---|
q(0.50) = mode |
0.000000 | 0.000000 | PASS |
q(0.05) [van Straalen headline] |
−1.674893 | −1.674893 | PASS |
q(0.025) |
−1.901767 | −1.901767 | PASS |
q(0.10) |
−1.354045 | −1.354045 | PASS |
q(0.25) |
−0.717439 | −0.717439 | PASS |
symmetry q(0.95) = −q(0.05) |
1.674893 | 1.674893 | PASS |
F(mode) |
0.500000 | 0.5 | PASS |
F(q0.05) |
0.050000 | 0.05 | PASS |
F(lower support) |
0.000000 | 0 | PASS |
F(upper support) |
1.000000 | 1 | PASS |
E[log X] (n=2e5) |
−0.000423 | 0 | PASS |
Var[log X] (n=2e5) |
0.994099 | 1 | PASS |
| RNG support respected | min −2.4407, max 2.4397 | within ±2.44949 | PASS |
| ordering: ltri most conservative | −1.6753 < −1.6450 < −1.6212 | ltri < lnorm < llogis | PASS |
The reviewer-relevant claim holds: at matched log-scale mean/SD the bounded triangular is more
conservative in the lower tail than the infinite-tailed families
(ltri −1.6753 vs lnorm −1.6450 vs llogis −1.6212; analytic −1.6749 / −1.6449 / −1.6232).
The distribution math (p/q/r) is correct. ssd_qltriangle reproduces van Straalen's −1.675 to
six decimal places. No FAIL to interpret.
Reminder, as instructed: van Straalen's fitted
h/q/HC values were deliberately not
used as targets. He fits by nonlinear least-squares on empirical plotting positions; ssdtools fits
by maximum likelihood. Only Appendix A's estimator-independent closed-form properties were
asserted, and every check above calls the distribution functions directly with no fitting, so
§4.1 isolates the p/q/r implementation from the estimation machinery. Differences between his
published fits and an ssdtools refit would be an expected estimator difference, not a bug.
3.2 The estimation machinery is where the problem is
Because §3.1 isolates p/q/r, I ran a separate battery against the fitting path. Normal use is
healthy — on ccme_boron, ltriangle fits (locationlog=2.5096, scalelog=2.8316, support
[0.725, 208.8] covering data [1, 70.7]), gives HC1/HC5/HC10/HC20 = 1.082 / 1.774 / 2.571 / 4.344,
bootstrap CIs work (se=0.605, lcl=1.33, ucl=3.87, 100 boot), and model averaging works.
ssd_hp() returning 0 below the support bound is correct for a bounded family, not an artefact.
But the floor breaks down under outliers. Replicating the TMB objective exactly
(ll_ltriangle.hpp:84-93), for an observation outside the candidate support the contribution is:
log(1e-8 · s) − 2·log(s) − log(y) = log(1e-8) − log(s) − log(y)
As s → 0, −log(s) → +∞. So the log-likelihood increases without bound and the nll diverges
to −∞, when for a true triangular it should be +∞ (density is exactly 0 out there).
Measured, with locationlog parked far from all data so every point is floored:
log_scalelog=-2 nll= 563.78
log_scalelog=-5 nll= 470.78
log_scalelog=-20 nll= 5.78
log_scalelog=-50 nll= -924.22 <- lower nll = "better" fit
log_scalelog=-200 nll= -5574.22
The floor is an attractor, not a barrier. Two further consequences, both measured:
- Zero gradient in
locationlog. In the floored regime the objective is independent of
locationlog— nll is identically604.293499atlocationlog= 40, 50, 60, 70. Once a point
is outside the support, nothing pulls the mode back toward it. - The relative floor removes its own guard at the limit.
floor = 1e-8 * scalelogunderflows
to exactly0whenscalelogunderflows, solog(0)→-Inf/NaN:
log_scalelog=-745 → nll=Inf;-746 → nll=NaN. This is the direct source of the observed
L-BFGS-B needs finite values of 'fn'fit failures. (An absolute floor does not have this
particular underflow, but diverges faster at−2·log(s), so it is not the fix either — commit
76496b21made the divergence rate slower, not absent.)
Observable behaviour, tight cluster of 30 points at ~10 plus one low outlier:
| data | fitted support | outcome |
|---|---|---|
ccme_boron (well-behaved) |
[0.725, 208.8] |
0/28 excluded ✅ |
| 30 tight + outlier ×1e−2 | [6.36, 15.27] |
1/31 excluded |
| 30 tight + outlier ×1e−4 | [6.36, 15.27] |
1/31 excluded |
| 30 tight + outlier ×1e−6 | [6.36, 15.27] |
1/31 excluded |
| 30 tight + outlier ×1e−10 | — | fit fails |
The fitted support is identical regardless of how extreme the outlier is — the signature of the
zero gradient in (1). And the transition is n-dependent, exactly as the algebra predicts
(excluding a point costs −log(1e-8) ≈ 18.42; shrinking scalelog by factor k gains 2n·log k,
so exclusion becomes profitable around n ≈ 14):
| n | fitted support | excluded |
|---|---|---|
| 5 | [4.6e−09, 5.2e+07] |
0/6 ✅ |
| 10 | [1.9e−08, 1.5e+08] |
0/11 ✅ |
| 20 | [8.35, 13.19] |
2/21 |
| 40 | [7.62, 13.79] |
1/41 |
| 80 | — | fit fails |
Guideline-relevant consequence. On a 31-point dataset with one sensitive outlier, silently and
with no warning:
ltriangle fitted support = [7.494, 15]; data min = 1e-07; points below support = 1
ltriangle HC5 = 8.36301
lnorm HC5 = 0.0270088
ssd_hp() at the excluded observation (1e-07) = 0 %
HC5 is ~310× less protective, because the fit discarded the most sensitive species and then
reported that species as having zero probability of occurring. ssd_fit_dists() emits no warning
that the fitted support fails to cover the data. For an SSD package this is the failure mode that
matters most, and it is precisely the one that mirroring llogis could never surface.
Mitigating context, and why I'd still call this fixable rather than fatal: bcanz=FALSE keeps this
out of the default guideline path, the vignette labels it experimental, and every well-behaved
dataset I tried (including the PR's own endosulfan and permethrin fixtures) fits correctly.
4. Categorisation
Blocking
- B1 — The density floor rewards leaving the support (
ll_ltriangle.hpp:86-93). The nll diverges
to −∞ asscalelog → 0once points are floored, so the "soft barrier" is an attractor. Observable
as (a) silent fits whose support excludes data, with HC5 ~310× less protective than lnorm on the
same data, and (b) hardL-BFGS-B needs finite values of 'fn'failures. Suggested directions:
make the out-of-support penalty increase with distance (e.g. penaliseabsdev - scalelog
quadratically) instead of flattening; and/or constrainscalelog >= max|log(y) - locationlog|so
the support is covering by construction — which for a triangular is where the true MLE lives
anyway. At minimum, warn when the fitted support does not cover the data. - B2 — Censored branch is unguarded (
ll_ltriangle.hpp:95-102).pright - pleftcan be exactly
0 under bounded support →log(0)→ non-finite objective → fit failure. Needs the same treatment
as the uncensored branch. Also currently untested for this distribution.
Non-blocking
- N1 —
qdistextreme quantiles ignore bounded support (R/pqr.R:116-124).ssd_qltriangle(0)
returns 0 and(1)returnsInfinstead ofexp(m∓s), making the function discontinuous at its
own endpoints. Shared infrastructure — exposed by this PR, not introduced by it. Could reasonably
be a follow-up issue rather than a blocker on Add log-triangular (ltriangle) distribution #169. - N2 — Test coverage for the failure modes above. Add a fit test with an extreme outlier
asserting the fitted support covers the data (or that a warning is raised), and a censored-data fit
test exercising theleft < rightbranch. - N3 — Missing provenance in
distributions.Rmd. Add van Straalen (2002) and Stephan et al.
(1985); consider surfacing the US EPA SSD Toolbox reference already present in the test comments.
Nits
- n1 — One header comment in
ll_ltriangle.hppnoting the log-likelihood is piecewise-smooth
with kinks at each observation (inherent to the triangular; not a defect). - n2 —
R/data.R:49glossestailsas "has both tails"; a symmetric triangular does have two
finite tails. Consider "both unbounded tails" sotails=FALSEreads unambiguously. - n3 —
DESCRIPTIONhas noRoxygenNote, so nothing pins the roxygen2 version that produces the
committedNAMESPACEformatting (see §2.1). Pre-existing, repo-wide.
Explicitly verified as correct — no action
bound=FALSE (parameter bounds, not support bounds — correct); model-averaging wiring; the p/q/r
math (14/14 analytic checks, van Straalen's −1.675 to 6 dp); RNG construction and variance identity;
starting-value hardening; scale invariance; Air formatting; TMB compilation.
The density floor in `ll_ltriangle.hpp` was an attractor rather than a barrier. For an observation outside the candidate support the contribution reduced to `log(1e-8) - log(scalelog) - log(y)`, so the negative log-likelihood diverged to `-Inf` as `scalelog` approached zero: shrinking the support and flooring the excluded point was rewarded. The `locationlog` gradient was also identically zero once a point was floored, and `1e-8 * scalelog` underflowed to zero, giving `L-BFGS-B needs finite values of 'fn'`. On 30 concentrations clustered at 10 plus one observation at 1e-6 the fit returned a support of [7.4, 13.3] that excluded the outlier, with no warning, an HC5 of 8.1 against 0.054 for `lnorm`, and `ssd_hp()` of exactly 0 at the excluded observation. Write the log density in terms of the dimensionless `t = 1 - |log(y) - locationlog| / scalelog` and replace `log()` with its C1 linear extension below `1e-6`. The penalty is then scale free and grows without bound as the support shrinks away from an observation. The extension is inactive at any sensible optimum, so the likelihood and AIC are unchanged: the `ccme_boron` fit is identical to seven significant figures and the endosulfan and US EPA SSD Toolbox permethrin reference fits still reproduce. Apply the same softening to the censored branch. `ptri_ltriangle()` returns exactly 0 and exactly 1 outside the support, so unlike the unbounded distributions the interval mass reaches exactly 0 and `log()` was non-finite whenever a censored interval fell entirely outside the candidate support. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add Stephan et al. (1985), van Straalen (2002) and the US EPA SSD Toolbox user's manual to the log-triangular section of the distributions article, which previously cited only Wikipedia, and note that the zero hazard below the lower support limit is a property of the bounded distribution rather than an artefact of the fit. Also gloss `dist_data$tails` as "both unbounded tails", since a symmetric triangular does have two finite tails and `tails = FALSE` would otherwise read as wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Softening log() on the censored interval mass kept the objective finite but left it flat: a censoring interval entirely outside the candidate support has mass exactly 0 with zero gradient, so its cost was a constant 23 nats and the optimizer could sit on that plateau. A left-censored non-detect (`left = 0`, `right = DL`) below 30 concentrations clustered at 10 was silently excluded for DL = 1 and 0.1 (fitted lower support 7.38) and failed with `ABNORMAL_TERMINATION_IN_LNSRCH` for DL <= 0.01, while the same values as uncensored observations were covered. Non detects at the sensitive end are the normal case in SSD data. Add the same scale-free linear barrier as the uncensored branch, on the distance from the mode to the nearest point of the censoring interval, so that `tc = 1 - dist / scalelog` is positive iff the interval overlaps the support and excluding a censored observation is never profitable. The fitted lower support now sits below the detection limit for every DL from 1 to 1e-6, right-censoring above the data is covered, and the `ccme_boron` fit is unchanged to seven significant figures. Also narrow the article's claim that the distribution "is only fitted when explicitly requested" to `ssd_fit_dists()`'s default set, since `valid = TRUE` includes it in `ssd_dists()` and `ssd_dists_all()`, and note that the reported standard error for `locationlog` is unreliable because the log-likelihood is not differentiable at a mode that coincides with an observation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…to joethorley/ltriangle-barrier
Now that ssd_q*() returns each distribution's own support limits at p = 0 and p = 1 (#195), test_dist() takes the expected endpoints explicitly. The log-triangular defaults locationlog = 0 and scalelog = 3 give exp(-3) and exp(3). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…arrier Fix ltriangle fits excluding observations from the fitted support
Pin the bounded-support contract that #195 established for invpareto: ssd_qltriangle() returns exp(locationlog -+ scalelog) at p = 0 and p = 1, ssd_pltriangle() is exactly 0 below and 1 above the support, the two round trip at the limits, a mixture weighted on ltriangle inherits its finite limits while any unbounded component makes the mixture unbounded, and ssd_hc() at proportion 0 and 1 on a fitted distribution returns the fitted support with ssd_hp() exactly 0 and 1 there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
beckyfisher
left a comment
There was a problem hiding this comment.
Both blocking defects are resolved.
B1 is fixed at the mechanism. Writing the log density in terms of the dimensionless
t = 1 - |log(y) - locationlog| / scalelog leaves the in-support likelihood algebraically
unchanged, and makes the out-of-support contribution to nll grow without bound as scalelog
shrinks rather than fall without bound. That is the sign reversal the previous floor lacked.
B2 is fixed, and the follow-up found the case that mattered more than the one originally
reported: softening log() on the censored interval mass left a flat plateau, so a left-censored
non-detect below the data was still silently excluded. Non-detects at the sensitive end are the
normal case in SSD data. The censored branch now applies the same scale-free barrier as the
uncensored one, on the distance from the mode to the nearest point of the censoring interval.
The correction on the direction of the censored divergence is accepted: nll went to +Inf
there, not -Inf, so only its finiteness was broken.
N1 is resolved upstream in #195 by removing the hard-coded p == 0 and p == 1 branches from
.qd(), which is the general fix rather than a special case for ltriangle. N2, N3 and both nits
are done, and the four new regression tests cover both original failure modes and the non-detect
case.
The algebra and branch conditions were checked by reading the header at dbd4b674; the figures
in the replies above are the author's measurements and were not independently reproduced.
Two follow-ups, neither blocking this PR:
- The coverage assertion. The reason given for omitting it, that the barrier makes non-coverage
unreachable rather than unlikely, held for one day before the censored non-detect case disproved
it. A post-fit check that the fitted support covers every observation is independent of the
likelihood implementation and would have caught that case without requiring the insight that
found it. Worth an issue. - Bootstrap fits.
ssd_hc(ci = TRUE)refits many resampled datasets against a much stiffer
objective, and the new tests are all single fits. A bootstrap test on a dataset with an extreme
low observation would cover that interaction.
|
The log-triangular has been looked at before in the ANZ/Canadian work and the position has been consistently negative. Fox et al. (2021, ETC) describe it as a curious inclusion in the USEPA SSD Toolbox, on the basis that its tail characteristics aren't ones encountered in practice, and note that its only formal regulatory role is the Stephan et al. (1985) four-point fit to the most sensitive genera rather than a whole-of-dataset SSD. Fox et al. (2022) §4.4.3 then names it explicitly as the example of a distribution that is unrealistic for ecotoxicology because it has no left or right tails, and excludes it from the candidate set on that basis before any simulation work. The 2024 final report leaves the default set unchanged. None of that is an argument against having it available in the package. But it does mean anything that makes it reachable through ssd_fit_bcanz() or the default candidate set would run against a standing published recommendation, and under the consultation rules that's a MAJOR-version Technical Committee matter. Happy to help frame it that way in the docs if useful. |
Add a log-triangular distribution where log(concentration) follows a symmetric triangular distribution with mode
locationlogand half-widthscalelog. Wired in following thellogispattern: