Skip to content

Bound the inverse Pareto scale correction and document the estimator - #201

Open
joethorley wants to merge 1 commit into
devfrom
joethorley/invpareto-estimator
Open

joethorley wants to merge 1 commit into
devfrom
joethorley/invpareto-estimator

Conversation

@joethorley

Copy link
Copy Markdown
Member

Follows the review of the existing invpareto estimator. The likelihood is correct; the estimation procedure and its documentation had three problems.

Summary

The scale correction was unbounded. sinvpareto() fixes scale at the largest observation inflated by n * shape / (n * shape - 1), with shape the conditional MLE at the maximum. As n * shape approaches 1 the factor diverges, so a small, highly dispersed sample produces a nonsensical fit that reports success: for c(1, 1, 1, 1, 1, 400), n * shape is 1.20 and the fitted scale was 2380, six times the largest observation. This PR uses (n * shape + 1) / (n * shape), the exact unbiasing correction for known shape (the largest of n power-function observations has expectation scale * n * shape / (n * shape + 1)), which is bounded above by 2.

Monte Carlo at shape = 2, scale = 1, 20 000 replicates, mean of the fixed scale:

n largest observation previous factor this PR
6 0.923 0.993 0.987
10 0.953 0.998 0.995
30 0.984 0.9997 0.9994

Estimates move slightly as a result: on ccme_boron, scale 75.26 to 74.96 and HC5 0.3869 to 0.3898 (+0.7%).

Right-censored data failed unhelpfully. The scale comes from the largest observation, which is unknown when any row is right censored, so the starting value was Inf and the optimiser failed with the generic "L-BFGS-B needs finite values of 'fn'". sinvpareto() now errors with a message saying the distribution requires a finite largest concentration, which ssd_fit_dists() reports through its usual "failed to fit" warning.

The article misdescribed the estimator. additional-technical-details.Rmd gave the closed-form MLEs without saying that ssdtools does not use the MLE for scale, and its expression for the shape MLE, [ln(g_X / b̂)]^-1 with b̂ = 1/max, had the geometric mean and the limit inverted and was not dimensionless. The section now gives [ln(max / g_X)]^-1, notes the classical biases (E[λ̂] = nλ/(n-2)), and gains an "Estimation in ssdtools" subsection describing the fixed, bias-corrected scale, the conditional MLE shape, the residual small-sample bias (about 28% in shape at n = 6, about +12% median in HC5), why the log-likelihood is not comparable for model averaging, and the right-censoring restriction. Five references added to references.bib.

Not changed

The shape remains biased upwards in small samples. A further (n - 1) / n correction would bring the mean HC5 bias at n = 6 to about -3% but the median to about -15%; that trade-off is documented rather than decided here.

Verification

Tests written first and watched fail: the new correction factor (12.30 vs 11.87 expected on a six-point vector), the dispersed-sample bound (scale 2380 vs the required < 800), and the right-censored error (previously silent). Existing invpareto snapshot values and the extreme-data literals updated; the ssd_hc, Burrlioz-fallback and predict snapshots that contain an invpareto row shift by the same 0.7%.

Full suite 1410 passing, 0 failures; R CMD check 0 errors, 0 warnings, 0 notes; air format --check clean; the technical article renders with all five new citations resolving.

Downstream

The ssdtests hc5_gm and bcanz_hc snapshots contain invpareto rows and will shift by the same small amount; bcgov/ssdtests#11 already covers regenerating those snapshots for ssddata 2.0.0, so this can be folded into that.

The inverse Pareto `scale` is not estimated by maximum likelihood, which
would place it on the largest observation, but fixed at the largest
observation inflated by a bias correction. The factor used,
n * shape / (n * shape - 1), is unbounded as n * shape approaches 1: six
observations `c(1, 1, 1, 1, 1, 400)` gave a fitted `scale` of 2380, six
times the largest observation, and the fit reported success. Use the
factor (n * shape + 1) / (n * shape) instead. It is the exact unbiasing
correction for known shape, since the largest of n observations has
expectation scale * n * shape / (n * shape + 1), it is bounded above by
two, and in simulation it performs within one percent of the previous
factor at n = 6 and identically for larger n. Estimates shift slightly
as a result: the `ccme_boron` HC5 moves from 0.3869 to 0.3898.

Refuse right-censored data with a clear message. The `scale` is taken
from the largest observation, which is unknown when any observation is
right censored; previously the infinite starting value surfaced as the
generic "L-BFGS-B needs finite values of 'fn'" failure.

Document what is estimated in the additional technical details article,
which described the closed-form MLEs without saying that ssdtools does
not use them, and correct its expression for the shape MLE, which had
the geometric mean and the scale inverted and was not dimensionless.
Record the residual small-sample bias of the `shape` (about 28% at
n = 6) and its effect on HC5, and cite Quandt (1966), Malik (1970),
Baxter (1980), Johnson, Kotz and Balakrishnan (1994) and Arnold (2015)
for the underlying Pareto results.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@joethorley
joethorley marked this pull request as ready for review September 4, 2026 21:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant