Bound the inverse Pareto scale correction and document the estimator - #201
Open
joethorley wants to merge 1 commit into
Open
joethorley wants to merge 1 commit into
joethorley wants to merge 1 commit into
Conversation
The inverse Pareto `scale` is not estimated by maximum likelihood, which would place it on the largest observation, but fixed at the largest observation inflated by a bias correction. The factor used, n * shape / (n * shape - 1), is unbounded as n * shape approaches 1: six observations `c(1, 1, 1, 1, 1, 400)` gave a fitted `scale` of 2380, six times the largest observation, and the fit reported success. Use the factor (n * shape + 1) / (n * shape) instead. It is the exact unbiasing correction for known shape, since the largest of n observations has expectation scale * n * shape / (n * shape + 1), it is bounded above by two, and in simulation it performs within one percent of the previous factor at n = 6 and identically for larger n. Estimates shift slightly as a result: the `ccme_boron` HC5 moves from 0.3869 to 0.3898. Refuse right-censored data with a clear message. The `scale` is taken from the largest observation, which is unknown when any observation is right censored; previously the infinite starting value surfaced as the generic "L-BFGS-B needs finite values of 'fn'" failure. Document what is estimated in the additional technical details article, which described the closed-form MLEs without saying that ssdtools does not use them, and correct its expression for the shape MLE, which had the geometric mean and the scale inverted and was not dimensionless. Record the residual small-sample bias of the `shape` (about 28% at n = 6) and its effect on HC5, and cite Quandt (1966), Malik (1970), Baxter (1980), Johnson, Kotz and Balakrishnan (1994) and Arnold (2015) for the underlying Pareto results. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
joethorley
marked this pull request as ready for review
September 4, 2026 21:06
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows the review of the existing
invparetoestimator. The likelihood is correct; the estimation procedure and its documentation had three problems.Summary
The scale correction was unbounded.
sinvpareto()fixesscaleat the largest observation inflated byn * shape / (n * shape - 1), withshapethe conditional MLE at the maximum. Asn * shapeapproaches 1 the factor diverges, so a small, highly dispersed sample produces a nonsensical fit that reports success: forc(1, 1, 1, 1, 1, 400),n * shapeis 1.20 and the fittedscalewas 2380, six times the largest observation. This PR uses(n * shape + 1) / (n * shape), the exact unbiasing correction for known shape (the largest ofnpower-function observations has expectationscale * n * shape / (n * shape + 1)), which is bounded above by 2.Monte Carlo at
shape = 2,scale = 1, 20 000 replicates, mean of the fixedscale:Estimates move slightly as a result: on
ccme_boron,scale75.26 to 74.96 and HC5 0.3869 to 0.3898 (+0.7%).Right-censored data failed unhelpfully. The
scalecomes from the largest observation, which is unknown when any row is right censored, so the starting value wasInfand the optimiser failed with the generic "L-BFGS-B needs finite values of 'fn'".sinvpareto()now errors with a message saying the distribution requires a finite largest concentration, whichssd_fit_dists()reports through its usual "failed to fit" warning.The article misdescribed the estimator.
additional-technical-details.Rmdgave the closed-form MLEs without saying that ssdtools does not use the MLE forscale, and its expression for the shape MLE,[ln(g_X / b̂)]^-1withb̂ = 1/max, had the geometric mean and the limit inverted and was not dimensionless. The section now gives[ln(max / g_X)]^-1, notes the classical biases (E[λ̂] = nλ/(n-2)), and gains an "Estimation in ssdtools" subsection describing the fixed, bias-correctedscale, the conditional MLEshape, the residual small-sample bias (about 28% inshapeatn = 6, about +12% median in HC5), why the log-likelihood is not comparable for model averaging, and the right-censoring restriction. Five references added toreferences.bib.Not changed
The
shaperemains biased upwards in small samples. A further(n - 1) / ncorrection would bring the mean HC5 bias atn = 6to about -3% but the median to about -15%; that trade-off is documented rather than decided here.Verification
Tests written first and watched fail: the new correction factor (12.30 vs 11.87 expected on a six-point vector), the dispersed-sample bound (
scale2380 vs the required < 800), and the right-censored error (previously silent). Existinginvparetosnapshot values and the extreme-data literals updated; thessd_hc, Burrlioz-fallback andpredictsnapshots that contain aninvparetorow shift by the same 0.7%.Full suite 1410 passing, 0 failures;
R CMD check0 errors, 0 warnings, 0 notes;air format --checkclean; the technical article renders with all five new citations resolving.Downstream
The ssdtests
hc5_gmandbcanz_hcsnapshots containinvparetorows and will shift by the same small amount; bcgov/ssdtests#11 already covers regenerating those snapshots for ssddata 2.0.0, so this can be folded into that.