viz smart: guard inequality (Gini/Lorenz) framing for heterogeneous-unit tables; label zero-run as pipeline stage

[jqnatividad]

Summary

viz smart --smarter applies inequality framing (Gini + Lorenz curve) to any additive measure with high Gini — but on a table whose unit is a heterogeneous project, a high Gini is expected and is not an equity finding. The inequality vocabulary reads to the public as unfairness, inviting a serious misread.

Why this matters

• The CPDB unit is "capital project," and projects are radically heterogeneous — a subway extension should cost a thousand times a playground resurfacing. Concentration across heterogeneous units is nearly tautological, so "NYC capital spending is wildly unequal (Gini 0.96)" is true and meaningless.
• Zero-inflation conflates two populations. With 45–60% zeros, the flat run of the Lorenz curve is the zeros — projects with nothing committed/spent yet (a funding-pipeline stage), not "small/poor" projects. Measured inequality is partly a pipeline artifact.

Proposal

• Gate or caveat the Gini/Lorenz framing: prefer it when the row unit is a person/household/geography, and add a caveat when the unit is a heterogeneous entity.
• When zero-share is high, label the flat Lorenz segment as the zero/not-yet-funded stage and/or additionally report Gini on the non-zero subset.
• Consider surfacing the planned → committed → spent pipeline (currently buried in % zeros title annotations) as a funnel — the actually-actionable governance signal.

Related: the inequality-panel encoding itself is covered separately in the skew-aware-encoding issue.

Reproducing example

viz smart --smarter on the NYC Capital Projects Database (Checkbook NYC-sourced), 12,587 rows:

curl -s "https://data.cityofnewyork.us/resource/fi59-268w.csv?\$select=maprojid,magencyacro,magencyname,description,typecategory,totalplannedcommit,commit_total,spent_total,spent_total_checkbooknyc&\$limit=20000" -o cpdb.csv
qsv viz smart cpdb.csv --smarter --output cpdb_dashboard.html

The money columns are extremely right-skewed and zero-inflated (Gini 0.93–0.96; 45–60% zeros).

Filed from a critical design review of the viz smart dashboard; part of a set (skew-aware encoding, robust stats, redundant-dimension collapse, inequality-framing guard, 3D/density panels).

---

Status (updated 2026-07-25)

Two of the three asks shipped in #4280 (commit c5589d4). This issue stays open for ask 3 only — do not close it as completed.

Done — asks 1 & 2

Every Lorenz panel now carries a caveat line beneath its title:

zero share	subtitle
< 30%	concentration is expected unless rows are comparable units
≥ 30%	flat run = 60% zeros, not small values · concentration is expected unless rows are comparable units

Two decisions worth recording, since both were judgement calls:

• The unit caveat is unconditional — there is no gate. There is no row-unit signal anywhere in the stats cache: whether a row is a person, a household, a geography or a heterogeneous capital project is semantics, not a statistic. A name-heuristic proxy would either delete a legitimate equity finding or silently keep a misleading one, so every Lorenz panel carries the caveat instead. No gate, no false negatives.
• No Gini over the non-zero subset. The panel's Gini is the cached coefficient, never recomputed, so it agrees with the box hint / pivotp / scoresql. A subset Gini would break that contract for a second number readers would conflate with the first. The zero-run label conveys the same information without recomputing anything.

The zero share comes from the existing zero_share() helper, and ZERO_SHARE_MIN was hoisted out of box_shape_hint so the caveat and the % zeros box-title hint read one threshold through one helper — they appear on the same dashboard and must not disagree.

Remaining — ask 3: the pipeline funnel

Surfacing planned → committed → spent as a funnel is untouched. It is a larger piece of work than the two above:

• a new PanelKind plus its render arms (typed subplot grid and inline), and a new data pass
• an unsolved design question: how does viz smart recognise that a set of measures forms a pipeline? Column-name heuristics, dictionary concept/role, or an explicit flag. This has the same shape as the unit-heterogeneity problem above — it is semantics, not a statistic — so it deserves the same explicit treatment rather than a guessed proxy.
• a monotonicity question: the funnel framing only holds if each stage is a subset of the prior one, which is worth verifying from the data rather than assuming.

[jqnatividad]

Note on ask 3 (the funnel), from implementing it in #4281.

The motivating dataset is not a valid funnel, and the implementation correctly declines to draw one for it. Worth recording so this isn't later filed as a bug.

Running viz smart on the real CPDB (12,587 rows, fetched from NYC Open Data):

viz smart: pipeline funnel skipped — commit_total is not contained
           in totalplannedcommit (80% of rows)

The three columns are not nested:

stage	total	violations
totalplannedcommit	$191.9B	—
commit_total	$28.4B	commit > planned on 19.5% of rows
spent_total	$81.0B	spent > commit on 46.3% of rows

Spent is 2.9× committed, and 851 projects have zero planned but non-zero committed. The dataset's own column descriptions explain it:

• totalplannedcommit — "total planned commitments for the project allocated in the Capital Commitment Plan"
• commit_total — "total committed funding associated with the project within the City's budget"
• spent_total — "total funding spent associated with the project within the City's budget"

Three independent aggregates on different bases and time windows, not three stages of one nesting. A funnel over them would assert a containment the source data contradicts — so the detector requires both a stage vocabulary and row-wise containment, and here the second gate refuses. The threshold was deliberately not relaxed to make the panel appear.

Asks 1 and 2 land on this dataset exactly as intended, and it is now a gallery figure for that: Lorenz panels at Gini 0.93/0.96 reading flat run = 60% zeros, not small values — the zero-inflation clause no other gallery dataset exercised.

The funnel itself ships in two forms: auto-detection in viz smart for column-shaped pipelines, and a new standalone qsv viz funnel for pipelines encoded as rows (a stage column plus a value column), which is the shape the smart path explicitly declines to guess at.