fix(outliers): zero-IQR fallback fences; never clip declared id/target columns

[kevincostner17]

Summary

This PR fixes two cases where outliers= did the wrong thing.

• Zero IQR. A non-constant numeric column whose quartiles coincide got no Tukey fences. An example is 95 zeros plus spikes of 1000, 5000 and -800. For that column, outlier_method="iqr" and "auto" flagged, clipped, capped and removed nothing, even though "zscore" still flagged the 5000.
• Declared roles under clip. outliers="clip" rewrote values in columns declared through id_columns and target_column. A legitimate customer_id of 10,000,000 was clipped, and a declared target's maximum fell from 5000 to 138.1. That breaks the documented guarantee that identifiers and targets are never modified.

Root cause

• Zero IQR. With Q1 == Q3 the fences collapse to a single value. The step treated that like a constant column and skipped it.
• Declared roles. The explicit outlier step never consulted the declared roles. Only the decision engine did.

Fix

Zero-IQR fallback (steps/outliers.py, engine/outliers.py, FreshCore kernel)

• Fences. When Q1 == Q3, the IQR is replaced by 1.6907 × MeanAD, where MeanAD is the mean absolute deviation around the median. The constant is sqrt(pi/2) (MeanAD as a σ estimate) times 1.349. The fences are median ± factor × 1.6907 × MeanAD.
• 5% cap. The fallback applies only when it flags at most 5% of non-missing values. Above that, the off-center values are a second mode rather than rare outliers, and the column is left untouched, as a constant column is.
• Report text. Action descriptions say "IQR is zero, so fences use 1.69 x mean absolute deviation from the median".
• auto. Method selection is unchanged.
• Non-zero IQR. Columns with a non-zero IQR get identical fences and output.
• Backends. The FreshCore Rust kernel applies the same rule. Polars, DuckDB and Spark still skip zero-IQR columns; this is documented in docs/backends.md, and docs/cleaning-engine.md describes the rule.

Clip respects declared roles (steps/outliers.py)

• What is skipped. Clipping skips declared id_columns and target_column, matched exactly or by snake-case name after renaming.
• Report text. When such a column has values outside the fences, the report records "skipped: identifier column" or "skipped: target column".
• Name-only roles. Roles inferred only from column names are not skipped.
• Flagging. Flagging still reports these columns and never changes their values.
• Context-protected columns. Columns protected by context (mutable=False or policy) are skipped under clip too. They used to raise ProtectedColumnError.

Default-output changes

Yes.

• New outliers on zero-IQR columns. Default fd.clean (outlier_method="auto"/"iqr") can now flag outliers on zero-IQR columns where it found none before. Under the default flag action, this adds <col>_outlier columns and outlier actions or counts to the report.
  • The golden adult_income.balanced report is refreshed and gains one flagged column.
  • Wide sparse datasets such as spambase gain up to 20 flag columns.
• clip with declared roles. Declared id_columns and target_column are no longer modified, and the report gains "skipped: …" entries.
• clip with context-protected columns. These columns are skipped instead of raising.

Tests

• tests/test_outliers.py:
  • the zero-IQR spike repro for iqr and auto, under flag, clip, cap and remove
  • the 5% second-mode cap leaves the column untouched
  • constant columns are still skipped
  • non-zero-IQR columns are unchanged
  • declared id and target columns are not clipped, including after renaming, and the "skipped" messages are recorded
  • context-protected columns are skipped under clip
• tests/test_execution/test_freshcore_native_parity.py:
  • pandas and FreshCore flags match for a zero-IQR column under iqr and zscore
  • the kernel's clip values match pandas
• Golden fixtures are refreshed: adult_income.balanced.report.json and golden_diff_summary.jsonl.

Verification

• ruff check . passes.
• test_freshcore_native_parity.py, test_outliers.py, test_engine_outliers.py and test_golden.py pass:
  • py3.12 / pandas 2.3.3: 74 passed, 1 skipped
  • py3.9 / pandas 1.5.3: 74 passed, 1 skipped
  • a scratch venv with the FreshCore native extension built from this branch: 121 passed, 1 skipped
• Full not online and not large lanes on the rebased branch: py3.12 / pandas 2.3.3 6396 passed, 14 skipped; py3.9 / pandas 1.5.3 6367 passed, 18 skipped.
• The rebase onto current main conflicted in test_freshcore_native_parity.py: both sides appended tests there. I resolved it by keeping both, and the tests above were rerun after the rebase.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e38b2135-8a9b-406d-9bb7-a8d5f30e2f34

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)