Merge pull request #137 from FreshCode-Org/feature/validation-gauntlet-jwd

Validation Gauntlet: gold-labelled disposition benchmark + six defect fixes

Author: JohnnyWilson16

.github/workflows/gauntlet.yml

@@ -0,0 +1,59 @@
+# Validation Gauntlet: gold-labelled disposition benchmark for the validation,
+# domain and text-cleaning surfaces. Runs the lightweight fixtures on every PR
+# and gates on the absolute thresholds plus no-regression vs the stored
+# baseline (benchmarks/gauntlet/baseline.json). Heavier sizes stay manual.
+name: Validation Gauntlet
+
+on:
+  pull_request:
+    paths-ignore:
+      - "docs/**"
+      - "*.md"
+  workflow_dispatch:
+    inputs:
+      rows:
+        description: "rows per fixture"
+        default: "300"
+      update_baseline:
+        description: "re-pin baseline.json from this run (commit it manually)"
+        type: boolean
+        default: false
+
+permissions:
+  contents: read
+
+jobs:
+  gauntlet:
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+      - name: Run gauntlet with gates
+        run: |
+          ROWS="${{ github.event.inputs.rows || '300' }}"
+          EXTRA=""
+          if [ "${{ github.event.inputs.update_baseline }}" = "true" ]; then
+            EXTRA="--update-baseline"
+          fi
+          python -m benchmarks.gauntlet run --rows "$ROWS" --check $EXTRA
+      - name: Job summary
+        if: always()
+        run: |
+          if [ -f benchmarks/gauntlet/results/gauntlet.md ]; then
+            cat benchmarks/gauntlet/results/gauntlet.md >> "$GITHUB_STEP_SUMMARY"
+          fi
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: gauntlet-results
+          path: benchmarks/gauntlet/results/
+          if-no-files-found: warn

.gitignore

@@ -34,3 +34,4 @@ crates/freshcore/target/
 # dev artifact (regenerated by running teacher tasks), not committed content.
 training/cache/*
 !training/cache/.gitkeep
+benchmarks/gauntlet/results/

CHANGELOG.md

@@ -6,7 +6,63 @@ adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Added
+- **Validation Gauntlet** (`benchmarks/gauntlet/`, `docs/validation-gauntlet.md`):
+  a gold-labelled disposition benchmark for the validation, domain and
+  text-cleaning surfaces. Five deterministic fixtures (finance, healthcare,
+  CRM, e-commerce, adversarial text) label every injected defect with the
+  disposition FreshData should choose (preserve / repair / flag / review) and
+  the harness scores detection P/R/F1, repair accuracy, review routing,
+  preservation, corruption, escapes, false positives, audit completeness,
+  determinism, trust monotonicity and runtime/memory. Runs on every PR
+  (`gauntlet.yml`) with absolute gates plus no-regression checks against the
+  stored `baseline.json`.
+- `CleanReport.coerced_cells`: per-cell record (`{column: {row: original}}`)
+  of values that `fix_dtypes` nulled because they did not parse as the
+  column's inferred type — the recovery source for quarantined cells, also
+  included in `report.to_dict()`.
+- Date-field range validation in `fd.validate_fields`: `FieldSpec.min_value`
+  / `max_value` now accept a date string or timestamp for `date`/`datetime`
+  fields, so a future date of birth or an 1875 admission date is flagged as a
+  `domain_mismatch` (gauntlet finding).
+- Case-variant vocabulary suggestions in `fd.validate_fields`: a value that
+  matches an `allowed_values` entry except for case (`ACTIVE` vs `active`) is
+  no longer silently accepted — it gets a warning-severity issue with the
+  canonical form as `suggestion` and action `accept_with_warning` (gauntlet
+  finding).
+
 ### Fixed
+- **Unparseable values are quarantined, never fabricated** (gauntlet finding,
+  the `'apple'`-in-a-price-column case): when `fix_dtypes` converts a
+  mostly-numeric (or datetime) text column, cells that fail to parse used to
+  become `NaN` and then be silently imputed by the auto engine — turning
+  junk into a fabricated median. They now stay missing, are excluded from
+  auto-imputation, keep their originals in `report.coerced_cells`, and the
+  decision is a `human_review` action in the audit trail. Genuine missing
+  values (true `NaN`, sentinels like `"N/A"`) keep the documented
+  auto-impute behaviour, and an explicit `impute=` request still fills
+  everything.
+- Formatted-number stragglers (`"$1,234.56"`, `"1,200,500.00"`) in a
+  mostly-plain numeric column are now parsed by the existing locale-aware
+  rescue instead of being coerced to missing — the rescue previously only
+  engaged when the plain parse failed the threshold entirely (gauntlet
+  finding).
+- `fd.validate_fields` consensus inference now honours the same
+  contamination boundary as the `fix_dtypes` warning that points users at it
+  (dominant share ≥ 60% with at most a handful of stragglers). Previously
+  the warning fired from a 60% parse share but the consensus gate required
+  80%, so the exact frame the warning named sailed through
+  `validate_fields` silently (gauntlet finding).
+- Explicitly allowed values are no longer swallowed by null-marker
+  heuristics in `fd.validate_fields`: with
+  `FieldSpec(allowed_values={"US", "DE", "NA"})`, `"NA"` is Namibia, not a
+  missing value (gauntlet finding).
+- `clean_text` / `validate_fields` text normalization no longer rewrites
+  typography in content-bearing fields: for `free_text`, `text` and entity
+  name types, the punctuation→ASCII mapping (curly quotes, em-dashes, prime
+  marks — `12″` became `12"`) is withheld, matching the field-aware safety
+  contract. Untyped columns keep the existing behaviour (gauntlet finding).
+
 - `anonymize()` called with no `rules` and no `detection_config` now emits
   a `UserWarning` instead of silently returning the data unchanged — a
   privacy call that does nothing must say so. Behavior is otherwise

benchmarks/README.md

@@ -9,6 +9,12 @@ pyjanitor baselines.
 > The harness calls FreshData exactly as a user would. It never modifies library
 > internals.
 
+> Sibling harness: `benchmarks/gauntlet/` (the **Validation Gauntlet**)
+> scores per-cell dispositions — preserve / repair / flag / review — for
+> the validation, domain and text-cleaning surfaces against gold labels,
+> and gates every PR via `.github/workflows/gauntlet.yml`. See
+> `docs/validation-gauntlet.md`.
+
 ## Layout
 
 ```

benchmarks/gauntlet/__init__.py

@@ -0,0 +1,28 @@
+"""FreshData Validation Gauntlet.
+
+Gold-labelled adversarial fixtures plus a harness that measures how
+FreshData's validation surfaces (``fd.clean``, ``fd.validate_fields``,
+``fd.clean_text``, domain packs, the semantic layer, PII detection) treat
+each labelled cell: preserve, repair, flag, or route to review.
+
+Unlike CleanBench (which scores whole-frame repair fidelity against a clean
+oracle), the gauntlet scores *dispositions*: every injected defect carries the
+disposition FreshData should choose, and every adversarial trap is a valid
+value that must survive cleaning untouched.
+
+Run ``python -m benchmarks.gauntlet run`` from the repo root.
+"""
+
+from .fixtures import FIXTURES, GauntletFixture, GoldCell, build_fixture
+from .metrics import compute_metrics
+from .runner import run_fixture, run_gauntlet
+
+__all__ = [
+    "FIXTURES",
+    "GauntletFixture",
+    "GoldCell",
+    "build_fixture",
+    "compute_metrics",
+    "run_fixture",
+    "run_gauntlet",
+]

benchmarks/gauntlet/__main__.py

@@ -0,0 +1,67 @@
+"""Validation Gauntlet CLI.
+
+Run from the repo root::
+
+    python -m benchmarks.gauntlet run                 # run + write results/
+    python -m benchmarks.gauntlet run --check         # also gate (CI mode)
+    python -m benchmarks.gauntlet run --update-baseline
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from .fixtures import DEFAULT_ROWS, DEFAULT_SEED, FIXTURES
+from .metrics import compute_metrics
+from .report import check_gates, render_markdown, results_payload, write_json
+from .runner import run_gauntlet
+
+RESULTS_DIR = Path(__file__).parent / "results"
+BASELINE_PATH = Path(__file__).parent / "baseline.json"
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="python -m benchmarks.gauntlet")
+    sub = parser.add_subparsers(dest="command", required=True)
+    run_p = sub.add_parser("run", help="run the gauntlet and write JSON + Markdown")
+    run_p.add_argument("--rows", type=int, default=DEFAULT_ROWS)
+    run_p.add_argument("--seed", type=int, default=DEFAULT_SEED)
+    run_p.add_argument("--fixtures", nargs="*", choices=sorted(FIXTURES))
+    run_p.add_argument("--check", action="store_true",
+                       help="exit 1 when a gate fails or the baseline regresses")
+    run_p.add_argument("--update-baseline", action="store_true",
+                       help="write this run as the stored baseline")
+    args = parser.parse_args(argv)
+
+    runs = run_gauntlet(n_rows=args.rows, seed=args.seed, fixtures=args.fixtures)
+    metrics = {name: compute_metrics(r) for name, r in runs.items()}
+    payload = results_payload(metrics, n_rows=args.rows, seed=args.seed)
+
+    write_json(payload, RESULTS_DIR / "gauntlet.json")
+    markdown = render_markdown(payload)
+    (RESULTS_DIR / "gauntlet.md").write_text(markdown)
+    print(markdown)
+    print(f"results: {RESULTS_DIR / 'gauntlet.json'}")
+
+    if args.update_baseline:
+        write_json(payload, BASELINE_PATH)
+        print(f"baseline updated: {BASELINE_PATH}")
+
+    if args.check:
+        baseline = (json.loads(BASELINE_PATH.read_text())
+                    if BASELINE_PATH.exists() else None)
+        problems = check_gates(payload, baseline)
+        if problems:
+            print("\nGATE FAILURES:", file=sys.stderr)
+            for p in problems:
+                print(f"  - {p}", file=sys.stderr)
+            return 1
+        print("all gates passed")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())