Skip to content

docs(examples): split before normalization to prevent data leakage in 12 notebooks - #2761

Merged
gpleiss merged 5 commits into
cornellius-gp:mainfrom
umi008:fix-819-example-normalization
Jul 10, 2026
Merged

docs(examples): split before normalization to prevent data leakage in 12 notebooks#2761
gpleiss merged 5 commits into
cornellius-gp:mainfrom
umi008:fix-819-example-normalization

Conversation

@umi008

@umi008 umi008 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Closes #819

Context

12 of the 50 example notebooks in this repository normalize features (and sometimes labels) using statistics computed from the full dataset (including test data) before splitting into train/test. This silently leaks test-set statistics into the training pipeline and contradicts the pattern used by the clean notebooks in the same directories.

The anti-pattern is uniform: X = X - X.min(0)[0] / X = 2 * (X / X.max(0)[0]) - 1 (and sometimes y normalization) is computed on the full X, then a train_n = int(floor(0.8 * len(X))) split is performed afterwards.

PR #2752 (open, by r69shabh) adds documentation and a verification script for this same issue but does not fix the notebooks themselves. This PR is complementary — it fixes the 12 offending notebooks, while #2752 adds the docs and the check_normalization.py detector. Both can merge independently.

Fix

In each affected notebook, the data-loading cell is restructured to:

  1. Split into train/test first (using the dataset's existing shuffle + index).
  2. Compute normalization statistics (min/max, mean/std) from the training split only.
  3. Apply the same statistics to both train and test.

This is the pattern already used by the clean reference notebooks (KeOps_GP_Regression, Simple_MultiGPU_GP_Regression) and is the only correct approach when test data must remain unseen.

Notebooks fixed (12)

  • examples/02_Scalable_Exact_GPs/SGPR_Regression_CUDA.ipynb
  • examples/02_Scalable_Exact_GPs/Scalable_Kernel_Interpolation_for_Products_CUDA.ipynb
  • examples/02_Scalable_Exact_GPs/Simple_GP_Regression_With_LOVE_Fast_Variances_and_Sampling.ipynb
  • examples/04_Variational_and_Approximate_GPs/Modifying_the_variational_strategy_and_distribution.ipynb
  • examples/04_Variational_and_Approximate_GPs/Natural_Gradient_Descent.ipynb
  • examples/04_Variational_and_Approximate_GPs/SVGP_Regression_CUDA.ipynb
  • examples/04_Variational_and_Approximate_GPs/SVGP_CIQ.ipynb (also normalizes y)
  • examples/04_Variational_and_Approximate_GPs/VNNGP.ipynb (also normalizes y; uses X.max(0)[0].clamp_min(1e-6))
  • examples/05_Deep_Gaussian_Processes/Deep_Gaussian_Processes.ipynb
  • examples/05_Deep_Gaussian_Processes/Deep_Sigma_Point_Processes.ipynb (also normalizes y; 75/25 split with shuffle and X.size(0) indexing — special case)
  • examples/06_PyTorch_NN_Integration_DKL/KISSGP_Deep_Kernel_Regression_CUDA.ipynb
  • examples/08_Advanced_Usage/TorchScript_Variational_Models.ipynb

The remaining 38 notebooks are clean (no change).

Test plan

  • Manual smoke test of the split-then-normalize pattern with synthetic data confirms test inputs may exceed the train-input range (correct, no leakage)
  • All 12 notebook data-loading cells produce the correct train/test tensor shapes after the change
  • Verification script (if/when docs: Add normalization best practices and verification script #2752 merges) should report zero remaining offenders
  • pre-commit run --files <notebook-paths> — the require-ascii hook excludes notebooks by design (exclude: ^(examples/.*\.ipynb)|(.github/ISSUE_TEMPLATE/.*)); ufmt is a no-op on JSON-derived .ipynb; all other hooks pass

Net change: 52 insertions, 417 deletions across 12 files.

umi008 and others added 5 commits July 9, 2026 13:12
…819)

Notebooks fixed:
- SGPR_Regression_CUDA.ipynb
- Scalable_Kernel_Interpolation_for_Products_CUDA.ipynb
- Simple_GP_Regression_With_LOVE_Fast_Variances_and_Sampling.ipynb

The data-loading cell was computing X.min/max on the full dataset BEFORE
the train/test split, leaking test statistics into training. The fix
moves normalization AFTER the split and computes min/max from training
data only, then applies the same scaler to test data.

Refs #819. PR #2752 is complementary (adds docs + verification script)
but does not modify these notebooks.
…otebooks (#819)

Notebooks fixed:
- Modifying_the_variational_strategy_and_distribution.ipynb
- Natural_Gradient_Descent.ipynb
- SVGP_Regression_CUDA.ipynb
- SVGP_CIQ.ipynb (also normalizes y)
- VNNGP.ipynb (also normalizes y)

The data-loading cell was computing X.min/max (and in 2 of 5 cases,
y.mean/std) on the full dataset BEFORE the train/test split, leaking
test statistics into training. The fix moves normalization AFTER the
split and computes statistics from training data only, then applies
the same scaler/standardization to test data.

Refs #819.
Notebooks fixed:
- Deep_Gaussian_Processes.ipynb
- Deep_Sigma_Point_Processes.ipynb (also normalizes y, special case: 75/25 split
  with shuffle + no .contiguous() in the original)

The data-loading cell was computing X.min/max (and in Deep_Sigma, y.mean/std)
on the full dataset BEFORE the train/test split, leaking test statistics
into training. The fix moves normalization AFTER the split and computes
statistics from training data only, then applies the same scaler/standardization
to test data.

Refs #819.
…ebooks (#819)

Notebooks fixed:
- KISSGP_Deep_Kernel_Regression_CUDA.ipynb
- TorchScript_Variational_Models.ipynb

Both had the same anti-pattern: X.min/max computed on the full dataset
BEFORE the train/test split. The fix moves normalization AFTER the split
and computes statistics from training data only.

Refs #819.
@gpleiss
gpleiss enabled auto-merge (squash) July 10, 2026 17:27
@gpleiss
gpleiss merged commit 93222a0 into cornellius-gp:main Jul 10, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Some examples normalize training data with test data

2 participants