docs(examples): split before normalization to prevent data leakage in 12 notebooks - #2761
Merged
gpleiss merged 5 commits intoJul 10, 2026
Merged
Conversation
…819) Notebooks fixed: - SGPR_Regression_CUDA.ipynb - Scalable_Kernel_Interpolation_for_Products_CUDA.ipynb - Simple_GP_Regression_With_LOVE_Fast_Variances_and_Sampling.ipynb The data-loading cell was computing X.min/max on the full dataset BEFORE the train/test split, leaking test statistics into training. The fix moves normalization AFTER the split and computes min/max from training data only, then applies the same scaler to test data. Refs #819. PR #2752 is complementary (adds docs + verification script) but does not modify these notebooks.
…otebooks (#819) Notebooks fixed: - Modifying_the_variational_strategy_and_distribution.ipynb - Natural_Gradient_Descent.ipynb - SVGP_Regression_CUDA.ipynb - SVGP_CIQ.ipynb (also normalizes y) - VNNGP.ipynb (also normalizes y) The data-loading cell was computing X.min/max (and in 2 of 5 cases, y.mean/std) on the full dataset BEFORE the train/test split, leaking test statistics into training. The fix moves normalization AFTER the split and computes statistics from training data only, then applies the same scaler/standardization to test data. Refs #819.
Notebooks fixed: - Deep_Gaussian_Processes.ipynb - Deep_Sigma_Point_Processes.ipynb (also normalizes y, special case: 75/25 split with shuffle + no .contiguous() in the original) The data-loading cell was computing X.min/max (and in Deep_Sigma, y.mean/std) on the full dataset BEFORE the train/test split, leaking test statistics into training. The fix moves normalization AFTER the split and computes statistics from training data only, then applies the same scaler/standardization to test data. Refs #819.
…ebooks (#819) Notebooks fixed: - KISSGP_Deep_Kernel_Regression_CUDA.ipynb - TorchScript_Variational_Models.ipynb Both had the same anti-pattern: X.min/max computed on the full dataset BEFORE the train/test split. The fix moves normalization AFTER the split and computes statistics from training data only. Refs #819.
gpleiss
enabled auto-merge (squash)
July 10, 2026 17:27
gpleiss
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #819
Context
12 of the 50 example notebooks in this repository normalize features (and sometimes labels) using statistics computed from the full dataset (including test data) before splitting into train/test. This silently leaks test-set statistics into the training pipeline and contradicts the pattern used by the clean notebooks in the same directories.
The anti-pattern is uniform:
X = X - X.min(0)[0]/X = 2 * (X / X.max(0)[0]) - 1(and sometimesynormalization) is computed on the fullX, then atrain_n = int(floor(0.8 * len(X)))split is performed afterwards.PR #2752 (open, by r69shabh) adds documentation and a verification script for this same issue but does not fix the notebooks themselves. This PR is complementary — it fixes the 12 offending notebooks, while #2752 adds the docs and the check_normalization.py detector. Both can merge independently.
Fix
In each affected notebook, the data-loading cell is restructured to:
This is the pattern already used by the clean reference notebooks (KeOps_GP_Regression, Simple_MultiGPU_GP_Regression) and is the only correct approach when test data must remain unseen.
Notebooks fixed (12)
examples/02_Scalable_Exact_GPs/SGPR_Regression_CUDA.ipynbexamples/02_Scalable_Exact_GPs/Scalable_Kernel_Interpolation_for_Products_CUDA.ipynbexamples/02_Scalable_Exact_GPs/Simple_GP_Regression_With_LOVE_Fast_Variances_and_Sampling.ipynbexamples/04_Variational_and_Approximate_GPs/Modifying_the_variational_strategy_and_distribution.ipynbexamples/04_Variational_and_Approximate_GPs/Natural_Gradient_Descent.ipynbexamples/04_Variational_and_Approximate_GPs/SVGP_Regression_CUDA.ipynbexamples/04_Variational_and_Approximate_GPs/SVGP_CIQ.ipynb(also normalizesy)examples/04_Variational_and_Approximate_GPs/VNNGP.ipynb(also normalizesy; usesX.max(0)[0].clamp_min(1e-6))examples/05_Deep_Gaussian_Processes/Deep_Gaussian_Processes.ipynbexamples/05_Deep_Gaussian_Processes/Deep_Sigma_Point_Processes.ipynb(also normalizesy; 75/25 split with shuffle andX.size(0)indexing — special case)examples/06_PyTorch_NN_Integration_DKL/KISSGP_Deep_Kernel_Regression_CUDA.ipynbexamples/08_Advanced_Usage/TorchScript_Variational_Models.ipynbThe remaining 38 notebooks are clean (no change).
Test plan
pre-commit run --files <notebook-paths>— therequire-asciihook excludes notebooks by design (exclude: ^(examples/.*\.ipynb)|(.github/ISSUE_TEMPLATE/.*)); ufmt is a no-op on JSON-derived.ipynb; all other hooks passNet change: 52 insertions, 417 deletions across 12 files.