Add LUCID: regime-adaptive deconfounding for latent confounders - #35
Closed
fesanghary wants to merge 1 commit into
Closed
Add LUCID: regime-adaptive deconfounding for latent confounders#35fesanghary wants to merge 1 commit into
fesanghary wants to merge 1 commit into
Conversation
Ports the LUCID library (shipped internally via qfr-ml/causal-ts#25, #26) to the public mirror: run_lucid() -> LucidResult, .deconfound()/.tetrad_filter()/ .pds_filter() on every CausalResult subclass, and the standalone filters (pds_filter, tetrad_filter) and router diagnostics (spectral_gap, mp_factor_count) in causalts.confounders. LUCID diagnoses whether latent confounding is sparse or pervasive from the residual spectrum (against a Marchenko-Pastur no-factor null) and applies the matching correction; both regimes run the same base discovery engine, so the routing decision only changes the correction applied afterward -- which is what makes .deconfound() valid for reusing an already-discovered graph from any algorithm (CDNOTS, CEDAR, GRACE, ...). Also adds causalts.synthetic_data.confounding.apply_confounding (needed by the test suite) and a new Unobserved Confounders (LUCID) tutorial notebook, gallery entry, and API doc page. Exports are lazy (causalts.__init__'s _LAZY_ATTRS pattern) rather than eager, matching internal: confounders pulls in statsmodels + scikit-learn, which plain `import causalts` doesn't currently load, and keeping that path cheap is this file's whole design intent. Verified: 35/35 test_confounders.py, 308/308 full suite, notebook executes clean (0 errors) with sane metrics (CDNOTS F1=0.172 -> tetrad F1=0.385 -> LUCID F1=0.556 on the same pervasive-confounding scenario), black 24 (CI's pin)/isort/flake8/codespell clean via the exact commands+config CI runs, sphinx-build succeeds with no new warnings beyond the pre-existing autoapi duplicate-object-description pattern shared by every other result class.
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Contributor
Author
|
Closing to redo with a corrected commit message that referenced an internal-only repo name. Reopening as a fresh PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
causalts.confoundersmodule.run_lucid()->LucidResult, plus.deconfound()/.tetrad_filter()/.pds_filter()added to everyCausalResultsubclass (works on CDNOTS, CEDAR, GRACE, ... results, not just LUCID's own)..deconfound()valid for reusing an already-discovered graph from any algorithm.causalts.synthetic_data.confounding.apply_confounding(needed by the test suite), a new "Unobserved Confounders (LUCID)" tutorial notebook + gallery entry, and an API doc page.causalts.__init__'s existing lazy (_LAZY_ATTRS) pattern rather than eager, sincecausalts.confounderspulls instatsmodels/scikit-learn, which plainimport causaltsdoesn't currently load — keeping that path cheap is this file's whole design intent.Test plan
pytest tests/test_confounders.py -v— 35/35 passpytest -q(full suite) — 308/308 pass, no regressions from thecausalts/result.py/causalts/__init__.py/causalts/synthetic_data/__init__.pyeditsjupyter nbconvert --execute): 0 error cells, sane metrics (CDNOTS F1=0.172 -> tetrad F1=0.385 -> LUCID F1=0.556 on the same pervasive-confounding scenario); the committed notebook's baked-in outputs are byte-identical to what ships (confirmed clean, no stderr/warnings) —docs/conf.pysetsnb_execution_mode = "off"so Sphinx renders exactly those saved outputs, not a fresh re-runblack==24.*pinned (black --check .),isort --check .,flake8 .(withFlake8-pyprojectinstalled, since flake8 needs it to read[tool.flake8]frompyproject.tomlat all) — all clean;codespellclean (added one justifiedretunedignore-word for a real word codespell mistakes for "returned")sphinx-build -b html docs docs/_build— succeeds; no new warnings beyond the pre-existing autoapi duplicate-object-description pattern already present for every other result class