cns-prevent-ad

Processing and merging scripts for the PREVENT-AD behavioral dataset, produced by CNS Fellows.

---

Requirements

• Python ≥ 3.11
• uv for virtual environment management

---

1. Get the Data

Extract the source data archive into data/:

mkdir -p data
tar -xzf <archive>.tar.gz --strip-components=1 -C data/

The data/ directory is git-ignored and its contents are never committed.

---

2. Create the Virtual Environment

From the repo root:

uv venv
uv pip install pandas pyarrow

This creates .venv/ in the repo root (also git-ignored). Activate it with source .venv/bin/activate if needed, or prefix commands with .venv/bin/python.

---

3. Run the Pipeline

Step 1 — Build the merged frame

Loads all source CSV tables, normalises visit labels, resolves duplicates, and outer-merges everything into a single long-format dataframe keyed on (CONP_ID, Visit_label):

python scripts/merge.py

Step 2 — Extract core features

Selects and renames the key descriptive, diagnostic, cognitive, and biomarker columns; derives APOE genotype, family-history composite, age in years, and TMT B/A ratio; and writes a BIDS-style data dictionary:

python scripts/extract_features.py

---

Outputs

All files are written to outputs/ (git-ignored).

File	Description	Shape
preventad_merged.parquet	Full long-format frame — all source variables	3,206 rows × 1,106 cols
preventad_merged.csv	CSV copy of the full frame	—
preventad_somascan.parquet	CSF SomaScan 7K proteomics (separate due to width)	67 rows × ~7,323 cols
preventad_actigraphy.parquet	Actigraphy / sleep, keyed on CONP_ID + Date_taken	—
preventad_meg.parquet	MEG relative power (one row per participant)	97 rows
preventad_features.parquet	Analysis-ready frame — renamed core features	3,206 rows × 52 cols
preventad_features.csv	CSV copy of the feature frame	—
preventad_features.json	BIDS-style data dictionary for the feature frame	—

---

Pipeline Internals

This section documents where each loading, cleaning, and merging decision is made so that specific operations can be located, audited, and adjusted.

Shared helpers — scripts/utils.py

Two functions called by every load script:

• load_csv(fname) — reads data/<fname> relative to the repo root. DATA_DIR is the single variable to change if the data directory moves.
• normalize_visit_label(df) — when a source table has no Visit_label column (or it is all-null), the 3-character study-arm prefix (PRE / NAP) is stripped from Study_visit_label to derive one. All cross-table visit harmonisation flows through this function.

---

Demographics and clinical — scripts/load_demographics.py

Source CSV	Treatment	Notes
Demographics_Registered_PREVENTAD_2025-03-21.csv	Participant-level	Base frame; one row per CONP_ID
Genetics_Registered_PREVENTAD_2025-03-21.csv	Participant-level	APOE and other genetic markers; left-joined on CONP_ID
GWAS_Bellenguez_Polymorphisms_Registered_PREVENTAD_2025-03-21.csv	Participant-level	GWAS SNP dosages; left-joined on CONP_ID
EL_CAIDE_Registered_PREVENTAD_2025-03-21.csv	Participant-level	Eligibility CAIDE risk score; visit metadata columns dropped before join
EL_Medical_history_Registered_PREVENTAD_2025-03-21.csv	Participant-level	Family history and medical history flags; visit metadata dropped before join
Clinical_diagnosis_Registered_PREVENTAD_2025-03-21.csv	Participant-level	External clinical diagnoses (~32 participants); sorted by date and last record kept per participant (groupby.last())
MCI_Registered_PREVENTAD_2025-03-21.csv	Visit-level	Research consensus MCI classifications; returned separately as df_mci

The first six tables are merged into a single df_participant frame (one row per CONP_ID). df_mci is returned separately and enters the visit-level merge in merge.py.

---

Cognitive assessments — scripts/load_cognitive.py

All tables are visit-level and outer-merged on (CONP_ID, Visit_label). When a metadata column (Date_taken, Candidate_Age, etc.) appears in more than one source file, the second occurrence gets a _dup suffix and is dropped immediately after each merge — the first occurrence's value is kept.

Source CSV	Key columns
RAVLT_Registered_PREVENTAD_2025-03-21.csv	List A learning trials, delayed recall
RBANS_Registered_PREVENTAD_2025-03-21.csv	Total and domain index scores
TMT_Registered_PREVENTAD_2025-03-21.csv	Part A and Part B completion times
DKEFS-CWIT_Registered_PREVENTAD_2024-08-01.csv	Inhibition and switching scaled scores
EL_CDR_MoCA_Registered_PREVENTAD_2025-03-21.csv	CDR and MoCA at eligibility visit only
CDR_FU_Registered_PREVENTAD_2025-03-21.csv	CDR at follow-up visits; same CDR_total_score column, non-overlapping visits
AD8_Registered_PREVENTAD_2025-03-21.csv	Informant-rated dementia screening total
APS_Registered_PREVENTAD_2025-03-21.csv	Subjective cognitive complaints total
Smell_identification_Registered_PREVENTAD_2025-03-21.csv	Smell identification score
Auditory_processing_Registered_PREVENTAD_2025-03-21.csv	Auditory processing measures

CDR is split across two files because eligibility and follow-up CDR assessments were collected separately. Both files share the column name CDR_total_score; the outer merge on the visit key unites them correctly because they cover non-overlapping visit labels by study design.

---

Biomarkers — scripts/load_biomarkers.py

All main tables are visit-level and outer-merged on (CONP_ID, Visit_label).

Duplicate-draw deduplication is applied to every biomarker CSV before merging: if a participant has more than one record at the same visit (e.g. p-tau217 measured in two batches at BL00), the earliest Date_taken is kept. To change this tie-break rule, edit _load_visit() in this script.

Source CSV	Notes
Plasma_4plex_IPMS_Registered_PREVENTAD_2025-03-21.csv	Plasma Aβ42, Aβ40, ratio (IP-MS); NfL, GFAP (Simoa 4-plex)
Plasma_p-tau217_Registered_PREVENTAD_2025-03-21.csv	Plasma p-tau217 (Simoa); subject to duplicate-draw dedup
CSF_proteins_Registered_PREVENTAD_2025-03-21.csv	CSF Aβ42/40 (ELISA + Lumipulse), total tau, p-tau181, p-tau217, cytokines
BP_Pulse_Weight_Registered_PREVENTAD_2025-03-21.csv	Blood pressure, pulse, weight
Lab_Registered_PREVENTAD_2025-03-21.csv	Blood lab values
Med_use_Registered_PREVENTAD_2025-03-21.csv	Medication use flags
CSF_SomaScan_7K_proteins_Registered_PREVENTAD_2025-03-21.csv	Returned separately as df_somascan; ~7,300 CSF protein columns would dominate the main frame
Actigraphy_Registered_PREVENTAD_2025-03-21.csv	Returned separately as df_actigraphy; keyed on CONP_ID + Date_taken (no Visit_label)

---

Neuroimaging — scripts/load_imaging.py

All tables are visit-level and outer-merged on (CONP_ID, Visit_label).

Source CSV	Key operations
FreeSurfer8.0_Registered_PREVENTAD_2025-03-21.csv	Multi-run dedup: when run-001 and run-002 both exist for a visit, run-001 is kept. Edit the priority dict in load() to change this.
ASL_CBF_Registered_PREVENTAD_2025-03-21.csv	Cerebral blood flow from ASL; loaded as-is
WMH_Registered_PREVENTAD_2025-03-21.csv	Sequence pivot: one source row per MRI sequence (t1/flr/t2) is pivoted into column suffixes (e.g. WMH_Total_Volume_t1). Edit the groupby("MRI_sequence") block to include or exclude sequences.
PET_info_participants_Registered_PREVENTAD_2024-08-01.csv	PET visit mapping: maps (CONP_ID, PET_session) → Freesurfer_visit_label (the co-registered MRI visit). PET files have no Visit_label of their own.
PET_NAV_SUVR_ref-wholeCerebellum_Registered_PREVENTAD_2025-03-21.csv	NAV tau SUVR, primary reference region (whole cerebellum)
PET_NAV_SUVR_ref-cerebellumCortex_Registered_PREVENTAD_2025-03-21.csv	NAV tau SUVR, alternate reference region; columns suffixed _cblCtx to avoid collision
PET_FTP_SUVR_ref-infCerebellarGray_Registered_PREVENTAD_2024-08-01.csv	FTP tau SUVR (inferior cerebellar gray reference)
PET_FreeSurfer53_volumes_Registered_PREVENTAD_2024-08-01.csv	PET-aligned FreeSurfer53 parcellation volumes; columns prefixed PETvol_
MEG_relative_power_DK_Registered_PREVENTAD_2025-03-21.csv	Returned separately as df_meg; one row per participant, no Visit_label

---

Assembly — scripts/merge.py

The merge runs in three phases:

1. Visit spine — the union of all (CONP_ID, Visit_label) pairs across the four visit-level domain frames (MCI, cognitive, biomarkers, imaging). A participant × visit row is only created if at least one domain has an observation for it.

2. Outer merges — each domain frame is outer-merged onto the spine on (CONP_ID, Visit_label). Shared metadata columns (Date_taken, Candidate_Age, Study_visit_label) that accumulate _x/_y suffixes from repeated merges are coalesced immediately after each step via coalesce_suffixed_cols() in utils.py.

3. Participant broadcast — df_participant (one row per CONP_ID) is left-joined onto the merged frame last, broadcasting static attributes (demographics, genetics, CAIDE, family history) across all visits for each participant.

The merge order (df_mci → df_cognitive → df_biomarkers → df_imaging → df_participant) does not affect final values; it only determines which source's metadata value is preferred by coalesce_suffixed_cols when multiple tables record metadata for the same visit.

---

Feature extraction — scripts/extract_features.py

Three module-level objects control everything about the output frame without requiring changes to the merge pipeline:

• _RENAME — dict mapping source column names in preventad_merged.parquet to clean snake_case output names. Add an entry to include a new column; remove an entry to drop it. Columns not listed here are excluded from the feature frame.

• _COLUMN_ORDER — list that sets the output column order. Columns in this list that are absent from the merged frame are silently skipped; columns present in the merged frame but absent from this list are dropped.

• _DATA_DICT — BIDS-style sidecar dictionary written to preventad_features.json. Add LongName, Description, Units, and Levels entries here for any new column added to _RENAME.

Derived columns are computed inside extract() after the rename step and are each guarded by a column-presence check:

Derived column	Source	Formula
age_years	age (months)	age / 12, rounded to 1 decimal
apoe_genotype	APOE	Alleles sorted and formatted as e{a}e{b}
apoe4_count	APOE	Count of "4" alleles (0, 1, or 2)
family_hx_ad	father/mother/sibling_hx_ad	1 if any first-degree relative is 1; else 0
tmt_b_a_ratio	tmt_a, tmt_b	tmt_b / tmt_a; zero tmt_a replaced with NaN
csf_abeta_ratio_lp	csf_abeta42_lp, csf_abeta40_lp	csf_abeta42_lp / csf_abeta40_lp; zero denominator → NaN

---

Repository Layout

scripts/
  utils.py              Shared helpers: DATA_DIR, load_csv, normalize_visit_label, coalesce helpers
  load_demographics.py  Participant-level data: demographics, genetics, GWAS, CAIDE, medical history, MCI
  load_cognitive.py     Visit-level cognitive assessments: RAVLT, RBANS, TMT, DKEFS, MoCA, CDR, AD8, APS, …
  load_biomarkers.py    Visit-level biomarkers: plasma, CSF proteins, BP/labs; actigraphy and SomaScan separate
  load_imaging.py       Visit-level neuroimaging: FreeSurfer, ASL, WMH, PET SUVR/volumes; MEG separate
  merge.py              Assembles all domains → preventad_merged.parquet
  extract_features.py   Renames and derives the analysis-ready feature frame → preventad_features.parquet