A reproducible pipeline that predicts socioeconomic and physical disparity scores across Tehran's informal/peri-urban periphery, using space syntax centrality measures and a model trained on formal Tehran, then aggregates results into a composite Spatial Disparity Index (SDI) at H3 hexagon resolution.
Live dashboard: sepehrzh.github.io/sdi
Formal Tehran has surveyed socioeconomic and physical attribute data (population density, dilapidation status, mean plot area, land use, policy era of development). The periphery does not. Space syntax centrality measures, which describe the structural and accessibility properties of the street network, are available across the entire network, both formal and peripheral.
This pipeline:
- Compresses centrality features into principal components via PCA, fit only on formal Tehran
- Trains classifiers/regressors on formal Tehran to predict each surveyed attribute from those components
- Applies the trained models to the periphery, where ground truth doesn't exist
- Aggregates predictions into a single weighted, composite SDI per H3 hexagon, with weights derived from each model's own performance (Cramér's V for categorical attributes, R² for continuous)
The project serves three purposes simultaneously: an academic exercise with a reproducible pipeline, an experimental tool applicable to practice, and a potential contribution into a development process.
| Stage | Detail |
|---|---|
| Network features | 35 space syntax features: Choice (CH) and Integration (INT), non-normalised, at 8 metric radii (400–40,000m); Node Count (NC) and Total Depth (TD) at the same 8 radii; angular connectivity, topological connectivity, segment length. No global radius (Rn). |
| Dimensionality reduction | PCA(3) fit on CH+INT only (16 columns: 8 radii × 2 families), training set only — never refit on the prediction set, to avoid data leakage |
| Structural typology | KMeans (k=12) on CH+INT, fit on the full network — produces whole_CH_2, a visualisation-only typological axis with no SDI weight |
| Predictive models | Separate MLPClassifier per categorical attribute (policy_era, dilap_index, landuse); single MLPRegressor for continuous attributes (mean_area, perimeter, mean_pop) |
| Model inputs | PC1, PC2, PC3 (StandardScaler-transformed) |
| Weighting | Cramér's V (categorical) / R² (continuous) per attribute, MinMax-scaled to 0–1, normalised to sum to 1 |
| Aggregation | Segment-level scores joined to H3 resolution 8 hexagons (~0.74 km²/hex) by centroid, mean-aggregated |
See Known limitations for documented weaknesses in this approach — several are acknowledged in the accompanying paper rather than "fixed," because the fixes attempted (e.g. oversampling) didn't hold up.
segments.geojson (full network: formal Tehran + periphery)
│
▼
01_pca_clustering.ipynb PCA(3) on training-set CH+INT → PC1–3
KMeans(12) on full network CH+INT → whole_CH_2
saves: models/pca_scaler.joblib, models/pca_model.joblib
│
▼
02_model_train_evaluate.ipynb Per-attribute MLPClassifier ×3, MLPRegressor ×1
trained on PC1–3 from formal Tehran (labelled)
saves: trained_models/, model_scores.json
│
▼
03_predict.ipynb Transforms periphery CH+INT → PC1–3 using saved
scaler/PCA (no refit), applies trained models
│
▼
04_sdi.ipynb Maps categorical predictions to scores via
category_keys.json, MinMax-scales everything,
weights by model performance, computes SDI
│
▼
05_h3_aggregation.ipynb Aggregates segments to H3 res-8 hexagons,
computes quantile breaks, uploads to GCS
│
▼
hex.geojson + classification.json → gs://sdi-tehran-assets/
→ consumed by the dashboard
sdi-tehran/
├── README.md
├── LICENSE
├── .gitignore
├── requirements.txt
├── config.py # single source of truth: paths, constants, hyperparameters
├── notebooks/
│ ├── 01_pca_clustering.ipynb
│ ├── 02_model_train_evaluate.ipynb
│ ├── 03_predict.ipynb
│ ├── 04_sdi.ipynb
│ └── 05_h3_aggregation.ipynb
├── data/
│ ├── README.md # data dictionary, sourcing notes, what's not included
│ ├── category_keys.json # categorical label → numeric score mapping
│ └── model_scores.json # per-attribute model performance (generated by NB02)
├── models/ # PCA scaler/model artefacts (generated by NB01, gitignored)
├── trained_models/ # MLP classifiers/regressor + label encoders (generated by NB02, gitignored)
├── evaluation_reports/ # per-attribute classification/regression reports (generated by NB02, gitignored)
└── paper/
└── draft.md
Raw and processed geospatial/survey data, model artefacts, and pipeline outputs are not committed to this repository — see data/README.md for why, and where the public outputs live instead.
git clone https://github.com/sepehrzh/sdi-tehran.git
cd sdi-tehran
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txtRequires: Python 3.9+. The pipeline was developed and run on Python 3.9 — functional but past end-of-life; some Google client library imports emit FutureWarning. An upgrade is recommended for anyone forking this for active development.
Notebooks must be run in order, NB01 → NB05. Each notebook reads its inputs from paths defined in config.py and writes outputs to data/outputs/ (gitignored).
jupyter labOnly 05_h3_aggregation.ipynb touches Google Cloud, and only to upload hex.geojson and classification.json to the public bucket that backs the dashboard. This step is optional if you only want to reproduce the pipeline locally.
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/cloud-platformNo service account keys are stored anywhere in this repository — authentication is via Application Default Credentials (ADC) on the local machine running the upload.
The pipeline's final public outputs are hosted on Google Cloud Storage and consumed directly by the dashboard:
| File | URL |
|---|---|
hex.geojson |
https://storage.googleapis.com/sdi-tehran-assets/data/outputs/hex.geojson |
classification.json |
https://storage.googleapis.com/sdi-tehran-assets/data/outputs/classification.json |
category_keys.json |
https://storage.googleapis.com/sdi-tehran-assets/data/category_keys.json |
13,848 H3 resolution-8 hexagons, each carrying an SDI score, per-attribute component scores, quantile class bins, and a structural typology label (whole_CH_2).
These are documented methodological limitations, not bugs. Each will be acknowledged in the paper rather than silently corrected, because attempted fixes didn't hold up under inspection. In fact, the identified limitations are among the novelties of the project that are worth digging into. This includes studying other predictive methods, hyper parameters, and a deep study into the validity of the relationship between centrality measures and socioeconomic metrics.
-
mean_areahas negative R² (≈ −0.0003). The regressor performs worse than a mean baseline for this attribute. Its SDI weight is correspondingly near-zero (the weighting scheme correctly discounts a poorly-performing predictor), so the effect on the composite index is negligible. -
landuseclassifier collapses to the residential class. Severe class imbalance in the training set (~79k residential vs ~3.6k across all other classes) causes the classifier to predict residential almost everywhere. An oversampling experiment (minority classes resampled to majority size, ~9× dataset inflation) was attempted and rejected: the model failed to converge within 500 iterations, accuracy dropped to 0.15, and precision on minority classes was near-zero. Conclusion: PCA components derived from centrality measures alone carry insufficient signal to discriminate land use, which is a function of land economics rather than network structure. Low SDI weight (0.1058) limits the impact on the composite index. Future remediation: SMOTE, deeper feature engineering, or a dedicated remote-sensing/cadastral land use layer. -
dilap_indexis near-collapsed, skewed heavily toward the best-condition class due to similar class imbalance in the training data (~150:1 ratio).class_weight='balanced'alone is insufficient at this ratio. -
score_policy_era_classspans only classes 1–2 of a possible 5, rather than the full range.
No spatial cross-validation or hyperparameter search has been performed — the current model is a v1 baseline. These are listed as deferred future work, not omissions.
A formal citation will be added once the paper is finalised. In the meantime, please reference this repository directly if you build on this work.
MIT — placeholder, update if a different license is preferred before publishing.