A Hybrid Framework for LGD Estimation

Overview

This repository contains two main scripts for preparing and modelling Loss Given Default (LGD) on LendingClub loan data.
The framework combines Explainable Boosting Machines (EBM) for feature selection with Beta Regression for statistical calibration, and extends into segmented LGD bands with decision-tree and multinomial logistic gates.

---

Files

1. data_processing.py

• Purpose: Prepares LendingClub loan-level datasets for LGD modelling.
• Key steps:
  1. Compute realized LGD for defaulted/charged-off loans.
  2. Clean and transform borrower, loan, and payment features.
  3. Classify job titles into NAICS 2-digit sectors using a transparent, rule-based classifier.
  4. Merge point-in-time macroeconomic and industry-level covariates (e.g., GDP, inflation, consumer spending).
  5. Export a clean, analysis-ready dataset.

---

2. lgd_modeling.py

• Purpose: Implements the hybrid LGD modelling pipeline.
• Key components:
  1. Train single EBM models with hyperparameter tuning for feature ranking.
  2. Perform correlation and redundancy analysis on top-ranked features.
  3. Estimate baseline Beta Regression with probit/logit links.
  4. Extend to segmented Beta Regression with:
     • Decision Tree gates (soft and hard).
     • Multinomial Logistic regression gates.
  5. Evaluate models with metrics (R², MAE, RMSE) and residual diagnostics.
  6. Visualise predictive distributions, mixture densities, and marginal effects.

---

Data

The modelling is based on LendingClub loan-level data:

• Accepted applications: accepted_2007_to_2018Q4.csv
• Rejected applications: rejected_2007_to_2018Q4.csv

⚠️ Note: These files are not uploaded here due to their large size.
They can be downloaded directly from the Kaggle LendingClub Loan Data.

---

Dependencies

Python libraries:

• numpy, pandas, matplotlib, seaborn, scipy, statsmodels
• scikit-learn, interpret (for EBM)
• xgboost (optional)
• rpy2 (for R–Python integration)

R packages (via rpy2):

• betareg
• stats

---

Workflow

1. Data preparation
   Run data_processing.py to generate a clean dataset with engineered features.

2. LGD modelling
   Run lgd_modeling.py to:

   • Train EBM and Beta Regression models.
   • Compare predictive and statistical performance.
   • Generate plots and diagnostics.

---

AI / LLM Acknowledgements

• Design and methodology were fully developed by the author.
• Code implementation was supported with AI assistance:
  • Claude (Anthropic): keyword expansion for job–industry classification.
  • ChatGPT (OpenAI): coding support, documentation structuring, and simplifications.
• All outputs and modelling decisions were critically reviewed and validated by the author.

---

Acknowledgments

Special thanks to:

• Professor Gianluca Fusai (City St Georges, University of London) —
  For insightful discussions on loss modeling and constructive feedback on the approach.

• Mr. Massimo Liquori (BDO) For practical guidance on risk model validation and valuable comments that improved the robustness of the implementation.

---

Author

Gael Chen
Project: From Machine Learning to Statistical Rigor: A Hybrid Framework for LGD Estimation
Date: September 2025

---