Machine Learning Homework Assignments

Project Overview

This repository contains solutions to a series of machine learning homework assignments, implemented in Python using Jupyter Notebooks. The assignments cover fundamental concepts in regression, classification, and probability density estimation, utilizing standard data science libraries like NumPy, Pandas, Matplotlib, Scikit-learn, and SciPy. The primary datasets used are the Seoul Bike Sharing Demand dataset and the Scikit-learn Digits dataset.

Contents

This project is divided into four main assignments, each contained within its respective Jupyter Notebook:

1. HW2.ipynb: Assignment 1 - Linear Regression and Regularization
2. HW3.ipynb: Assignment 2 - Decision Trees for Regression
3. HW4.ipynb: Assignment 3 - Probability Density Estimation
4. HW5.ipynb: Assignment 4 - Classification Algorithms

---

Assignment Details

1. HW2.ipynb: Linear Regression and Regularization

• Focus: Predictive modeling of bike rentals using Linear Least Squares (LLS).
• Key Concepts: LLS implementation (from scratch & sklearn), polynomial feature engineering, train/validation/test splitting, RMSE evaluation, overfitting analysis, Ridge (L2) and LASSO (L1) regularization for model complexity control, and hyperparameter tuning (λ) using validation data.

2. HW3.ipynb: Decision Trees for Regression

• Focus: Applying Decision Trees to the bike rental regression problem.
• Key Concepts: Implementation of regression stumps and a general tree node, greedy tree growing algorithm based on RMSE reduction, identification of overfitting with tree depth, post-pruning using a validation set, and comparison with Scikit-learn's DecisionTreeRegressor.

3. HW4.ipynb: Probability Density Estimation

• Focus: Estimating and applying probability density functions (PDFs) using UTMB race data.
• Key Concepts: Non-parametric estimation via histograms, parametric estimation using the Beta distribution, parameter fitting via Maximum Likelihood Estimation (MLE) using grid search, comparison with scipy.stats.beta, and calculating probabilities through numerical integration of the joint PDF.

4. HW5.ipynb: Classification Algorithms

• Focus: Implementing and comparing various classification methods on the handwritten digits dataset.
• Key Concepts: Misclassification rate evaluation, K-Nearest Neighbors (KNN) (from scratch & sklearn) with K selection, Linear Discriminant Analysis (LDA) (from scratch & sklearn) using Mahalanobis distance, Linear Logistic Regression (from scratch) optimized with Gradient Descent, learning rate selection, and final model comparison on validation/test sets.

---

Key Concepts Explored

• Supervised Learning: Regression (Linear Regression, Decision Trees) and Classification (KNN, LDA, Logistic Regression).
• Model Evaluation: RMSE, Misclassification Rate.
• Model Training & Optimization: Least Squares, Greedy Algorithms, Gradient Descent, Maximum Likelihood Estimation.
• Feature Engineering & Complexity Control: Polynomial Features, Regularization (L1/L2), Pruning.
• Overfitting & Validation: Train/Validation/Test Splits, Hyperparameter Tuning.
• Probability & Statistics: Parametric/Non-parametric PDF Estimation, MLE, Numerical Integration.

---

Technologies Used

• Python 3
• NumPy
• Pandas
• Matplotlib
• Scikit-learn
• SciPy
• Graphviz (for HW3 tree visualization)

---