Machine Learning Pipeline- EDA + IQR

End-to-End Machine Learning Pipeline

A complete workflow covering data preprocessing, exploratory data analysis (EDA), feature engineering, and model preparation using real-world structured datasets.

Pipeline Overview

1. Data Understanding

Initial inspection includes checking data types, missing values, duplicates, and overall dataset structure.

2. Target Distribution

3. Numerical Feature Analysis

4. Outlier Detection

Outliers are detected using the Interquartile Range (IQR) method:

Lower = Q1 - 1.5 × IQR
Upper = Q3 + 1.5 × IQR

5. Feature Engineering

• Label Encoding (categorical → numeric)
• Feature Scaling (StandardScaler)
• Dimensionality Reduction (PCA)

6. Model Training

A Random Forest Classifier is used due to its robustness, ability to handle non-linear relationships, and resistance to overfitting.

Pipeline Summary

Data → Cleaning → EDA → Feature Engineering → Model → Evaluation
  

Exploratory Data Analysis (EDA) & Outlier Detection Pipeline

This section explains how Exploratory Data Analysis (EDA) and Outlier Detection using IQR are applied in the machine learning pipeline. The goal is to understand feature distributions, relationships, and data quality before modeling.

EDA Pipeline Overview

Load Data

→

Data Summary

→

Missing Values

→

Distribution Analysis

→

Correlation

→

Outlier Detection (IQR)

1. Exploratory Data Analysis (EDA)

EDA helps in understanding the dataset structure and feature behavior.

• Data Shape: Understand dataset size
• Data Types: Identify numerical and categorical features
• Missing Values: Detect incomplete data
• Feature Distributions: Using histograms and KDE plots
• Target Analysis: Class balance check

Target Encoding

\[ \text{Target} = \{Low = 0,\ Medium = 1,\ High = 2\} \]

2. Correlation Analysis

Correlation measures the relationship between features and the target variable.

\[ r = \frac{\text{cov}(X, Y)}{\sigma_X \sigma_Y} \]

Features with higher correlation values are more influential for prediction.

3. Distribution Analysis

Histograms and KDE plots are used to understand:

• Skewness of data
• Normal vs non-normal distributions
• Presence of extreme values

4. Outlier Detection using IQR Method

The Interquartile Range (IQR) method identifies extreme values in numerical features.

\[ IQR = Q3 - Q1 \] \[ \text{Lower Bound} = Q1 - 1.5 \times IQR \] \[ \text{Upper Bound} = Q3 + 1.5 \times IQR \]

Any data point outside these bounds is considered an outlier.

Implementation (Python)

Q1 = df[col].quantile(0.25)
Q3 = df[col].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

outliers = df[(df[col] < lower) | (df[col] > upper)]

Summary: EDA helps you understand your data, while IQR ensures that extreme values do not distort your model performance.