Cross-Validation Explained

Cross-Validation

Cross-validation helps measure model performance more reliably by using multiple subsets of the data instead of a single validation set.

Why Not Use a Single Validation Set?

• Using only one validation set can give noisy or luck-dependent results.
• Example: In a dataset with 5000 rows, keeping 1000 as validation may give a misleading score.

How Cross-Validation Works

1. Split data into k folds (e.g., 5 folds, each 20% of the data).
2. For each fold:
   • Use the fold as the validation set.
   • Use remaining folds for training.
3. Repeat for all folds so every row is used for validation once.
4. Average the performance metrics across all folds for a reliable score.

When to Use Cross-Validation

• Small datasets: Recommended, because you can reuse all data for validation.
• Large datasets: Single validation set is often sufficient.

Implementation Example (Python)

from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score

my_pipeline = Pipeline(steps=[
    ('preprocessor', SimpleImputer()),
    ('model', RandomForestRegressor(n_estimators=50, random_state=0))
])

# 5-fold cross-validation
scores = -1 * cross_val_score(
    my_pipeline, X, y, cv=5, scoring='neg_mean_absolute_error'
)

print("MAE scores:", scores)
print("Average MAE:", scores.mean())

Summary

• Reduces randomness in model evaluation.
• Gives a more accurate measure of performance.
• Very useful for small datasets or when comparing models.
• Pipelines simplify cross-validation and make code cleaner.