Lasso in Machine Learning

Lasso in Machine Learning

Lasso is a regularization technique used in linear models to prevent overfitting and perform feature selection. Its full name is Least Absolute Shrinkage and Selection Operator.

1. Purpose

• Reduce model complexity by penalizing large coefficients.
• Automatically shrinks some coefficients to zero, removing less important features.
• Useful for high-dimensional data (many features).

2. Mathematical Formulation

For a linear regression:

y = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ

Lasso adds an L1 penalty to the cost function:

Cost = MSE + α * Σ|βᵢ|

Where:

• MSE = mean squared error
• α (alpha) = regularization strength
• |βᵢ| = absolute value of coefficient

The L1 penalty forces some coefficients exactly to zero, performing feature selection.

3. How It Works

1. Fits a linear model like ordinary regression.
2. Adds a penalty proportional to the sum of absolute coefficients.
3. Minimizes the penalized error.
4. Coefficients of unimportant features shrink to 0, while important ones remain.

4. When to Use

• High-dimensional datasets (more features than samples).
• When you want automatic feature selection.
• To reduce overfitting in linear models.

5. Python Example

from sklearn.linear_model import Lasso
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Load example dataset
X, y = load_breast_cancer(return_X_y=True)

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Lasso regression
lasso = Lasso(alpha=0.01)
lasso.fit(X_train_scaled, y_train)

# Coefficients
print("Lasso coefficients:", lasso.coef_)

Lasso is for regression (continuous target), but can also be used for feature selection in classification tasks by encoding labels numerically.