Overview on Regularization

Ridge and Lasso regression are powerful techniques generally used for creating parsimonious models in presence of a ‘large’ number of features. Here ‘large’ can typically mean either of two things:

  1. Large enough to enhance the tendency of a model to overfit (as low as 10 variables might cause overfitting)
  2. Large enough to cause computational challenges. With modern systems, this situation might arise in case of millions or billions of features

Though Ridge and Lasso might appear to work towards a common goal, the inherent properties and practical use cases differ substantially. If you’ve heard of them before, you must know that they work by penalizing the magnitude of coefficients of features along with minimizing the error between predicted and actual observations. These are called ‘regularization’ techniques. The key difference is in how they assign penalty to the coefficients:

  1. Ridge Regression:
  2. Performs L2 regularization, i.e. adds penalty equivalent to square of the magnitude of coefficients
  3. Minimization objective = LS Obj + α * (sum of square of coefficients)
  4. Lasso Regression:
  5. Performs L1 regularization, i.e. adds penalty equivalent to absolute value of the magnitude of coefficients
  6. Minimization objective = LS Obj + α * (sum of absolute value of the coefficients)

Note that here ‘LS Obj’ refers to ‘least squares objective’, i.e. the linear regression objective without regularization.

If terms like ‘penalty’ and ‘regularization’ seem very unfamiliar to you, don’t worry we’ll talk about these in more detail through the course of this article. Before digging further into how they work, lets try to get some intuition into why penalizing the magnitude of coefficients should work in the first place.

Why shrink coefficients at all?

Ordinary least squares (OLS) is happy to fit noise. If two features are correlated, OLS will often give both large coefficients with opposite signs. The fit on the training data looks great and the model falls apart on new data. That is high variance.

A large coefficient says “this feature gets a lot of weight.” If that feature is noisy, the prediction swings around with it. Pulling coefficients toward zero is a bias we accept on purpose. A slightly worse fit on the training set is worth a model that generalizes.

That tradeoff is the whole point of shrinkage:

α (sometimes written λ) is the knob. α = 0 is OLS. As α grows, coefficients are forced smaller. If α is huge, the model collapses toward a constant (the intercept).

One more practical reason: OLS is unstable when features are collinear or when p is close to n. Ridge still has a unique solution in those cases.

The least squares starting point

We have n observations and p features. In matrix form the linear model is

$$\hat{y} = X\beta$$

OLS minimizes residual sum of squares:

$$RSS(\beta)=\sum_{i=1}^{n}(y_i-x_i^T\beta)^2=|y-X\beta|^2_2$$

The closed form is $\hat{\beta}=(X^TX)^{-1}X^Ty$ when $X^TX$ is invertible. When columns of X are correlated, $X^TX$ is ill-conditioned and those coefficients explode. Shrinkage adds a penalty so the problem stays well posed.

Ridge regression (L2)

Ridge adds the squared L2 norm of the coefficients:

$$\hat{\beta}^{ridge}=\arg\min_{\beta}\left{|y-X\beta|^2_2+\alpha|\beta|^2_2\right}$$

The intercept is usually left unpenalized. The closed form is

$$\hat{\beta}^{ridge}=(X^TX+\alpha I)^{-1}X^Ty$$

Adding $\alpha I$ to $X^TX$ is what “shrinks” the solution. Even if two columns are almost the same, the extra diagonal term makes the matrix invertible.

Ridge never zeros out a coefficient. It spreads the weight across correlated features. That is useful when you believe many variables contribute a little.

Because the penalty depends on the scale of each column, standardize the features before fitting Ridge. Otherwise a variable measured in millimeters gets a different penalty than the same variable in meters.

Lasso regression (L1)

Lasso uses the L1 penalty:

$$\hat{\beta}^{lasso}=\arg\min_{\beta}\left{|y-X\beta|^2_2+\alpha|\beta|_1\right}$$

where $|\beta|_1=\sum_j|\beta_j|$.

The L1 ball has corners on the axes. The RSS contours hit those corners, so some coefficients become exactly zero. Lasso is OLS plus feature selection.

That sparsity is the reason people reach for Lasso when p is large and they want a smaller, readable model. The cost is that when several features are strongly correlated, Lasso tends to keep one and drop the others somewhat arbitrarily. Ridge would have kept a blend of them.

There is no simple closed form like Ridge. Coordinate descent is the usual solver, which is what scikit-learn uses.

A geometric picture

Think of the constraint forms of the same problems:

The unpenalized OLS solution sits somewhere in the plane. RSS is a family of ellipses around that point. The first place an ellipse touches the diamond is often a vertex, so a coefficient is zero. The disk is round, so the touch point is almost never on an axis.

α maps to t. Larger α is a smaller disk or diamond.

Elastic Net

If you want sparsity and a more stable treatment of correlated features, Elastic Net mixes both penalties:

$$\hat{\beta}^{en}=\arg\min_{\beta}\left{|y-X\beta|^2_2+\alpha\left(\rho|\beta|_1+\frac{1-\rho}{2}|\beta|^2_2\right)\right}$$

ρ = 1 is Lasso. ρ = 0 is Ridge. Values in between are the usual practical default when you have grouped, correlated predictors.

Picking α

Do not pick α by staring at the training R². Use cross-validation.

A useful picture is the regularization path: fit the model over a grid of α and plot each coefficient. In Ridge the paths decay smoothly toward zero. In Lasso they hit zero and stay there.

scikit-learn’s RidgeCV and LassoCV do this for you.

Python example

We’ll use the diabetes dataset and compare OLS, Ridge, and Lasso on a holdout set. Features are standardized so the penalties are comparable.


import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error, r2_score

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42)

def report(name, model):
    pred = model.predict(X_test)
    if name == 'Ridge':
        coef = model.named_steps['ridgecv'].coef_
    elif name == 'Lasso':
        coef = model.named_steps['lassocv'].coef_
    else:
        coef = model.named_steps['linearregression'].coef_
    print(name)
    print('  RMSE', round(np.sqrt(mean_squared_error(y_test, pred)), 3))
    print('  R2  ', round(r2_score(y_test, pred), 3))
    print('  nonzero coefs', np.sum(np.abs(coef) > 1e-8), '/', coef.size)


ols = make_pipeline(StandardScaler(), LinearRegression())
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-3, 3, 50)))
lasso = make_pipeline(StandardScaler(), LassoCV(alphas=np.logspace(-3, 3, 50), cv=5))

ols.fit(X_train, y_train)
ridge.fit(X_train, y_train)
lasso.fit(X_train, y_train)

report('OLS', ols)
report('Ridge', ridge)
report('Lasso', lasso)

print('Ridge alpha', ridge.named_steps['ridgecv'].alpha_)
print('Lasso alpha', lasso.named_steps['lassocv'].alpha_)
print('Lasso coefficients')
print(np.round(lasso.named_steps['lassocv'].coef_, 3))

On this data you should see Ridge and Lasso land in a similar RMSE neighborhood as OLS, with Lasso using fewer features. That is the shrinkage bargain in miniature: we give up a bit of training flexibility and keep a stabler, smaller model.

To plot a Lasso path:


import matplotlib.pyplot as plt
from sklearn.linear_model import lasso_path

alphas, coefs, _ = lasso_path(StandardScaler().fit_transform(X_train), y_train)
plt.figure(figsize=(8, 5))
plt.plot(np.log10(alphas), coefs.T)
plt.xlabel('log10(alpha)')
plt.ylabel('coefficient')
plt.title('Lasso regularization path')
plt.show()

As log(α) increases, paths snap to zero one after another. The features that survive the longest are the ones Lasso considers most useful.

Which one should you use?

Standardize numeric features. Leave dummy variables on a 0/1 scale or be consistent. Do not penalize the intercept. Choose α with cross-validation, not by hand.

Takeaways

Regularization is just OLS with a budget on how large the coefficients are allowed to be. L2 (Ridge) shrinks everything but keeps the whole team on the field. L1 (Lasso) is willing to send coefficients to zero, which is why it doubles as a feature selector. The penalty is extra bias paid to buy down variance, and α is how much you are willing to pay.

References: