Least Squares Regression

Ordinary Least Squares (OLS)

Estimates the parameters in a regression model by minimizing the sum of squared residuals.

OLS Assumptions:

1. Model is linear in the coefficients and the error term
2. The error term has a population mean of zero
3. All independent variables are uncorrelated with error term
4. Observations of the error term are uncorrelated with each other
5. The error term has a constant varinace (no heteroskedasticity)
6. No independent variable is a perfect linear function of other explanatory variables
7. The error term is normally distributed

Mathematical Form

Basically, given a vector of inputs we predict the output or endogenous variable via the following model.

$$\hat{Y} = \hat{B}0\sum^p{j=1}X_j\hat{\beta}_j$$

B0=intercept

Intercepts are the known bias inherent within a model.

Let's rewrite the model in linear model in vector form.

ˆY=XTˆβ

RSS(β)=N∑i=1(yi−xTiβ)2

References: