Econometric Theory/Ordinary Least Squares (OLS)

Ordinary Least Squares or OLS is one of the more simple (if you can call it so) methods of linear regression. The goal of OLS is to closely "fit" a function with the data. It does so by minimizing the sum of squared errors from the data.

Why we Square Errors before Summing

Notice that we are not just trying to minimize the sum of errors, but rather the sum of squared errors. Lets take a brief look at our sweater story again.

Model A

Model B

model	data point	error from line
A	1	5
A	2	10
A	3	-5
A	4	-10
B	1	3
B	2	-3
B	3	3
B	4	-3

Notice that the Sum of Model A $\sum _{}5+10-5-10=0$ and that the Sum of Model B $\sum _{}3-3+3-3=0$

Both Models sum to 0 and both are great fits! NO!!

So to account for the signs, when ever we sum errors, we square the terms first.

The Model

The line that we want to estimate with OLS is $Y_{=}\alpha +\beta X_i+{ϵ}_i$ Where α and β are parameters, Y and X are variables and ε is the stochastic error term. Our parameters are unknown, and so we must estimate the parameters, creating statistics. Generally, parameters are lower case greek letters, and our statistics are the same wearing a "hat". Our estimated model will look like this: $\hat{Y}=E(Y|X=x)=\hat{\alpha }+\hat{\beta }{x}_i$

The error terms are not parameters, but can be estimated using $\widehat{epsilo{n}_i}=Y_i-\widehat{E(Y|X=x}$

Our parameters are estimated using the Normal Equations, which are unrelated to the Normal Distribution.

The Normal Equations

The Normal Equations will give us estimates for our parameters α and β so that we can estimate our model.

$\hat{\beta }=\frac{\sum Y_i{X}_i-\frac{(\sum X_i)(\sum Y_i)}{n}}{\sum X_i^2-\frac{(\sum X_i{)}^2}{n}}$

$\hat{\alpha }=\overline{Y}-\hat{\beta }\overline{X}$