Econometric Theory/Normal Equations Proof

Below is the proof of the Normal Equations for OLS.

The goal of OLS is to minimize the sum of squared error terms to find the best fit.

Known: $\widehat{{ϵ}_i}=Y_i-\widehat{Y_i}=Y_i-\alpha -\beta X_i$

$\sum {\widehat{{ϵ}_i}}^2$

$=\sum (Y_i-\widehat{Y_i}{)}^2$

$=\sum (Y_i-\hat{\alpha }-\hat{\beta }{X}_i{)}^2$

$mi{n}_{\alpha ,\beta }\sum {\widehat{{ϵ}_i}}^2\frac{\partial \sum {\widehat{{ϵ}_i}}^2}{\partial \hat{\alpha }}=\sum 2\widehat{{ϵ}_i}\frac{\partial \widehat{{ϵ}_i}}{\partial \hat{\alpha }}=2\sum \widehat{{ϵ}_i}(-1)=2\sum (Y_i-n\hat{\alpha }-\hat{\beta }{X}_i)(-1)=0$

$\frac{\partial \sum {\widehat{{ϵ}_i}}^2}{\partial \hat{\beta }}=\sum 2\widehat{{ϵ}_i}\frac{\partial \widehat{{ϵ}_i}}{\partial \hat{\alpha }}=2\sum \widehat{{ϵ}_i}(-X_i)=2\sum (Y_i-\hat{\alpha }-\hat{\beta }{X}_i)(-X_i)=0$

So we have two equations:

$\sum (Y_i-n\hat{\alpha }-\hat{\beta }{X}_i)(-1)=0$

and

$\sum (Y_i-\hat{\alpha }-\hat{\beta }{X}_i)(-X_i)=0$

setting them both equal to $\sum Y_i$

We get

$\sum Y_i=n\hat{\alpha }+\hat{\beta }\sum X_i$

and

$\sum Y_i{X}_i=\hat{\alpha }\sum X_i^2+\hat{\beta }\sum X_i^2$

Divide the first equation by n

$\frac{1}{n}\sum Y_i=\hat{\alpha }+\frac{1}{n}\hat{\beta }\sum X_i$

Leaves us with (${\sum }^n{W}_i\frac{1}{n}=\overline{W}$)

$\overline{Y}=\hat{\alpha }+\hat{\beta }\overline{X}\iff \hat{\alpha }=\overline{Y}-\hat{\beta }\overline{X}$

Now we know how to get α(hat), we can work on β(hat)

$\sum Y_i{X}_i=\hat{\alpha }\sum X_i^2+\hat{\beta }\sum X_i^2=[\overline{Y}-\hat{\beta }\overline{X}]\sum X_i^2+\hat{\beta }\sum X_i^2=[\frac{(\sum X_i)(\sum Y_i)}{n}]+\hat{\beta }[\sum X_i^2-\frac{(\sum X_i{)}^2}{n}]$

We can move β(hat) to one side

$\hat{\beta }=\frac{\sum Y_i{X}_i-\frac{(\sum X_i)(\sum Y_i)}{n}}{\sum X_i^2-\frac{(\sum X_i{)}^2}{n}}$

And now we have our Normal equations for OLS.