Why do we use a Least Squares fit?

Question

I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the *square* of the errors between the line and the data points. If we use a less common method like LAD, where we just consider the absolute deviation, then outliers make less difference to the final model, while if we take the *cube* of the error (or any other power higher than 2), then outliers are far more significant than with the least squares model. I suppose what I'm really asking is *mathematically*, is raising the error to the power of 2 really that special. Is it say more "accurate" in some sense than raising the error to the power of 1.95 or 2.05??? Thanks!

Answer 1

No

Consider the following functions:

y = f(x) = sqrt(x^2) vs g(x) = |x|

THEY ARE EQUAL!!! They are the exact same function...

So by squaring the error and then taking the square root, we are finding the "average" of

the deviations.

Secondly, the least squares formula come from the definition that the trend line satisfies the property

that the SUM of the distances from the line to each data point is zero.