Least Squares Problem

Question

Least squares: Consider the data arranged as ordered pairs (0, 0), (3, 4), and (5, 8). Construct the error function E(m, b) as discussed in class (it does not need to be simplified). Give the two associated partial derivatives, Em and Eb. Use your line to predict the value of y when x = 2.

Answer 1

Hi Keegan! Thanks for asking a good question! To answer your question, we need to understand how least squares work. The idea behind it is that we want to minimize the difference between the true y-value and the predicted best-line-fitted y-values (this difference, in statistics, is called the residual). One application of multivariable calculus includes minimizational optimization!

We know that the best line of fit is y1 = mx + b, where y1 is the predicted y-value, m is slope, and b is y-intercept. Note that in this context, y and x are data numbers, and m and b are our variables (because we are interested in finding these numbers)!

We will consider the sum of the squares of the residuals to be:

E = ∑ (y - y1) ^ 2 = ∑ ( y - (mx + b) ) ^ 2 = ∑ ( y ^ 2 - 2y * (mx + b) + (mx+b) ^ 2 )

E (m, b) = ∑ ( y ^ 2 ) - 2 * ∑ ( y * (mx + b) ) + ∑ (mx + b) ^ 2

Becuase since the function has been slightly simplified, substitute the data numbers (x, y) into the summation formula and try to further simplify it!

∑ y ^ 2 = 0 ^ 2 + 4 ^ 2 + 8 ^ 2 = 80

∑ ( y * (mx + b) ) = 0 * (0m + b) + 4 * (3m + b) + 8 * (5m + b) = 52m + 12b

∑ (mx + b) ^ 2 = (0m + b) ^ 2 + (3m + b) ^ 2 + (5m + b) ^ 2 = 34m ^ 2 + 16mb + 3b ^ 2

Reentering these values into the E (m, b) functions:

E (m, b) = 80 - 2 * ( 52m + 12b ) + ( 34m^2 + 16mb + 3b^2 ) = 80 - 104m - 24b + 34m^2 + 16mb + 3b^2

Because we have two variables m and b of interest, we can use partial derivatives to find minimum values and set the partial derivatives equal to 0:

∂/∂m (80 - 104m - 24b + 34m^2 + 16mb + 3b^2) = -104 + 68m + 16b = 0 ⇒ 104 = 68m + 16b

∂/∂b (80 - 104m - 24b + 34m^2 + 16mb + 3b^2) = -24 + 16m + 6b = 0 ⇒ 24 = 16m + 6b

System of equations! We can either employ substitution or elimination (either works) to find:

104 = 68m + 16b ⇒ 26 = 17m + 4b

24 = 16m + 6b ⇒ 12 = 8m + 3b

Therefore, we conclude that m = 30/19 and b = -4/19 yield the line of best fit; in other words:

y1 = 30x / 19 - 4 / 19 <<<<< ANSWER!

Using the line, y1 = 30 * 2 / 19 - 4 / 19 = 56 / 19 <<<<< ANSWER!