Class 8: Deriving OLS Estimators
Key Ideas
A linear regression model posits that each observation is generated by
\[y_i = \beta_0 + \beta_1 x_i + u_i,\]
where \(\beta_0\) and \(\beta_1\) are fixed but unknown parameters and \(u_i\) is an unobservable error term. We never see \(\beta_0\), \(\beta_1\), or \(u_i\); we see only the pairs \((x_i, y_i)\).
An estimator is a rule that turns data into a guess about a parameter. Hats mark the guesses: \(\hat{\beta}_0\) and \(\hat{\beta}_1\) define a fitted line \(\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i\), and the gap between an observation and its fitted value is the residual \(e_i = y_i - \hat{y}_i\). Residuals are computable; errors are not.
Any line produces residuals, so “best fit” requires a criterion. Ordinary least squares (OLS) chooses \((\hat{\beta}_0, \hat{\beta}_1)\) to minimize the sum of squared residuals. Because that sum is a quadratic function of the two unknowns, the minimum is found by setting both partial derivatives to zero, and the solution can be written in closed form using only \(x_i\), \(y_i\), and \(n\).
Summation rules. The derivation manipulates sums over \(i = 1, \ldots, n\). Four rules, established in Q19, are all that it requires. In each, \(c\) is any quantity that does not depend on \(i\): a number, a data summary such as \(\bar{x}\), or a candidate coefficient such as \(\hat{\beta}_1\).
- \(\sum_{i=1}^{n} (a_i + b_i) = \sum_{i=1}^{n} a_i + \sum_{i=1}^{n} b_i\). A sum splits across addition.
- \(\sum_{i=1}^{n} c\, a_i = c \sum_{i=1}^{n} a_i\). A factor that does not depend on \(i\) comes out of the sum.
- \(\sum_{i=1}^{n} c = nc\). A constant summed \(n\) times is \(n\) copies of it.
- \(\sum_{i=1}^{n} x_i = n\bar{x}\), by the definition of the mean.
Two expressions that look like rules are not: \(\sum x_i y_i \neq \sum x_i \sum y_i\) and \(\sum x_i^2 \neq \left(\sum x_i\right)^2\). Sums of products and sums of squares must be carried along as they are.
Problems of the Day
Q20: The Judge
Three observations:
| \(x\) | 0 | 2 | 4 |
|---|---|---|---|
| \(y\) | 1 | 5 | 6 |
Three students each propose a line of best fit. Ana’s line is \(\hat{y} = 1 + \frac{5}{4} x\). Ben’s line is \(\hat{y} = 4 + 0.5x\). Cal’s line is \(\hat{y} = 1.5 + \frac{5}{4} x\). Plot the points and all three lines.
You are the judge. For each line, fill in a table of fitted values \(\hat{y}_i\) and residuals \(e_i\). Before you start, decide: is \(e_i = y_i - \hat{y}_i\) or \(e_i = \hat{y}_i - y_i\)? Explain what a positive residual means under your choice, and why the convention matters for the phrase “the line underpredicts.”
Now compare the lines using three possible scores: the sum of the residuals \(\sum e_i\), the sum of their absolute values \(\sum |e_i|\), and the sum of their squares \(\sum e_i^2\). Which line wins under each score?
Extension. Two of the three scores are unusable as a definition of “best fit.” Say which two, and for each one describe a line that would beat all three students under it.
The method of least squares picks the line that minimizes \(\sum_{i=1}^n e_i^2\).
Q21: The Bottom of the Bowl
In this question, we’ll derive formulas for \(\hat{\beta}_0\) and \(\hat{\beta}_1\), the intercept and slope of the line of best fit, in terms of the data alone. Strap in: we’re about to do some math.
Q20 ended with a definition: the line of best fit is the one that makes \(\sum e_i^2\) as small as possible. Written as an optimization problem, we are looking for the pair of numbers \((\hat{\beta}_0, \hat{\beta}_1)\) that solves
\[\min_{\hat{\beta}_0,\, \hat{\beta}_1} \; S(\hat{\beta}_0, \hat{\beta}_1) = \sum_{i=1}^n \left(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i\right)^2 .\]
The data are fixed; the only things we get to move are \(\hat{\beta}_0\) and \(\hat{\beta}_1\). Because \(S\) is a sum of squares, its graph is a bowl, and the bottom of a bowl is where the slope is zero in every direction. That is why we’ll take derivatives: one with respect to \(\hat{\beta}_0\) and one with respect to \(\hat{\beta}_1\), each set equal to zero. The only derivative we will need is the slope of a parabola.
Step 1: Square out the trinomial term by term: there are nine products, and some will combine.
Step 2: Take the derivative with respect to \(\hat{\beta}_0\) and set it equal to zero. When you take the derivative with respect to \(\hat{\beta}_0\), any term without \(\hat{\beta}_0\) in it drops out. A term of the form \(c\,\hat{\beta}_0\) becomes \(c\), and a term of the form \(c\,\hat{\beta}_0^2\) becomes \(2c\,\hat{\beta}_0\), where \(c\) is whatever multiplies it (a number, a data sum, or an expression involving \(\hat{\beta}_1\)). Then simplify. You should to get: \(\hat{\beta}_0 = \bar{y} - \hat{\beta_1} \bar{x}\).
Step 3: Take the derivative with respect to \(\hat{\beta}_1\) and set it equal to zero. The same rules apply with the roles swapped: terms without \(\hat{\beta}_1\) drop out, \(c\,\hat{\beta}_1\) becomes \(c\), and \(c\,\hat{\beta}_1^2\) becomes \(2c\,\hat{\beta}_1\), where now \(c\) may involve \(\hat{\beta}_0\). Then simplify to get: \(\hat{\beta}_1 = \frac{\sum_i x_i y_i}{\sum_i (x_i^2)} - \hat{\beta_0} \frac{\sum_i x_i}{\sum_i (x_i)^2}\).
Step 4: Notice that the formula for \(\hat{\beta}_1\) has \(\hat{\beta}_0\) in it. Substitute in for \(\hat{\beta}_0\) to get a formula for \(\hat{\beta}_1\) with only the data on the right hand side (\(x_i\), \(y_i\), and \(n\)). You should end up with:
\[\hat{\beta_1} = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i (x_i^2) - \bar{x} \sum_i x_i}\]
Step 5: Use the fact that \(\bar{x} = \frac{\sum_i x_i}{n}\) to simplify further. You should end up with \(\hat{\beta_1} = \frac{\sum_i x_i y_i - n \bar{x} \bar{y}}{\sum_i (x_i^2) - n \bar{x}^2}\)
Q22: Truth and Estimate
Apply the formulas from Q21 to the three observations in Q20. Compute \(\hat{\beta}_1\), then \(\hat{\beta}_0\), then the fitted values and residuals. Confirm that the residuals sum to zero.
| \(x\) | \(y\) | \(\hat{y}_i\) | \(e_i\) |
|---|---|---|---|
| 0 | 1 | ||
| 2 | 5 | ||
| 4 | 6 |
Extension: Now suppose that you are told the true relationship: \(y_i = 1 + 1.5 x_i + u_i\), so \(\beta_0 = 1\) and \(\beta_1 = 1.5\). Compute the unobservable term \(u_i\) for each observation and place it in a table next to \(e_i\). Is \(e_i\) always a good approximation for \(u_i\)?
Q20: The Judge
The convention. Use \(e_i = y_i - \hat{y}_i\). A positive residual then means the point sits above the line: the actual value is larger than the fitted one, so the line underpredicts. Under the other convention the same point has a negative residual and “underpredicts” would mean \(e_i < 0\). Either rule works, but the sign of every residual flips, so the phrase is meaningless until the convention is fixed. The rest of these answers use \(y_i - \hat{y}_i\).
The tables.
| \(x\) | \(y\) | Ana \(\hat{y}\) | Ana \(e\) | Ben \(\hat{y}\) | Ben \(e\) | Cal \(\hat{y}\) | Cal \(e\) |
|---|---|---|---|---|---|---|---|
| 0 | 1 | 1 | 0 | 4 | \(-3\) | 1.5 | \(-0.5\) |
| 2 | 5 | 3.5 | 1.5 | 5 | 0 | 4 | 1 |
| 4 | 6 | 6 | 0 | 6 | 0 | 6.5 | \(-0.5\) |
The scores.
| \(\sum e_i\) | \(\sum \lvert e_i \rvert\) | \(\sum e_i^2\) | |
|---|---|---|---|
| Ana | 1.5 | 1.5 | 2.25 |
| Ben | \(-3\) | 3 | 9 |
| Cal | 0 | 2 | 1.5 |
Sum of residuals: Cal, if “best” means closest to zero; Ben, if “best” means smallest. Sum of absolute values: Ana. Sum of squares: Cal. Three scores, three different winners, which is why the choice of score has to be made before the judging starts.
Extension. The sum of residuals is unusable under either reading. If smallest wins, the line \(\hat{y} = 1000\) has residuals of about \(-999\) each and beats everyone, and \(\hat{y} = 10^6\) beats that: there is no best line. If closest to zero wins, every line through the point of means \((\bar{x}, \bar{y}) = (2, 4)\) scores exactly 0, including the flat line \(\hat{y} = 4\) and the absurd \(\hat{y} = 4 + 100(x - 2)\). Positive and negative residuals cancel, so the score cannot tell a good line from a bad one.
The sum of absolute values does not fail in the same way: it rewards closeness, and no line beats Ana’s 1.5 on these three points. (With three points, the best line under this score always passes through two of them; the three candidates score 1.5, 3, and 3, and Ana’s is the winner.) Its defects are of a different kind. It has no slope at zero, so the calculus of Q21 cannot be used to find the minimum; and with other data sets it often has a whole range of lines tied for first place rather than a unique answer. Squares keep the rewarding-closeness property, cannot cancel, and give a smooth bowl with a single bottom.
Q21: The Bottom of the Bowl
Step 1. For one observation, \((y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2\) has nine products that collapse to six terms:
\[y_i^2 + \hat{\beta}_0^2 + \hat{\beta}_1^2 x_i^2 - 2\hat{\beta}_0 y_i - 2\hat{\beta}_1 x_i y_i + 2\hat{\beta}_0 \hat{\beta}_1 x_i .\]
Summing over \(i\) and pulling constants out of each sum (the \(\hat{\beta}\)’s do not depend on \(i\)):
\[S = \sum y_i^2 + n\hat{\beta}_0^2 + \hat{\beta}_1^2 \sum x_i^2 - 2\hat{\beta}_0 \sum y_i - 2\hat{\beta}_1 \sum x_i y_i + 2\hat{\beta}_0 \hat{\beta}_1 \sum x_i .\]
Step 2. Only three terms contain \(\hat{\beta}_0\). Differentiating:
\[\frac{\partial S}{\partial \hat{\beta}_0} = 2n\hat{\beta}_0 - 2\sum y_i + 2\hat{\beta}_1 \sum x_i = 0 .\]
Divide by \(2n\) and use \(\bar{y} = \sum y_i / n\) and \(\bar{x} = \sum x_i / n\): \(\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}\). The fitted line passes through the point of means.
Step 3. Only three terms contain \(\hat{\beta}_1\):
\[\frac{\partial S}{\partial \hat{\beta}_1} = 2\hat{\beta}_1 \sum x_i^2 - 2\sum x_i y_i + 2\hat{\beta}_0 \sum x_i = 0 ,\]
so \(\hat{\beta}_1 \sum x_i^2 = \sum x_i y_i - \hat{\beta}_0 \sum x_i\) and
\[\hat{\beta}_1 = \frac{\sum x_i y_i}{\sum x_i^2} - \hat{\beta}_0 \frac{\sum x_i}{\sum x_i^2} .\]
Step 4. Substitute \(\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}\) into the Step 3 line before dividing:
\[\hat{\beta}_1 \sum x_i^2 = \sum x_i y_i - \bar{y}\sum x_i + \hat{\beta}_1 \bar{x} \sum x_i .\]
Collect the \(\hat{\beta}_1\) terms on the left, \(\hat{\beta}_1 \left(\sum x_i^2 - \bar{x}\sum x_i\right) = \sum x_i y_i - \bar{y}\sum x_i\), and divide:
\[\hat{\beta}_1 = \frac{\sum x_i y_i - \bar{y}\sum x_i}{\sum x_i^2 - \bar{x}\sum x_i} .\]
Step 5. Replace \(\sum x_i\) with \(n\bar{x}\) in both numerator and denominator:
\[\hat{\beta}_1 = \frac{\sum x_i y_i - n\bar{x}\bar{y}}{\sum x_i^2 - n\bar{x}^2} .\]
Both formulas use only \(x_i\), \(y_i\), and \(n\). Dividing top and bottom by \(n\) shows the slope is the covariance of \(x\) and \(y\) divided by the variance of \(x\).
Q22: Truth and Estimate
The sums. \(n = 3\), \(\sum x_i = 6\), \(\bar{x} = 2\), \(\sum y_i = 12\), \(\bar{y} = 4\), \(\sum x_i y_i = 0 + 10 + 24 = 34\), \(\sum x_i^2 = 0 + 4 + 16 = 20\).
The slope. \(\hat{\beta}_1 = \dfrac{34 - 3 \cdot 2 \cdot 4}{20 - 3 \cdot 2^2} = \dfrac{10}{8} = 1.25\).
The intercept. \(\hat{\beta}_0 = 4 - 1.25 \cdot 2 = 1.5\). The least squares line is \(\hat{y} = 1.5 + 1.25x\): Cal’s line, the one that won the sum-of-squares contest in Q20.
The table.
| \(x\) | \(y\) | \(\hat{y}_i\) | \(e_i\) |
|---|---|---|---|
| 0 | 1 | 1.5 | \(-0.5\) |
| 2 | 5 | 4 | 1 |
| 4 | 6 | 6.5 | \(-0.5\) |
\(\sum e_i = -0.5 + 1 - 0.5 = 0\). This is not luck: the Step 2 condition \(\sum y_i - n\hat{\beta}_0 - \hat{\beta}_1 \sum x_i = 0\) is exactly the statement that the residuals sum to zero, so every least squares line balances its points.
Extension. With \(\beta_0 = 1\) and \(\beta_1 = 1.5\), \(u_i = y_i - 1 - 1.5x_i\):
| \(x\) | \(y\) | \(u_i\) | \(e_i\) |
|---|---|---|---|
| 0 | 1 | 0 | \(-0.5\) |
| 2 | 5 | 1 | 1 |
| 4 | 6 | \(-1\) | \(-0.5\) |
The residual matches the error for the middle point and misses by \(0.5\) at each end. \(e_i\) is computed from the estimated line and \(u_i\) from the true one; the two lines differ (slope 1.25 versus 1.5, intercept 1.5 versus 1), so the gaps between a point and each line differ too. Residuals are estimates of errors, and with only three observations they are rough ones. Their sum is forced to zero by the fitting procedure, whereas the errors sum to zero here only by coincidence.