Class 9: More on deriving OLS estimators
Key Ideas
Class 8 derived formulas for \(\hat{\beta}_0\) and \(\hat{\beta}_1\) from the method of least squares. Today we derive nothing new. Instead we rewrite the slope formula, several times, into algebraically equivalent forms. Each form is the same number, but each makes a different property of that number visible: what the slope is built from, and which observations have the most say in it.
Problems of the Day
Q23: Three Faces of the Slope
Class 8 ended with
\[\hat{\beta}_1 = \frac{\sum_{i=1}^n x_i y_i - n\bar{x}\bar{y}}{\sum_{i=1}^n x_i^2 - n\bar{x}^2}.\]
That formula is fine for a calculator, but we can do more to learn about the OLS slope by manipulating it a little more.
Show that \(\hat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}\). Hint: work backwards by starting from \(\hat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}\) and get to \(\hat{\beta}_1 = \frac{\sum_{i=1}^n x_i y_i - n\bar{x}\bar{y}}{\sum_{i=1}^n x_i^2 - n\bar{x}^2}\). You can work on the numerator and denominator separately.
Extension: Divide the top and bottom of \(\frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}\) by \(n - 1\) and rewrite \(\hat{\beta}_1\) using the sample covariance \(\hat{\sigma}_{XY}\) and sample variance \(\hat{\sigma}^2_X\) from Class 5. What does this imply about the OLS slope \(\hat{\beta_1}\)? If \(X\) and \(Y\) are independent, what should \(\hat{\beta}_1\) be near, and why only “near”?
Extension #2: Show that the \(\bar{y}\) can vanish from the numerator, so that \(\hat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})\, y_i}{\sum_{i=1}^n (x_i - \bar{x})^2}\).
Q24: The Regrade
Q23 says \(\hat{\beta}_1 = \sum_{i=1}^n w_i y_i\) with
\[w_i = \frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2}\]
So the slope is a weighted sum of the \(y\)’s, with weights fixed by the \(x\)’s alone. Call \(w_i\) observation \(i\)’s leverage. An observation has low leverage over the slope if it’s near \(\bar{x}\); it has higher leverage over the slope the further it is from \(\bar{x}\).
Three students studied 0, 2, and 4 hours for a 20-point quiz and scored 0, 5, and 10 points.
| hours \(x_i\) | \(w_i\) | score \(y_i\) |
|---|---|---|
| 0 | 0 | |
| 2 | 5 | |
| 4 | 10 |
Fill in the weights \(w_i\) and compute the slope. Then check your answer by sketching a plot of the data points and eyeballing the line of best fit. What are the “high leverage” and “low leverage” observations in this data set?
Extension. The first student appeals their grade and gets an extra 10 points on the regrade. Re-calculate the slope: what happens to it? Do the same procedure for each other student, one at a time, and explain what happens to the slope from the perspective of their leverage \(w_i\). Then use what you’ve learned about leverage to make a statement about outliers: how will outliers with different leverages effect the line of best fit?
Q25: How Much Is Explained
Define the total variation in \(y\) as \(\sum (y_i - \bar{y})^2\) and the unexplained variation as \(\sum e_i^2\). The R-squared is
\[R^2 \equiv 1 - \frac{\sum_{i=1}^n e_i^2}{\sum_{i=1}^n (y_i - \bar{y})^2}.\]
Fill in the table and then compute \(R^2\) for this data set, given that the OLS parameters are \(\hat{\beta_0} = 0\) and \(\hat{\beta_1} = 2.5\):
| \(x_i\) | \(y_i\) | \(\hat{y} = \hat{\beta_0} + \hat{\beta_1} x_i\) | \(e_i\) |
|---|---|---|---|
| 0 | 0 | ||
| 2 | 5 | ||
| 4 | 10 |
This tells us that _____% of the variation here in Y can be explained by X.
Extension. Compute \(R^2\) again for this slightly different data set, given that the OLS parameters are \(\hat{\beta_0} = \frac{25}{3}\) and \(\hat{\beta_1} = 0\):
| \(x_i\) | \(y_i\) | \(\hat{y} = \hat{\beta_0} + \hat{\beta_1} x_i\) | \(e_i\) |
|---|---|---|---|
| 0 | 10 | ||
| 2 | 5 | ||
| 4 | 10 |
This tells us that _____% of the variation here in Y can be explained by X. What two numbers will the \(R^2\) always be between?
Q23: Three Faces of the Slope
The numerator. Multiply out \((x_i - \bar{x})(y_i - \bar{y}) = x_i y_i - \bar{x} y_i - \bar{y} x_i + \bar{x}\bar{y}\) and sum. The means are constants, so they come out of each sum, and \(\sum x_i = n\bar{x}\), \(\sum y_i = n\bar{y}\):
\[\sum (x_i - \bar{x})(y_i - \bar{y}) = \sum x_i y_i - \bar{x}\, n\bar{y} - \bar{y}\, n\bar{x} + n\bar{x}\bar{y} = \sum x_i y_i - n\bar{x}\bar{y} .\]
The denominator. The same move with \(y\) replaced by \(x\):
\[\sum (x_i - \bar{x})^2 = \sum x_i^2 - 2\bar{x}\sum x_i + n\bar{x}^2 = \sum x_i^2 - 2n\bar{x}^2 + n\bar{x}^2 = \sum x_i^2 - n\bar{x}^2 .\]
Top and bottom both match the Class 8 formula, so the two expressions for \(\hat{\beta}_1\) are the same number.
Extension. Dividing top and bottom by \(n - 1\) gives
\[\hat{\beta}_1 = \frac{\frac{1}{n-1}\sum (x_i - \bar{x})(y_i - \bar{y})}{\frac{1}{n-1}\sum (x_i - \bar{x})^2} = \frac{\hat{\sigma}_{XY}}{\hat{\sigma}^2_X} .\]
The slope is the sample covariance rescaled by the sample variance of \(X\). Since \(\hat{\sigma}^2_X > 0\), the slope has the sign of the covariance: \(X\) and \(Y\) move together, the slope is positive; they move apart, it is negative. If \(X\) and \(Y\) are independent the population covariance is zero, so \(\hat{\beta}_1\) should be near zero. Only “near” because \(\hat{\sigma}_{XY}\) is computed from a sample and is a random variable; in any finite sample it will land close to zero but almost never exactly on it.
Extension #2. Split the numerator: \(\sum (x_i - \bar{x})(y_i - \bar{y}) = \sum (x_i - \bar{x})\, y_i - \bar{y}\sum (x_i - \bar{x})\). The last sum is \(\sum x_i - n\bar{x} = 0\), so the \(\bar{y}\) term vanishes and
\[\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})\, y_i}{\sum (x_i - \bar{x})^2} = \sum_{i=1}^n w_i y_i, \qquad w_i = \frac{x_i - \bar{x}}{\sum_j (x_j - \bar{x})^2} .\]
The deviations of \(x\) from its mean sum to zero, so centering \(y\) changes nothing.
Q24: The Regrade
The weights. \(\bar{x} = 2\) and \(\sum (x_j - \bar{x})^2 = 4 + 0 + 4 = 8\).
| hours \(x_i\) | \(x_i - \bar{x}\) | \(w_i\) | score \(y_i\) | \(w_i y_i\) |
|---|---|---|---|---|
| 0 | \(-2\) | \(-0.25\) | 0 | 0 |
| 2 | 0 | 0 | 5 | 0 |
| 4 | 2 | 0.25 | 10 | 2.5 |
The slope. \(\hat{\beta}_1 = 0 + 0 + 2.5 = 2.5\) points per hour. The three points lie exactly on the line \(\hat{y} = 2.5x\), which is what a sketch shows: the eyeballed line passes through all three.
Leverage. The students at 0 and 4 hours have high leverage (\(|w_i| = 0.25\)); the student at 2 hours has none at all (\(w_i = 0\)), because they sit exactly at \(\bar{x}\).
Extension. Adding 10 points to student \(i\) changes the slope by exactly \(w_i \times 10\), since the weights depend only on the \(x\)’s and do not move.
| Regraded student | New scores | New slope | Change |
|---|---|---|---|
| 0 hours | 10, 5, 10 | \(-0.25(10) + 0 + 0.25(10) = 0\) | \(-2.5\) |
| 2 hours | 0, 15, 10 | \(0 + 0 + 2.5 = 2.5\) | 0 |
| 4 hours | 0, 5, 20 | \(0 + 0 + 0.25(20) = 5\) | \(+2.5\) |
A regrade at the left end flattens the line to zero, one at the right end doubles the slope, and one in the middle leaves the slope untouched (the middle student’s extra points shift the whole line up through the intercept, since the line must pass through the point of means, but do not tilt it). An outlier’s effect on the slope is its leverage times its size: a wild \(y\) at an extreme \(x\) can swing the fitted slope dramatically, while an equally wild \(y\) near \(\bar{x}\) barely tilts the line.
Q25: How Much Is Explained
The table. With \(\hat{y} = 2.5x\):
| \(x_i\) | \(y_i\) | \(\hat{y}_i\) | \(e_i\) |
|---|---|---|---|
| 0 | 0 | 0 | 0 |
| 2 | 5 | 5 | 0 |
| 4 | 10 | 10 | 0 |
R-squared. \(\bar{y} = 5\), so the total variation is \(25 + 0 + 25 = 50\) and the unexplained variation is \(0\). \(R^2 = 1 - 0/50 = 1\): 100% of the variation in \(Y\) is explained by \(X\). Every point is on the line, so there is nothing left over.
Extension. With \(\hat{y} = 25/3\) for every \(x\):
| \(x_i\) | \(y_i\) | \(\hat{y}_i\) | \(e_i\) |
|---|---|---|---|
| 0 | 10 | \(25/3\) | \(5/3\) |
| 2 | 5 | \(25/3\) | \(-10/3\) |
| 4 | 10 | \(25/3\) | \(5/3\) |
Here \(\bar{y} = 25/3\) as well, so the fitted value equals the mean and each residual equals \(y_i - \bar{y}\). Unexplained variation: \(25/9 + 100/9 + 25/9 = 150/9 = 50/3\). Total variation: the same \(50/3\). \(R^2 = 1 - 1 = 0\): 0% of the variation in \(Y\) is explained by \(X\). A flat line uses no information from \(x\), so it explains nothing.
\(R^2\) is always between 0 and 1. It cannot exceed 1 because \(\sum e_i^2 \geq 0\). It cannot fall below 0 because the flat line \(\hat{y} = \bar{y}\) is one of the candidates least squares considers, with \(\sum e_i^2 = \sum (y_i - \bar{y})^2\); the OLS line does at least as well, so its unexplained variation is never larger than the total. The two data sets here are the two ends of the scale: a perfect fit and a fit no better than the mean.