Polytechnique Montréal MTH1008 tutoring

Define the matrices, predictions and residual

Let $$X=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}$$ be the design matrix, $$\beta=\begin{pmatrix}a\\b\end{pmatrix}$$ the coefficient vector, and $$d=\begin{pmatrix}1\\1\\4\end{pmatrix}$$ the data vector.

Each row of $X$ contains a constant $1$ and that point’s input, so the predicted outputs are $X\beta$. The residual vector is $r=d-X\beta$.

Derive the normal equations from the squared error

Least squares minimizes the sum of squared residuals $$\begin{aligned}&S(a,b)\\&=(1-a)^2+(1-a-b)^2\\&\quad +(4-a-2b)^2\end{aligned}$$.

The partial derivatives are $$\begin{aligned}&\frac{\partial S}{\partial a}\\&=-2[(1-a)+(1-a-b)\\&\qquad +(4-a-2b)]\end{aligned}$$ and $$\begin{aligned}&\frac{\partial S}{\partial b}\\&=-2[(1-a-b)\\&\qquad +2(4-a-2b)]\end{aligned}$$.

Setting both to zero gives $3a+3b=6$ and $3a+5b=9$, called the normal equations.

Solve the equations and verify the unique minimum

Subtract the first equation from the second: $2b=3$, so $b=3/2$. Then $3a+9/2=6$, giving $a=1/2$.

The predictions are $(1/2,2,7/2)$, so $r=(1/2,-1,1/2)$.

The columns of $X$ are $(1,1,1)$ and $(0,1,2)$. Their dot products with $r$ are $1/2-1+1/2=0$ and $0-1+1=0$.

The residual vector is therefore perpendicular to the change in predicted values caused by any change to the line. By the Pythagorean theorem, the new sum of squared residuals equals the current sum plus the squared length of that change. It cannot be smaller. Because the columns are linearly independent, only one pair of coefficients gives these predictions, so the best-fit line is unique.

Explain the residual and the effect of shifting the data

The first two points would force an exact-fit line to have $a=1$ and $b=0$, which predicts $1$, not $4$, at the third point. Orthogonality therefore verifies the best approximation; it does not claim zero error.

Adding a constant to all outputs shifts the fitted intercept by that constant while leaving the slope and residual unchanged.