Understanding least squares as a projection (Rewritten from an earlier blog post)

Environment

Python 3.14.6, NumPy 2.5.4, Matplotlib 3.11.2 (Jupyter Notebook).

This article is a rewrite of a post from around 2013–2016, with calculations performed in the environment described above.

Overview

Least squares projects the observation vector orthogonally onto the column space of the explanatory variables. We derive the normal equations from projection onto a line and check a regression through three data points by hand and in Python. The aim is to use diagrams to see what the residual is orthogonal to.

Setup

Run the code in Jupyter Notebook from top to bottom. Round numbers to three significant digits only when displaying them.

import base64
import platform
from pathlib import Path
from pprint import pprint

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from IPython.display import Latex, display
from matplotlib.ticker import FuncFormatter, MaxNLocator

print(f'Python: {platform.python_version()}')
print(f'NumPy: {np.__version__}, Matplotlib: {matplotlib.__version__}')

# Specify significant digits, not digits after the decimal point.
def sig3(value):
    return format(float(value), '#.3g')


np.set_printoptions(formatter={'float_kind': sig3})
plt.rcParams.update({'font.size': 11, 'axes.grid': True,
                     'grid.alpha': 0.250, 'svg.fonttype': 'none'})
output_dir = Path('102_nb_files')
output_dir.mkdir(exist_ok=True)


def save_svg(fig, filename):
    # Use three significant digits for axis ticks too. Save and display the same figure.
    for ax in fig.axes:
        for axis in (ax.xaxis, ax.yaxis):
            axis.set_major_formatter(FuncFormatter(lambda v, _: sig3(v)))
        if hasattr(ax, 'zaxis'):
            ax.zaxis.set_major_formatter(FuncFormatter(lambda v, _: sig3(v)))
    path = output_dir / filename
    fig.savefig(path, format='svg', bbox_inches='tight')
    plt.close(fig)
    # Fit the HTML display to the output width and also retain the SVG in the notebook.
    svg = path.read_text(encoding='utf-8')
    encoded = base64.b64encode(svg.encode('utf-8')).decode('ascii')
    display({'image/svg+xml': svg, 'text/html': (
        f'<img src="data:image/svg+xml;base64,{encoded}" '
        f'alt="{filename}" style="max-width:100%;height:auto;">'
    )}, raw=True)
Python: 3.14.6
NumPy: 2.5.4, Matplotlib: 3.11.2

1. Projection onto a line

A point on the line spanned by a nonzero vector $\mathbf{a}$ can be written as $t\mathbf{a}$. Find the point closest to the observation $\mathbf{y}$ by minimizing the squared distance.

$$ \begin{aligned} f(t)&=\|\mathbf{y}-t\mathbf{a}\|^2\\ &=(\mathbf{y}-t\mathbf{a})^T(\mathbf{y}-t\mathbf{a})\\ &=\mathbf{y}^T\mathbf{y}-2t\mathbf{a}^T\mathbf{y}+t^2\mathbf{a}^T\mathbf{a},\\ f^{\prime}(t)&=-2\mathbf{a}^T\mathbf{y}+2t\mathbf{a}^T\mathbf{a}=0,\\ \hat t&=\frac{\mathbf{a}^T\mathbf{y}}{\mathbf{a}^T\mathbf{a}}. \end{aligned} $$

Since $f^{\prime\prime}(t)=2\mathbf{a}^T\mathbf{a}>0$, this point is a minimum. Defining the projection as $\hat{\mathbf{y}}=\hat t\mathbf{a}$ and the residual as $\mathbf{r}=\mathbf{y}-\hat{\mathbf{y}}$, we obtain

$$ \mathbf{a}^T\mathbf{r} =\mathbf{a}^T\mathbf{y} -\frac{\mathbf{a}^T\mathbf{y}}{\mathbf{a}^T\mathbf{a}}\mathbf{a}^T\mathbf{a}=0. $$

Thus, the residual is orthogonal to the line. For $\mathbf{a}=(1,1)^T$ and $\mathbf{y}=(2,0)^T$, we have $\hat t=2/2=1$, $\hat{\mathbf{y}}=(1,1)^T$, and $\mathbf{r}=(1,-1)^T$. In this example, $f(t)=(2-t)^2+t^2=2(t-1)^2+2$.

a = np.array([1, 1])
y_line = np.array([2, 0])
t_hat = (a @ y_line) / (a @ a)
yhat_line = t_hat * a
r_line = y_line - yhat_line
pprint({'t': sig3(t_hat), 'projection': yhat_line, 'residual': r_line,
        'inner product': sig3(a @ r_line)})
display(Latex(r'\(\boxed{\hat{\mathbf{y}}='
              r'\mathbf{a}\frac{\mathbf{a}^T\mathbf{y}}'
              r'{\mathbf{a}^T\mathbf{a}}},\quad'
              r'\mathbf{a}^T\mathbf{r}=0\)'))

fig, ax = plt.subplots(figsize=(6, 5), layout='constrained')
line = np.linspace(-0.400, 2.40, 100)
ax.plot(line, line, color='0.550', label=r'$\mathrm{span}(\mathbf{a})$')
for vector, color, label in [
    (y_line, '#176b96', r'$\mathbf{y}$'),
    (yhat_line, '#387f51', r'$\hat{\mathbf{y}}$'),
]:
    ax.quiver(0, 0, *vector, angles='xy', scale_units='xy', scale=1,
              color=color, label=label)
ax.quiver(*yhat_line, *r_line, angles='xy', scale_units='xy', scale=1,
          color='#bd503b', label=r'$\mathbf{r}$')
# Mark the right angle at the projected point.
u = a / np.linalg.norm(a) * 0.160
v = r_line / np.linalg.norm(r_line) * 0.160
corner = np.array([yhat_line + u, yhat_line + u + v, yhat_line + v])
ax.plot(corner[:, 0], corner[:, 1], color='0.250')
ax.set(xlim=(-0.400, 2.40), ylim=(-0.400, 2.40),
       xlabel='Coordinate 1', ylabel='Coordinate 2', title='Projection onto a line')
ax.set_aspect('equal')
ax.legend(loc='upper left')
save_svg(fig, '01_line_projection.svg')

trial_t = np.linspace(-0.500, 2.50, 200)
loss_line = np.sum((y_line - trial_t[:, None] * a)**2, axis=1)
fig, ax = plt.subplots(figsize=(6, 4), layout='constrained')
ax.plot(trial_t, loss_line, color='#176b96')
ax.scatter([t_hat], [r_line @ r_line], color='#bd503b', zorder=3)
ax.axvline(t_hat, color='0.500', linestyle='--')
ax.set(xlabel='$t$', ylabel='$f(t)$', title='Squared distance to the line')
save_svg(fig, '02_scalar_loss.svg')
{'t': '1.00',
 'inner product': '0.00',
 'projection': array([1.00, 1.00]),
 'residual': array([1.00, -1.00])}

$$ \boxed{\hat{\mathbf{y}}=\mathbf{a}\frac{\mathbf{a}^T\mathbf{y}}{\mathbf{a}^T\mathbf{a}}},\quad\mathbf{a}^T\mathbf{r}=0 $$

01_line_projection.svg 02_scalar_loss.svg

2. Column space and the normal equations

Let $\mathbf{X}\in\mathbb{R}^{n\times p}$, $\mathbf{b}\in\mathbb{R}^p$, and $\mathbf{y}\in\mathbb{R}^n$. The vector $\mathbf{X}\mathbf{b}$ is a linear combination of the columns, and the set of all such combinations is the column space. Here we assume that the columns are linearly independent.

$$ \begin{aligned} S(\mathbf{b})&=\|\mathbf{y}-\mathbf{X}\mathbf{b}\|^2\\ &=\mathbf{y}^T\mathbf{y}-\mathbf{y}^T\mathbf{X}\mathbf{b} -\mathbf{b}^T\mathbf{X}^T\mathbf{y} +\mathbf{b}^T\mathbf{X}^T\mathbf{X}\mathbf{b}\\ &=\mathbf{y}^T\mathbf{y}-2\mathbf{b}^T\mathbf{X}^T\mathbf{y} +\mathbf{b}^T\mathbf{X}^T\mathbf{X}\mathbf{b},\\ \nabla S&=-2\mathbf{X}^T\mathbf{y}+2\mathbf{X}^T\mathbf{X}\mathbf{b}. \end{aligned} $$

The two middle terms are the same scalar, so we combine them. Setting the derivative to zero gives the normal equations.

$$ \mathbf{X}^T\mathbf{X}\hat{\mathbf{b}}=\mathbf{X}^T\mathbf{y} \quad\Longleftrightarrow\quad \mathbf{X}^T(\mathbf{y}-\mathbf{X}\hat{\mathbf{b}})=\mathbf{0}. $$

The residual is orthogonal to each column, and therefore to every vector in the column space. The normal equations state that the residual, pointing from the fitted values to the observations, is perpendicular to the column space.

We also check that this is a minimum. Write any coefficient vector as $\hat{\mathbf{b}}+\mathbf{d}$. Then

$$ \begin{aligned} S(\hat{\mathbf{b}}+\mathbf{d}) &=\|\mathbf{r}-\mathbf{X}\mathbf{d}\|^2\\ &=\|\mathbf{r}\|^2-2\mathbf{d}^T\mathbf{X}^T\mathbf{r}+\|\mathbf{X}\mathbf{d}\|^2\\ &=S(\hat{\mathbf{b}})+\|\mathbf{X}\mathbf{d}\|^2\ge S(\hat{\mathbf{b}}). \end{aligned} $$

If the columns are linearly independent, equality holds only when $\mathbf{d}=\mathbf{0}$, so the coefficients are unique. No probabilistic assumptions, such as normally distributed errors, have been needed.

3. The projection matrix

Setting $\mathbf{G}=\mathbf{X}^T\mathbf{X}$, the normal equations give

$$ \hat{\mathbf{b}}=\mathbf{G}^{-1}\mathbf{X}^T\mathbf{y},\qquad \hat{\mathbf{y}}=\mathbf{P}\mathbf{y},\qquad \mathbf{P}=\mathbf{X}\mathbf{G}^{-1}\mathbf{X}^T. $$

Since $\mathbf{G}^T=\mathbf{G}$, its inverse is also symmetric. Therefore,

$$ \begin{aligned} \mathbf{P}^T&=\mathbf{X}(\mathbf{G}^{-1})^T\mathbf{X}^T=\mathbf{P},\\ \mathbf{P}^2&=\mathbf{X}\mathbf{G}^{-1}(\mathbf{X}^T\mathbf{X})\mathbf{G}^{-1}\mathbf{X}^T\\ &=\mathbf{X}\mathbf{G}^{-1}\mathbf{G}\mathbf{G}^{-1}\mathbf{X}^T=\mathbf{P}. \end{aligned} $$

The identity $\mathbf{P}^2=\mathbf{P}$ means that projecting an already projected point leaves it unchanged. Also, $\mathbf{y}=\hat{\mathbf{y}}+\mathbf{r}$ and $\hat{\mathbf{y}}^T\mathbf{r}=0$ give

$$ \|\mathbf{y}\|^2=\|\hat{\mathbf{y}}\|^2+2\hat{\mathbf{y}}^T\mathbf{r}+\|\mathbf{r}\|^2 =\|\hat{\mathbf{y}}\|^2+\|\mathbf{r}\|^2. $$

# Display the main equation as LaTeX output.
display(Latex(r'\(\boxed{\hat{\mathbf{y}}=\mathbf{P}\mathbf{y}},\quad'
              r'\boxed{\mathbf{X}^T\mathbf{r}=\mathbf{0}},\quad'
              r'\mathbf{P}^T=\mathbf{P},\quad\mathbf{P}^2=\mathbf{P}\)'))

$$ \boxed{\hat{\mathbf{y}}=\mathbf{P}\mathbf{y}},\quad\boxed{\mathbf{X}^T\mathbf{r}=\mathbf{0}},\quad\mathbf{P}^T=\mathbf{P},\quad\mathbf{P}^2=\mathbf{P} $$

4. Regression with three data points

Fit $y_i\approx b_0+b_1x_i$ to $(x_i,y_i)=(0,1),(1,2),(2,2)$.

$$ \mathbf{X}=\begin{bmatrix}1&0\\1&1\\1&2\end{bmatrix},\quad \mathbf{y}=\begin{bmatrix}1\\2\\2\end{bmatrix},\quad \mathbf{X}^T\mathbf{X}=\begin{bmatrix}3&3\\3&5\end{bmatrix},\quad \mathbf{X}^T\mathbf{y}=\begin{bmatrix}5\\6\end{bmatrix}. $$

The normal equations are $3b_0+3b_1=5$ and $3b_0+5b_1=6$. Subtracting the first from the second gives $2b_1=1$, so $b_1=1/2$ and $b_0=(5-3/2)/3=7/6$.

$$ \hat{\mathbf{y}}=\begin{bmatrix}7/6\\5/3\\13/6\end{bmatrix},\quad \mathbf{r}=\begin{bmatrix}-1/6\\1/3\\-1/6\end{bmatrix},\quad \mathbf{X}^T\mathbf{r}=\begin{bmatrix}-1/6+1/3-1/6\\1/3-2/6\end{bmatrix}=\mathbf{0}. $$

The residual sum of squares is $1/36+1/9+1/36=1/6$. We use lstsq for the numerical calculation. Because this is a small example, we form the projection matrix explicitly, but solve a linear system instead of computing an inverse.

x = np.arange(3)
X = np.column_stack([np.ones(3), x])
y = np.array([1, 2, 2])
b_hat = np.linalg.lstsq(X, y, rcond=None)[0]
y_hat = X @ b_hat
residual = y - y_hat
rss = residual @ residual
P = X @ np.linalg.solve(X.T @ X, X.T)

pprint({'coefficients [intercept, slope]': b_hat, 'fitted values': y_hat,
        'residual': residual, 'residual sum of squares': sig3(rss), 'projection matrix': P})
display(Latex(
    rf'\(\hat{{\mathbf{{b}}}}=\begin{{bmatrix}}{sig3(b_hat[0])}\\'
    rf'{sig3(b_hat[1])}\end{{bmatrix}},\quad S={sig3(rss)}\)'
))

# Check agreement with the hand calculation and the projection properties before rounding.
np.testing.assert_allclose(b_hat, [7 / 6, 1 / 2], atol=1.00e-12)
np.testing.assert_allclose(X.T @ residual, 0, atol=1.00e-12)
np.testing.assert_allclose(P.T, P, atol=1.00e-12)
np.testing.assert_allclose(P @ P, P, atol=1.00e-12)
np.testing.assert_allclose(P @ y, y_hat, atol=1.00e-12)
np.testing.assert_allclose(rss, 1 / 6, atol=1.00e-12)
np.testing.assert_allclose(y @ y, y_hat @ y_hat + rss, atol=1.00e-12)
pprint({'orthogonality error': sig3(np.linalg.norm(X.T @ residual)),
        'reprojection error': sig3(np.linalg.norm(P @ y_hat - y_hat))})
print('Checked the hand calculation, orthogonality, symmetry, idempotence, and squared-norm decomposition.')
{'fitted values': array([1.17, 1.67, 2.17]),
 'coefficients [intercept, slope]': array([1.17, 0.500]),
 'projection matrix': array([[0.833, 0.333, -0.167],
       [0.333, 0.333, 0.333],
       [-0.167, 0.333, 0.833]]),
 'residual': array([-0.167, 0.333, -0.167]),
 'residual sum of squares': '0.167'}

$$ \hat{\mathbf{b}}=\begin{bmatrix}1.17\\0.500\end{bmatrix},\quad S=0.167 $$

{'reprojection error': '0.00', 'orthogonality error': '2.67e-15'}
Checked the hand calculation, orthogonality, symmetry, idempotence, and squared-norm decomposition.

The regression plot and the column-space plot

In the regression plot, residuals are vertical differences. In the three-dimensional space of observations, the column space is a plane through the origin, and the residual is perpendicular to that plane. The regression plot measures vertical differences; the column-space plot measures the shortest distance to the plane.

fig, axes = plt.subplots(1, 2, figsize=(10, 4), layout='constrained')
x_line = np.linspace(-0.200, 2.20, 100)
axes[0].scatter(x, y, color='#176b96', label='Observed', zorder=3)
axes[0].plot(x_line, b_hat[0] + b_hat[1] * x_line,
             color='#387f51', label='Fitted line')
axes[0].vlines(x, y_hat, y, color='#bd503b', linestyle='--', label='Residual')
axes[0].set(xlabel='$x$', ylabel='$y$', title='Regression in the data plot')
axes[0].legend()
axes[1].bar(x, residual, color='#bd503b', width=0.500)
axes[1].axhline(0, color='0.300')
axes[1].set(xlabel='$x$', ylabel='$r_i$', title='Residuals', xticks=x)
save_svg(fig, '03_regression_residuals.svg')

# Construct the plane from linear combinations of the two columns of X.
c0, c1 = np.meshgrid(np.linspace(-0.200, 1.60, 9),
                     np.linspace(-0.100, 0.700, 9))
plane = c0[..., None] * X[:, 0] + c1[..., None] * X[:, 1]
fig = plt.figure(figsize=(7, 6), layout='constrained')
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(plane[:, :, 0], plane[:, :, 1], plane[:, :, 2],
                alpha=0.200, color='#72a0bb', edgecolor='none')
for vector, color, label in [
    (y, '#176b96', r'$\mathbf{y}$'),
    (y_hat, '#387f51', r'$\hat{\mathbf{y}}$'),
]:
    ax.quiver(0, 0, 0, *vector, color=color, arrow_length_ratio=0.0800)
    ax.plot([0, vector[0]], [0, vector[1]], [0, vector[2]], color=color, label=label)
ax.quiver(*y_hat, *residual, color='#bd503b', arrow_length_ratio=0.200)
ax.plot([y_hat[0], y[0]], [y_hat[1], y[1]], [y_hat[2], y[2]],
        color='#bd503b', linewidth=3, label=r'$\mathbf{r}$')
ax.set(xlabel='Coordinate 1', ylabel='Coordinate 2', zlabel='Coordinate 3',
       title='Projection onto the column space')
# Use the same scale for all axes and choose a view that does not hide the residual.
ax.set(xlim=(-0.500, 3), ylim=(-0.500, 3), zlim=(-0.500, 3))
for axis in (ax.xaxis, ax.yaxis, ax.zaxis):
    axis.set_major_locator(MaxNLocator(nbins=4))
ax.set_box_aspect((1, 1, 1))
ax.view_init(elev=20, azim=-135)
ax.legend(loc='upper left')
save_svg(fig, '04_projection_plane.svg')
03_regression_residuals.svg 04_projection_plane.svg

The increase when the coefficients change

In this example, $\|\mathbf{X}\mathbf{d}\|^2=3d_0^2+6d_0d_1+5d_1^2$. The left plot shows contours of the residual sum of squares. The right plot shows a cross section with the intercept fixed at its estimated value.

b0, b1 = np.meshgrid(np.linspace(b_hat[0] - 1, b_hat[0] + 1, 121),
                     np.linspace(b_hat[1] - 1, b_hat[1] + 1, 121))
loss = np.sum((y - b0[..., None] - b1[..., None] * x)**2, axis=-1)
fig, axes = plt.subplots(1, 2, figsize=(10, 4), layout='constrained')
contours = axes[0].contour(b0, b1, loss, levels=[0.200, 0.500, 1, 2, 4, 8])
axes[0].clabel(contours, fmt=sig3)
axes[0].scatter(*b_hat, color='#bd503b', label='Least squares')
axes[0].set(xlabel='$b_0$', ylabel='$b_1$', title='Residual sum of squares')
axes[0].legend()
trial_b1 = np.linspace(b_hat[1] - 1, b_hat[1] + 1, 200)
section = np.sum((y - b_hat[0] - trial_b1[:, None] * x)**2, axis=1)
axes[1].plot(trial_b1, section, label='Direct calculation')
axes[1].plot(trial_b1, rss + 5 * (trial_b1 - b_hat[1])**2,
             linestyle='--', label='Orthogonal decomposition')
axes[1].scatter([b_hat[1]], [rss], color='#bd503b')
axes[1].set(xlabel='$b_1$', ylabel='$S$', title='Fixed intercept')
axes[1].legend()
for ax in axes:
    ax.xaxis.set_major_locator(MaxNLocator(nbins=5))
save_svg(fig, '05_loss_contours.svg')
05_loss_contours.svg

5. QR factorization and repeated columns

If the columns are linearly independent, the thin QR factorization $\mathbf{X}=\mathbf{Q}\mathbf{R}$ satisfies $\mathbf{Q}^T\mathbf{Q}=\mathbf{I}$. Substituting into the normal equations gives

$$ \mathbf{R}^T\mathbf{R}\hat{\mathbf{b}}=\mathbf{R}^T\mathbf{Q}^T\mathbf{y} \ \Longrightarrow\ \mathbf{R}\hat{\mathbf{b}}=\mathbf{Q}^T\mathbf{y} \ \Longrightarrow\ \hat{\mathbf{y}}=\mathbf{Q}\mathbf{Q}^T\mathbf{y}. $$

The normal equations square the condition number in the 2-norm, so lstsq or QR factorization is usually preferred. Repeated columns make the coefficients nonunique, but the projected fitted values remain unique. Below, we add another copy of the $x$ column to $\mathbf{X}$. The function lstsq returns the coefficient vector with the smallest norm.

Q, R = np.linalg.qr(X, mode='reduced')
b_qr = np.linalg.solve(R, Q.T @ y)
P_qr = Q @ Q.T
np.testing.assert_allclose(b_qr, b_hat, atol=1.00e-12)
np.testing.assert_allclose(P_qr, P, atol=1.00e-12)

# Redistributing the coefficients between identical columns does not change the fitted values.
X_duplicate = np.column_stack([X, x])
b_min = np.linalg.lstsq(X_duplicate, y, rcond=None)[0]
b_other = b_min + np.array([0, 1, -1])
np.testing.assert_allclose(X_duplicate @ b_min, y_hat, atol=1.00e-12)
np.testing.assert_allclose(X_duplicate @ b_other, y_hat, atol=1.00e-12)
pprint({'QR coefficients': b_qr, 'rank with repeated columns':
        int(np.linalg.matrix_rank(X_duplicate)),
        'minimum-norm coefficients': b_min, 'other coefficients with the same fit': b_other})

# Compare fitted values and residuals for each observation to check the decomposition.
fig, ax = plt.subplots(figsize=(7, 4), layout='constrained')
width = 0.250
for shift, values, label, color in [
    (-width, y, 'Observed', '#176b96'),
    (0, y_hat, 'Projected', '#387f51'),
    (width, residual, 'Residual', '#bd503b'),
]:
    ax.bar(x + shift, values, width=width, label=label, color=color)
ax.axhline(0, color='0.300')
ax.set(xlabel='Observation index', ylabel='Value', xticks=x,
       title='Observed = projected + residual')
ax.legend()
save_svg(fig, '06_vector_decomposition.svg')
{'QR coefficients': array([1.17, 0.500]),
 'rank with repeated columns': 2,
 'other coefficients with the same fit': array([1.17, 1.25, -0.750]),
 'minimum-norm coefficients': array([1.17, 0.250, 0.250])}
06_vector_decomposition.svg

Conclusion

We derived least squares as the orthogonal projection of the observation vector onto the column space and checked it in Python.

The normal equations express the orthogonality of the residual and the column space. This orthogonality ensures that no other choice of coefficients gives a smaller residual sum of squares. The coefficients are unique when the columns are linearly independent, and the fitted values remain unique even when the columns are dependent.