← Tags

Normal Equations

Mathematics Numerical computing

[Linear algebra] Understanding least squares as a projection (Rewritten from an earlier blog post)

Explain least squares as the orthogonal projection of an observation vector onto a column space. Derive the normal equations from projection onto a line and check that the projection matrix is symmetric and idempotent. Compare a three-point regression by hand and in Python, plot residual orthogonality, and examine QR factorization and repeated columns with nonunique coefficients.