Derive the normal equations shared by least-squares methods and compare their roles in physics, statistics, and machine learning. Estimate a spring constant from artificial measurements, examine precision-weighted estimation and maximum likelihood with normal errors, then use Python polynomial regression to choose Ridge regularization on validation data and evaluate predictions on unseen data.
Explain least squares as the orthogonal projection of an observation vector onto a column space. Derive the normal equations from projection onto a line and check that the projection matrix is symmetric and idempotent. Compare a three-point regression by hand and in Python, plot residual orthogonality, and examine QR factorization and repeated columns with nonunique coefficients.
Derive ordinary least squares (OLS) from maximum likelihood under independent normal errors with equal variance. Obtain the normal equations from the log-likelihood and work through a four-point regression by hand. Use Python, plots, and repeated simulations to compare OLS with numerical optimization, relate residual sums of squares to likelihood, and distinguish maximum-likelihood and unbiased variance estimates.