[Linear algebra] Deriving OLS from maximum likelihood (Rewritten from an earlier blog post)
Derive ordinary least squares (OLS) from maximum likelihood under independent normal errors with equal variance. Obtain the normal equations from the log-likelihood and work through a four-point regression by hand. Use Python, plots, and repeated simulations to compare OLS with numerical optimization, relate residual sums of squares to likelihood, and distinguish maximum-likelihood and unbiased variance estimates.
