Numerical methods with Python
Derive and test optimization and Runge–Kutta methods in Python, comparing updates, errors, and stability.
Prerequisites Differentiation, Taylor expansions, basic matrix operations, and basic Python.
- [Numerical computing] Why does Newton’s method converge quadratically? (Rewritten from an earlier blog post)
Derive why Newton’s method converges at least quadratically near a simple root using tangent lines and a Taylor expansion. Compute a square root in Python to check error reduction, error ratios, and convergence order. Compare slower convergence at a multiple root and failure to converge from some starting points, and use plots to show the accuracy limit caused by rounding errors.
- [Numerical computing] Examples of Newton’s method failing to converge in Python (Rewritten from an earlier blog post)
Reproduce three failures of Newton’s method in Python: iterates that alternate between zero and one, iterates whose absolute values double, and an update blocked by a zero derivative. Explain each with update formulas and plots. Compare different starting points and Brent’s method, checking residuals and stopping reasons as well as final values.
- [Numerical computing] What does BFGS approximate without computing the Hessian? (Rewritten from an earlier blog post)
Derive BFGS from gradient differences and the secant condition. Compare convergence, Hessian approximation error, and line searches in Python.