← Tags

Exponential Family

Mathematics Machine learning

[Across fields] Why is the exponential family important in statistics and machine learning? (Rewritten from an earlier blog post)

Derive statistical and machine-learning calculations from the common exponential-family form. Use Bernoulli, Poisson, and normal distributions to connect log-partition derivatives, sufficient statistics, and maximum likelihood. Compute beta-prior updates and logistic regression in Python, and distinguish convexity of the loss from the existence of a finite estimate.

Mathematics Machine learning

[Generalized linear models] Deriving GLMs from the exponential family (Rewritten from an earlier blog post)

Derive generalized linear models (GLMs) from the exponential family. Differentiate the normalization condition to obtain means and variances, and compare normal, Bernoulli, and Poisson distributions. Derive iteratively reweighted least squares (IRLS) from maximum likelihood with a canonical link, implement Poisson regression on artificial data in Python, and check the result against SciPy optimization.

Mathematics Machine learning

[Generalized linear models] What is a canonical link? (Rewritten from an earlier blog post)

A canonical link makes the linear predictor of a generalized linear model equal to its natural parameter. Derive the identity, logit, and log links from normal, Bernoulli, and Poisson distributions, and plot their relations to the mean. Estimate regression coefficients by maximum likelihood in Python, explain why the gradient and Hessian simplify, and compare with a probit link.