Understanding Maximum Likelihood Estimation

An Interactive Visualization

Created by Kristoffer Magnusson · Version 1.0.0

The maximum likelihood method is used to fit many models in statistics. In this visualization I want to give some intuition for how it works, and how it relates to three common hypothesis tests: the likelihood ratio, Wald, and score tests.

We'll use a normal model with just two unknown parameters, the mean and variance, so we can draw the whole log-likelihood as a surface. When we get to hypothesis testing, we'll focus on the mean. The variance will be a nuisance parameter, meaning we still need to estimate it even though it isn't what we want to test.

Likelihood Calculation

Before we can estimate anything, we need some data. Let's take 10 independent observations from a normal distribution and pretend we don't know its mean or variance. We'll fit the model $y \sim \mathcal N(\mu, \sigma^2)$ . Try moving the sliders to find a combination of $\mu$ and $\sigma^2$ that seems to fit the observations. The table scores your choice: it adds up how well the curve fits each observation, and the higher (less negative) the total, the better the fit.

Mean (

\mu

)

Variance (

\sigma^2

)

$\ell(\mu, \sigma^2) = \sum_{i=1}^{n} \ln f(y_i \mid \mu, \sigma^2)$

$i$	$y_i$	$\ln f(y_i)$	longer = worse fit
1	77.62	−3.25
2	95.12	−4.36
3	99.92	−5.21
4	100.22	−5.27
5	102.42	−5.73
6	107.42	−6.98
7	108.72	−7.35
8	110.42	−7.85
9	113.52	−8.84
10	124.62	−13.18
$\ell(\mu, \sigma^2)$		−68.01	the current candidate
$\ell(\hat\mu, \hat\sigma^2)$		−38.89	the maximum, at $\hat\mu$ = 104.00, $\hat\sigma^2$ = 139.92

We can calculate the likelihood for any combination you choose. Each observation has a density given by the height of the normal curve at that point, and because the observations are independent we can multiply these densities together. Notice that the data don't change when you move the sliders. We're comparing different parameter values for the same observations, not assigning probabilities to the parameters themselves.

In practice, multiplying lots of small densities can give us very small numbers. It's usually more convenient to work with the log-likelihood, where we add the log-densities instead. We'll write it as $\ell(\mu, \sigma^2)$ : for a given mean and variance, the sum of the log-densities $\ln f(y_i \mid \mu, \sigma^2)$ over the observations. That's the calculation in the table. The logarithm is increasing, so whichever parameter values give the largest likelihood also give the largest log-likelihood.

Those parameter values are our maximum likelihood estimates (MLEs). The last row of the table gives the answer, so you can compare it with the values you've chosen.

Finding the Maximum Likelihood Estimates

If we repeat the calculation for a range of means and variances, we get the hill below. The height is the log-likelihood, and the combination we're looking for is at the top.

For our simple normal model, we can find the MLEs using calculus. We take the partial derivatives with respect to the mean and variance, $\partial\ell/\partial\mu$ and $\partial\ell/\partial\sigma^2$ , which tell us the slope in each direction. Together these make up the gradient, also called the score. At the maximum both slopes are zero.

We can also let the computer search for the maximum. This is more work than we need for this example, but for many models we can't solve the score equations directly. The algorithms below start from your chosen values and try to climb the hill. Take one step with the +1 button and check the readout: $\ell$ should have gone up. Gradient ascent follows the slope uphill, and you choose how far it steps. Try a few step sizes: too small and it takes a long time to get anywhere; too large and it can jump past the peak. Fisher scoring uses the expected curvature as well, which helps it choose how far to step.

Step size γ 0.05

Iterations: 0

View

μ σ² ℓ drag to rotate · drag the ring to move

Try looking at the hill from one of the side views. Its outline matches the curve drawn on the back wall: for each value of one parameter, it keeps the highest log-likelihood we can get by varying the other. This is the profile log-likelihood. For the mean we write it $\ell_p(\mu)$ , and we'll use it below to test hypotheses about the mean.

Inference

Once we've estimated the mean, we usually want to say something about its uncertainty too. Let's test the hypothesis that the population mean equals some value $\mu_0$ . There are several ways to do this; here we'll compare the likelihood ratio, Wald, and score tests.

We still have to deal with the variance. Our null hypothesis specifies the mean, but it doesn't tell us what the variance should be. For the likelihood ratio and score tests, we therefore estimate it again, this time with the mean fixed at $\mu_0$ . We call this estimate $\hat\sigma_0^2$ . Repeat it for every possible mean and we get the profile log-likelihood $\ell_p(\mu)$ from the previous section. You may see this described as profiling out the variance. In the figure, the point labeled best under $H_0$ is the fit under the null. Move $\mu_0$ and it slides along the dotted ridge, the best variance for each mean, whose shadow on the back wall is $\ell_p(\mu)$ .

Away from the estimated mean, allowing the variance to change gives a better fit than keeping it fixed at $\hat\sigma^2$ . The Wald test uses a quadratic approximation to the profile, which agrees near the maximum but falls faster farther away.

I've also included a sample-size slider so you can see how much the tests depend on $n$ . This doesn't generate any new observations. It shows what would happen at a different sample size if we got exactly the same estimates of the mean and variance.

Null hypothesis (μ₀) 96

$H_0: \mu = 96$ versus $H_1: \mu \ne 96$

What if the sample size were… 10

View

μ σ² ℓ drag to rotate

95% intervals at n = 10: profile likelihood [95.90, 112.10]; Wald [96.67, 111.33]

Each test tab picks out the part of the figure that test uses. Move $\mu_0$ and watch the three p-values in the tabs.

The likelihood ratio test compares two fits of the model. In one, the mean and the variance are both free, which puts us at the top of the hill, the point labeled MLE. The other is the fit under the null, best under $H_0$ , with the mean fixed at $\mu_0$ and the variance at $\hat\sigma_0^2$ . Both points lie on the profile log-likelihood.

The null fit can never be higher than the MLE, because the MLE is the best fit over all means, $\mu_0$ included. What matters is how much lower it is. The bracket labeled ½·LR is that gap, $\ell(\hat\mu, \hat\sigma^2) - \ell(\mu_0, \hat\sigma_0^2)$ . If the data are consistent with $\mu_0$ , fixing the mean there costs little and the gap is small. Doubling the gap gives the test statistic:

\begin{aligned} \text{LR} &= -2[\ell(\mu_{0}, \hat\sigma^2_{0}) - \ell(\hat\mu, \hat\sigma^2)]\\ &= 3.77 \end{aligned}

Under the null, LR approximately follows a $\chi^2$ distribution with 1 degree of freedom in large samples. This gives p = .052. At the .05 level, we reject when LR exceeds about 3.84, which is a gap of 1.92. The means whose profile log-likelihood is within 1.92 of the maximum form the 95% profile likelihood interval, reported below the figure. It is exactly the set of $\mu_0$ values this test doesn't reject at the .05 level.

Increasing the sample size makes the peak narrower. The estimated mean hasn't moved, but values around it now fit considerably worse. For the mean in this model, the curvature at the peak is $n/\hat\sigma^2$ , the estimated Fisher information. Taking its inverse gives $\hat\sigma^2/n$ , our estimate of the variance of the sample mean. So the narrower peak corresponds to a smaller standard error.

With only 10 observations, the tests can give rather different answers, because each relies on a large-sample approximation, and so do the intervals. In this normal model the Wald test gives the smallest p-value and the score test the largest, with the likelihood ratio test in between. For this model we don't actually need any of them: the one-sample t-test is exact.

The reason to work through the likelihood-based tests anyway is that none of the steps depended on the normal model. Whatever the model, we write down its log-likelihood, climb to the peak, and read the tests off the same three features: the drop from the peak to the best fit under the null, the distance to the null measured against the curvature at the peak, and the slope at the null. The same steps work for logistic regression, mixed models, or structural equation models, where exact tests usually don't exist. There the surface has more dimensions than we can draw, and more nuisance parameters to profile out, but we handle them just as we handled the variance here.

FAQ

How do I use this visualization?

Use the sliders in the top bar to choose the mean and variance. You can also type a value or press MLE to set a parameter to its maximum likelihood estimate. New Sample draws 10 fresh observations from $\mathcal N(100, 225)$ .

Explore the log-likelihood

Drag the ring on the floor of the 3D figure to change both parameters at once. The two curves through the point (the slices in the key) show what happens when you vary one parameter while holding the other fixed. Below the figure you'll find the current log-likelihood and the two scores, the slopes in the mean and variance directions. Hover or tap a score to show that slope on the surface.

Run an optimizer

Choose gradient ascent or Fisher scoring and press play. Use +1 to take a single step and −1 to undo it. A dashed arrow previews the next step, and the path shows the steps already taken. Gradient ascent has an adjustable step size. For Fisher scoring, dotted parabolas show the local approximations used to choose the next step.

Move around the 3D view

Drag the surface away from the ring to rotate it. The View buttons switch between an angled view and two side views. From the side, the outline of the surface follows the profile curve on the back wall.

The full-view button at the right of each figure's header expands it to fill the window. Press it again, or press Esc, to return.

Why the surface is cut off

Near $\sigma^2 = 0$ , the log-likelihood drops so far that showing it all would make the peak look almost flat. I've cut the figure off 50 log-likelihood units below the maximum so we can see what's happening near the top. The surface fades out below that floor; if you drag a point there, only its floor ring is shown. Above the cutoff, the vertical scale is still linear in log-likelihood.

Inference

Choose a test tab and move the null mean $\mu_0$ . The dashed line across the floor marks the null mean, and the point labeled best under $H_0$ is the fitted model under that null, with variance $\hat\sigma_0^2$ . As you change $\mu_0$ , this point follows the dotted ridge.

The likelihood ratio tab highlights a vertical drop, the Wald tab a horizontal difference, and the score tab a slope. The sample-size slider changes the tests and intervals while keeping the estimated mean and variance fixed; it does not generate a larger sample. Hover or tap a legend entry to highlight its curve.

What are the formulas?

For $n$ independent observations from a normal distribution, the log-likelihood is

\ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{\sum_{i=1}^n (y_i - \mu)^2}{2\sigma^2}

and it is maximized by

\hat\mu = \frac{1}{n}\sum_{i=1}^n y_i, \qquad \hat\sigma^2 = \frac{\sum_{i=1}^n (y_i - \hat\mu)^2}{n}

The variance estimate $\hat\sigma^2$ divides by $n$ , not $n - 1$ , so the MLE of the variance is biased downward in small samples.

The score functions (first derivatives) are

U(\mu) = \frac{n(\bar y - \mu)}{\sigma^2}, \qquad U(\sigma^2) = -\frac{n}{2\sigma^2} + \frac{\sum_i (y_i - \mu)^2}{2\sigma^4}

and the expected (Fisher) information for each parameter is

I(\mu) = \frac{n}{\sigma^2}, \qquad I(\sigma^2) = \frac{n}{2\sigma^4}

The log-likelihood, AIC, and BIC

The maximum, $\ell(\hat\mu, \hat\sigma^2)$ , is the number statistical software reports as a model's log-likelihood. It is also where AIC and BIC come from: both take $-2\ell(\hat\mu, \hat\sigma^2)$ and add a penalty for the number of parameters $k$ , $2k$ for AIC and $k \ln n$ for BIC. Here $k = 2$ .

The three tests

Let $\hat\sigma_0^2$ be the variance estimate when the mean is fixed at $\mu_0$ . In this model, $\hat\sigma_0^2 = \hat\sigma^2 + (\hat\mu - \mu_0)^2$ . The three test statistics are shown below with their large-sample null distributions:

\text{LR} = -2\left[\ell(\mu_0, \hat\sigma_0^2) - \ell(\hat\mu, \hat\sigma^2)\right] = n \ln\frac{\hat\sigma_0^2}{\hat\sigma^2} \sim \chi^2_1

Z = \frac{\hat\mu - \mu_0}{\sqrt{\hat\sigma^2 / n}} \sim \mathcal N(0, 1)

S = \frac{U(\mu_0, \hat\sigma_0^2)^2}{I(\mu_0, \hat\sigma_0^2)} \sim \chi^2_1

The tests are asymptotically equivalent but can differ in small samples. The figure's geometric interpretation follows Buse (1982).

For the normal model all three are functions of the one-sample t statistic. With

T = \frac{\bar y - \mu_0}{s / \sqrt n}, \qquad s^2 = \frac{n}{n - 1}\hat\sigma^2, \qquad x = \frac{T^2}{n - 1}

they are

\text{LR} = n \ln(1 + x), \qquad Z^2 = n x, \qquad S = \frac{n x}{1 + x}

Since $x / (1 + x) \le \ln(1 + x) \le x$ , the statistics always satisfy $Z^2 \ge \text{LR} \ge S$ . Their approximate p-values therefore have the reverse order.

Each statistic is a monotone function of $T^2$ , and $T$ follows a $t_{n-1}$ distribution exactly under the null. If we used each statistic's exact null distribution, all three tests would give the same p-value as the two-sided t-test. The differences here come from using the large-sample $\chi^2_1$ approximation for LR, $Z^2$ , and S.

The profile likelihood interval

Inverting the likelihood ratio test gives the interval of means whose profile log-likelihood stays within $\chi^2_1(0.95)/2$ of the maximum:

\left\{\mu : \ell_p(\mu) \ge \ell_p(\hat\mu) - \tfrac{1}{2}\chi^2_{1}(0.95)\right\}, \qquad \tfrac{1}{2}\chi^2_{1}(0.95) \approx 1.92

Applying the same cutoff to the quadratic approximation gives the Wald interval: a distance of 1.96 standard errors corresponds to a drop of $1.96^2/2 \approx 1.92$ . For this model, the profile likelihood interval has the closed form

\hat\mu \pm \hat\sigma\sqrt{e^{\chi^2_1(0.95)/n} - 1}

The optimizers

Gradient ascent updates each parameter proportionally to its score,

\mu_{t+1} = \mu_t + \gamma\, U(\mu_t), \qquad \sigma^2_{t+1} = \sigma^2_t + \gamma\, U(\sigma^2_t)

where $\gamma$ is the step size (0.05 by default). The app runs this calculation on standardized data, then converts back to the original units for plotting. A small step size slows convergence; a large one can overshoot the maximum.

Fisher scoring is a variant of Newton-Raphson that uses expected information rather than observed information. It scales the score by the inverse information:

\theta_{t+1} = \theta_t + I(\theta_t)^{-1} U(\theta_t)

For this normal model, the first step puts the mean at $\hat\mu$ but updates the variance using the previous mean. The second step gives $\hat\sigma^2$ as well. The app stops a run if a step leaves the plotted range.

How do I cite this page?

Cite this page according to your favorite style guide.

The current version is 1.0.0 (changelog).

APA 7

Magnusson, K. (2026). Understanding maximum likelihood estimation: An interactive visualization (Version 1.0.0) [Web App]. R Psychologist. https://rpsychologist.com/likelihood/

BibTeX

@software{magnussonLikelihood,
  author = {Kristoffer Magnusson},
  title = {Understanding maximum likelihood estimation: An interactive visualization},
  url = {https://rpsychologist.com/likelihood/},
  version = {1.0.0},
  date = {2026-10-24},
}

I found a bug/error/typo or want to make a suggestion!

Please report errors or suggestions by opening an issue on GitHub; if you want to ask a question, use GitHub Discussions.

I'm gonna ask a large number of students to visit this site. Will it crash your server?

No, it will be fine. The app runs in your browser, so the server only needs to serve the files.

Can I include this visualization in my book/article/etc?

Yes, go ahead! This visualization is dedicated to the public domain (CC0), which means you can copy, modify, and distribute it, even for commercial purposes, without asking permission. Attribution is not required but always appreciated. The source code is licensed MIT, and the text on this page is CC-BY 4.0.

References

Buse, A. (1982). The likelihood ratio, Wald, and Lagrange multiplier tests: An expository note. The American Statistician, 36(3a), 153–157. https://doi.org/10.1080/00031305.1982.10482817

More visualizations

All visualizations →