Homepage > MCMC

Frequentist Approach

A random sample

$\displaystyle X_1,\ldots,X_n

is regarded as independent and identically distributed (iid) random variables governed by an underlying probability density function  $ f(x; \theta)$. A value $ \theta$ represents the characteristics of this underlying distribution, and is called a parameter. A point estimate is a ``best guess'' for the true value $ \theta$. Suppose that the underlying distribution is the normal distribution with $ (\mu,\sigma^2)$. Then the sample mean

$\displaystyle \bar{X} = \frac{1}{n}\sum_{i=1}^n X_i

is in some sense a best guess of the parameter $ \mu$.

Maximum likelihood estimate. Having observed a random sample $ (X_1,\ldots,X_n) = (x_1,\ldots,x_n)$ from an underlying pdf $ f(x; \theta)$, we can construct the likelihood function

$\displaystyle L(\theta,\mathbf{x}) = \prod_{i=1}^n f(x_i; \theta),

and consider it as a function of $ \theta$. Then the maximum likelihood estimate (MLE)  $ \hat{\theta}$ is the value of $ \theta$ which ``maximizes'' the likelihood function  $ L(\theta,\mathbf{x})$. It is usually easier to maximize the log likelihood

$\displaystyle \ln L(\theta,\mathbf{x}) = \sum_{i=1}^n \ln f(x_i; \theta).

Bernoulli trials. Let $ X$ be a random variable taking value only on 0 and 1. An experiment observing such a variable is called Bernoulli trial. It is determined by the parameter $ \theta$ (which represents the probability that $ X = 1$). Suppose that $ \mathbf{x} = (x_1,\ldots,x_n)$ are observed from $ n$ independent Bernoulli trials. Then the joint density function is given by

$\displaystyle f(\mathbf{x};\theta) = \theta^{\sum_{i=1}^n x_i}
(1-\theta)^{n-\sum_{i=1}^n x_i}

By solving the equation

$\displaystyle \frac{\partial\ln L(\theta,\mathbf{x})}{\partial\theta}
= {\text...
- {\textstyle\left(n - \sum_{i=1}^n x_i\right)} \frac{1}{1 - \theta} = 0,

we obtain $ \theta^* = \sum_{i=1}^n x_i \big/ n$. In fact, $ \theta^*$ maximizes $ \ln L(\theta,\mathbf{x})$, and therefore, it is the MLE of $ \theta$.

© TTU Mathematics