Homepage > MCMC

Mixture Model

A simple mixture model has a probability density function

$\displaystyle \theta_1 f_1(\xi) + \cdots + \theta_k f_k(\xi)

with weight parameter $ \theta = (\theta_1,\ldots,\theta_k)$. Here the components $ f_1, \ldots, f_k$ are probability density functions, and assumed to be entirely known.

Posterior density. Given the data $ x = (x_1,\ldots,x_n)$ of $ n$ independent observations from the mixture density and the flat prior, the posterior density $ \pi^\star(\theta)$ of weight parameter is proportional to

$\displaystyle f(x \:\vert\: \theta) =
\prod_{i=1}^n \: [\theta_1 f_1(x_i) + \cdots + \theta_k f_k(x_i)].

However, the numerical analysis of posterior density $ \pi^\star(\theta)$ on the simplex

$\displaystyle \mathcal{Q} = \{\theta\in (0,1)^k: \sum_{j=1}^k \theta_j = 1\}

is rather very hard. Alternatively, we can devise an MCMC scheme.

Latent Variables. Then we introduce the following latent variable setup: Let $ z_i$ be a latent variable indicating to which component the $ i$-th observation $ x_i$ belongs, and let $ z = (z_1,\ldots,z_n)$ be the vector of latent variables on the space

$\displaystyle \mathcal{Z} = \{1,\ldots,k\}^n.

By $ I_A(x)$ we denote the indicator function $ I_A(x) = 1$ or 0 accordingly as $ x\in A$ or not. So that we can define

$\displaystyle m_j(z) = \sum_{i=1}^n I_{\{j\}}(z_i), j = 1,\ldots,k

In short, $ m_j(z)$ denotes the number of $ z_i$'s set to $ z_i = j$. In this manner the latent variable $ z$ can be together lumped into $ z^\dagger = (m_1(z),\ldots,m_k(z))$.

© TTU Mathematics