Bayesian Statistics

\CustomizeMathJax

4.3 Conjugate pairs and the exponential family $(⊘)$

For continuous Bayesian models, a very general form of conjugacy is known. We’ll restrict here to continuous models with one unknown parameter. We need to introduce the exponential family of distributions. This term does not refer to the exponential distribution, but to a much wider class. We say that a real valued random variable $Y$ is from the exponential family of distributions with parameter $θ$ if it has a p.d.f. in the form

$\begin{matrix} (4.3) & f_{Y} (y) = h (y) g (θ) \exp (θ T (y)) . \end{matrix}$

Here $h, g$ and $T$ are arbitrary functions, with the restriction $h \geq 0$ . Many of the families of distributions that you are familiar with can be fitted into this mould, including the normal, exponential, gamma, chi-squared, and beta distributions.

Take a Bayesian model $(X, Θ)$ with model family given by (4.3), where both $X$ and $Θ$ take values in $R$ . That is,

$\begin{matrix} (4.4) & f_{M_{θ}} (x) = h (x) g (θ) \exp (θ T (x)) \end{matrix}$

for $x, θ \in R$ . We’ll focus on the version of this model that takes $n$ independent items of data, $x = (x_{1}, \dots, x_{n})$ in which case

$\begin{matrix} (4.5) & f_{M_{θ}} (x) = (\prod_{i = 1}^{n} h (x_{i})) g (θ)^{n} \exp (θ \sum_{i = 1}^{n} T (x_{i})) . \end{matrix}$

The prior distribution that provides a conjugate pair to this model is given by

$\begin{matrix} (4.6) & f_{Θ} (θ) \propto g (θ)^{a} \exp (b θ), \end{matrix}$

where $a > 0$ and $b \in R$ are parameters Note that this distribution is a specialized version of the form in (4.3), and that the function $g$ must be the same as in (4.5). Theorem 3.1.2 allows us to compute the posterior density

$\begin{aligned} f_{Θ |_{{X = x}}} (θ) & \propto f_{M_{θ}} (x) f_{Θ} (θ) \\ (4.7) & \propto g (θ)^{n + a} \exp [θ (b + \sum_{i = 1}^{n} T (x_{i}))] . \end{aligned}$ Therefore the Bayesian update in this case, from the parameters in (4.6) to (4.7) is that

$(a, b) \mapsto (a + n, b + \sum_{i = 1}^{n} T (x_{i})) .$

Example 4.3.1 Taking $h (x) = 1$ , $g (θ) = θ$ and $T (x) = x$ obtains the $Exp (θ)$ in (4.4). Making the same choice for $g$ , along with $a = α$ and $b = - β$ , obtains that $Gamma (α, β)$ distribution in (4.6). Putting these choices for $g$ and $T$ into (4.7) and applying Lemma 1.2.5 gives that the posterior is $Gamma (α + n, β + \sum_{1}^{n} x_{i})$ , as we already knew from Lemma 4.2.1.

Remark 4.3.2 There is also a version of this framework for multiple parameters, in which $θ$ and $b$ are row vectors and $T$ is a column vector, and multiplication of these quantities is done via the dot product. We won’t write down the details of that case here. There is also a version that applies to discrete Bayesian models, but we won’t detail that either.

Bayesian Statistics

4.3 Conjugate pairs and the exponential family (⊘)

4.3 Conjugate pairs and the exponential family $(⊘)$