Bayesian Statistics
Chapter 4 Conjugate priors
Recall that we define a Bayesian model \((X,\Theta )\) using a prior \(\Theta \) and a model family \((M_\theta )_{\theta \in \Pi }\). The main result from Chapters 2 and 3 is that we can (at least, in principle) obtain the distribution of the posterior \(\Theta |_{\{X=x\}}\). If the prior and posterior are from the same family of distributions then we say that this family of distributions is conjugate to the model family.
Note that the setup here involves two families of distributions: (1) the model family \((M_\theta )\) and (2) the conjugate family, to which the prior and posterior both belong. The formal definition is a bit of a mouthful:
-
Definition 4.0.1 Let \((M_\theta )_{\theta \in \Pi }\) and \((T_a)_{a\in A}\) be two model families, with parameter spaces \(\Pi \) and \(A\) respectively. We say that \((M_\theta )\) and \((T_a)\) are a conjugate pair if whenever \((X,\Theta )\) is a Bayesian model with model family \((M_\theta )\) and prior \(\Theta \sim T_a\), for all \(x\in R_X\) there exists \(b\in A\) such that \(\Theta |_{\{X=x\}}\sim T_b\). We say that the family \((T_a)\) is a conjugate prior for \((M_\theta )\).
The point of using conjugate pairs is that, to specify a Bayesian update step, we only need to describe how the parameters of the prior distribution should change, to obtain the posterior distribution. This is in general much simpler than (2.11) and (3.3). We will describe a few conjugate pairs in this chapter and discuss their limitations in Section 4.6.
4.1 Notation: proportionality
In (2.12) and (3.7) we used \(\frac {1}{Z}\) and \(\frac {1}{Z'}\) for normalizing constants. It was helpful not to worry about exactly what the value of these constant were. In longer calculations we might need to use several different normalizing constants in this way, and it is helpful to have some notation for doing so (beyond simply \(\frac {1}{Z''},\frac {1}{Z'''}\) and so on).
The relation \(\propto \) has several nice properties, which are easy to check and are left for you in Exercise 4.9. For example, for any function \(f\) we have \(f\propto f\). Also, \(f\propto g\) if and only if \(g\propto f\) and, lastly, if \(f\propto g\) and \(g\propto h\) then \(f\propto h\). We’ll use these properties frequently in calculations, without further comment.
-
Example 4.1.2 Using the notation \(\propto \), Lemma 1.2.5 says that for random variables \(X\) and \(Y\):
-
• If \(X\) and \(Y\) are discrete and \(p_X\propto p_Y\) then \(X\eqd Y\).
-
• If \(X\) and \(Y\) are continuous and \(f_X\propto f_Y\) then \(X\eqd Y\).
We will often use Lemma 1.2.5 in this way from now on, including in the next example.
-
-
Example 4.1.3 The calculation in (2.12) can be written simply as
\(\seteqnumber{0}{4.}{0}\)\begin{align*} f_{P|_{\{X=4\}}}(p) &\propto \P [\Bin (10,p)=4]f_{\Beta (2,8)}(p) \\ &\propto p^4(1-p)^{10-4}p^{2-1}(1-p)^{8-1} \\ &\propto p^{5}(1-p)^{13}. \end{align*} It follows immediately from Lemma 1.2.5 that \(P|_{\{X=4\}}\sim \Beta (6,14)\).
-
Example 4.1.4 More generally, the key equations (2.11) and (3.3) from Theorems 2.4.1 and 3.1.2 can be written
\(\seteqnumber{0}{4.}{0}\)\begin{align*} f_{\Theta |_{\{X=x\}}}(\theta )\propto p_{M_\theta }(x)f_{\Theta }(\theta ) & \qquad \text { for discrete Bayesian models,} \\ f_{\Theta |_{\{X=x\}}}(\theta )\propto f_{M_\theta }(x)f_{\Theta }(\theta ) & \qquad \text { for continuous Bayesian models.} \end{align*} We will often use Theorems 2.4.1 and 3.1.2 in this way from now on.
A complication of using \(\propto \) is that the symbol does not explicitly specify which variables should be treated as part of the proportionality, and which other variables can be treated as constants. For our purposes there is a simple way to resolve this difficulty. We we use \(\propto \) there will, in most cases, be a function on the left of the first \(\propto \) that appears within a calculation. The arguments of that function (not including subscripts) are the variables that proportionality applies to; everything else can be treated as constant in so far as \(\propto \) is concerned.
4.1.1 The Beta-Binomial pair
Here is our first example of a conjugate pair, which generalizes the calculations in Example 2.1.1-2.3.3.
-
Lemma 4.1.5 (Beta-Bernoulli conjugate pair) Let \(n\in \N \). Let \((X,\Theta )\) be a discrete Bayesian model with model family \(M_\theta \sim \Bern (\theta )^{\otimes n}\) and parameter \(\theta \in [0,1]\). Suppose that the prior is \(\Theta \sim \Beta (a,b)\) and let \(x\in \{0,1\}^n\). Then the posterior is \(\Theta |_{\{X=x\}}\sim \Beta (a+k,b+n-k)\) where \(k=\sum _1^n x_i\).
Proof: Note that \(k\) is the number of Bernoulli trials that generate a \(1\), and that we have \(n\) trials in total. Under \(M_p\), each trial has probability \(p\) of generating \(1\). From Theorem 2.4.1 we have that for \(\theta \in [0,1]\)
\(\seteqnumber{0}{4.}{0}\)\begin{align*} f_{\Theta |_{\{X=x\}}}(\theta ) &\propto \P [\Bern (\theta )^{\otimes n}=x] f_{\Beta (a,b)}(\theta ) \\ &\propto \l (\prod _{i=1}^n\P [\Bern (\theta )=x_i]\r ) f_{\Beta (a,b)}(\theta ) \\ &\propto \l (\theta ^k(1-\theta )^{n-k}\r )\l (\frac {1}{\mc {B}(a,b)}\theta ^{a-1}(1-\theta )^{b-1}\r )\\ &\propto \theta ^{a+k-1}(1-\theta )^{b+n-k-1}. \end{align*} By Lemma 1.2.5 we recognize this p.d.f. as \(\Theta |_{\{X=x\}}\sim \Beta (a+k,b+n-k)\). ∎
The value of \(k\) has an intuitive interpretation, because it is the number of successful trials observed in our data \(x\) (here we take a trial to result in \(1\) if successful, and \(0\) if failed). Looking back at Example 2.3.3, this allows us to do all of the Bayesian update calculations with one easy piece of arithmetic.
More generally, we can use the Binomial distribution in place of the Bernoulli distribution, as in the next lemma.
-
Lemma 4.1.6 (Beta-Binomial conjugate pair) Let \(n,m_i\in \N \). Let \((X,\Theta )\) be a discrete Bayesian model with model family
\[M_\theta \sim \Bin (m_1,\theta )\otimes \ldots \otimes \Bin (m_n,\theta ).\]
with the parameter \(\theta \in \Pi =[0,1]\). Suppose that the prior is \(\Theta \sim \Beta (a,b)\) and let \(x=(x_1,\ldots ,x_n)\) where \(x_i\in \{0,\ldots ,m_i\}\). Then the posterior is \(\Theta |_{\{X=x\}}\sim \Beta \l (a+\sum _1^n x_i,b+\sum _1^n m_i-\sum _1^n x_i\r )\).
Proof: From Theorem 2.4.1 we have that for \(\theta \in [0,1]\)
\(\seteqnumber{0}{4.}{0}\)\begin{align*} f_{\Theta |_{\{X=x\}}}(\theta ) &\propto \l (\prod _{i=1}^n\binom {m_i}{x_i}\theta ^{x_i}(1-\theta )^{m_i-x_i}\r )\l (\frac {1}{\mc {B}(a,b)}\theta ^{a-1}(1-\theta )^{b-1}\r )\\ &\propto \theta ^{a+\sum _1^n x_i-1}(1-\theta )^{b+\sum _1^nm_i-\sum _1^nx_i-1}. \end{align*} By Lemma 1.2.5 we recognize this p.d.f. as \(\Theta |_{\{X=x\}}\sim \Beta \l (a+\sum _1^n x_i,b+\sum _1^n m_i-\sum _1^n x_i\r )\). ∎