Bayesian Statistics

1.7 Families with random parameters

In this section we are interested to take a model family $(M_{\theta }{)}_{\theta \in \mathrm{\Pi }}$ and treat the parameter $\theta$ as a random variable, which will be denoted by a capital letter $\mathrm{\Theta }$. We think of first sampling the value of $\mathrm{\Theta }$ and then (using whatever value we obtain) taking a sample $X$ from $M_{\mathrm{\Theta }}$. The resulting distribution is sometimes known as a compound or mixture distribution. We will have a detailed discussion of how this idea becomes useful in Section 2.1. For now let us note that it increases the range of models that we have available.

To make sense of the idea, let us state it more precisely. We want random variables $X$ and $\mathrm{\Theta }$ such that $X{|}_{\{\mathrm{\Theta }=\theta \}}\stackrel{\mathrm{d}}{=}{M}_{\theta }$. In this section we show that a pair $(X,\mathrm{\Theta })$ with this property is given by the distribution

$$\begin{array}{}\text{(1.13)}& \mathbb{P}[X\in A,\mathrm{\Theta }\in B]={\int }_B\mathbb{P}[M_{\theta }\in A]f_{\mathrm{\Theta }}(\theta )d\theta .\end{array}$$

where $(M_{\theta })$ is a family of distributions with range $R\subseteq {\mathbb{R}}^n$, as defined in Section 1.3, and $f_{\mathrm{\Theta }}$ is a probability density function with range $\mathrm{\Pi }\subseteq {\mathbb{R}}^d$. This is a type of random variable you may not have seen before. We will shortly show that the $\mathrm{\Theta }$ part is a continuous random variable, but the $X$ part might be discrete or continuous, depending on $(M_{\theta })$.

Our notation strongly suggests that we expect $f_{\mathrm{\Theta }}$ to be the (marginal) probability density function of $\mathrm{\Theta }$, and we can confirm this by setting $A={\mathbb{R}}^n$, in which case equation (1.13) becomes $\mathbb{P}[\mathrm{\Theta }\in B]={\int }_B{f}_{\mathrm{\Theta }}(\theta )$. We can also find the marginal distribution of $X$, by setting $B={\mathbb{R}}^d$, giving

$$\mathbb{P}[X\in A]=\mathbb{P}[X\in A,\mathrm{\Theta }\in {\mathbb{R}}^d]={\int }_{{\mathbb{R}}^d}\mathbb{P}[M_{\theta }\in A]f_{\mathrm{\Theta }}(\theta )d\theta ,$$

but that formula doesn’t really explain what is going on here. The relationship that we are interested in comes from the following lemma.

Proof: $(\oslash )$ We give a sketch proof to illustrate the idea, in similar style to Lemma 1.6.1. From Lemma 1.5.1, for $A\in {\mathbb{R}}^d$ we have

$$\mathbb{P}[X{|}_{\{|\mathrm{\Theta }-\theta |\le ϵ\}}\in A]=\frac{\mathbb{P}[|\mathrm{\Theta }-\theta |\le ϵ,X\in A]}{\mathbb{P}[|\mathrm{\Theta }-\theta |\le ϵ]}=\frac{{\int }_{\theta -ϵ}^{\theta +ϵ}\mathbb{P}[M_t\in A]f_{\mathrm{\Theta }}(t)dt}{{\int }_{\theta -ϵ}^{\theta +ϵ}{f}_{\mathrm{\Theta }}(t)dt}$$

The second equality follows from (1.13) with $B=[\theta -ϵ,\theta +ϵ]$, for the numerator, and from the fact that $f_{\mathrm{\Theta }}$ is the p.d.f. of $\mathrm{\Theta }$, for the denominator. Using continuity, from the statement of the lemma and from Assumption 1.3.2, for $|\theta -t|\le ϵ$ we can approximate $f_{\mathrm{\Theta }}(t)\approx f_{\mathrm{\Theta }}(\theta )$ and $\mathbb{P}[M_t\in A]\approx \mathbb{P}[M_{\theta }\in A]$. This gives

$$\mathbb{P}[X{|}_{\{|\mathrm{\Theta }-\theta |\le ϵ}\in A]\approx \frac{{\int }_{\theta -ϵ}^{\theta +ϵ}\mathbb{P}[M_{\theta }\in A]f_{\mathrm{\Theta }}(\theta )dt}{{\int }_{\theta -ϵ}^{\theta +ϵ}{f}_{\mathrm{\Theta }}(\theta )dt}=\frac{2ϵf_{\mathrm{\Theta }}(\theta )\mathbb{P}[M_{\theta }\in A]}{2ϵf_{\mathrm{\Theta }}(\theta )}=\mathbb{P}[M_{\theta }\in A].$$

Letting $ϵ\to 0$ we have $\mathbb{P}[X{|}_{\{|\mathrm{\Theta }-\theta |\le ϵ}\in A]\to \mathbb{P}[M_{\theta }\in A]$, so by Definition 1.2.1 and (1.9) we have that $X{|}_{\{\mathrm{\Theta }=\theta \}}$ is well defined and $X{|}_{\{\mathrm{\Theta }=\theta \}}\stackrel{\mathrm{d}}{=}{M}_{\theta }$.   ∎