Bayesian Statistics
5.3 Reference priors
An interesting response to the argument in Section 5.2.1 was given by the statistician Harold Jeffreys in 1946. It leads to a particular suggestion for the choice of prior. Suppose that two different people, Alice and Bob, construct a Bayesian model, with one parameter. Alice uses the model family \((M_\theta )_{\theta \in \Pi }\) and Bob use the model family \((M_\varphi )_{\varphi \in \Pi }\), where \(\theta \) and \(\varphi \) are related by some function \(h(\theta )=\varphi \), where \(h:\Pi \to \Pi \) with \(\Pi \sw \R \) and \(h\) is strictly monotone increasing and differentiable. That is, they use the ‘same’ model family, but parametrize it differently.
Alice will use prior p.d.f. \(f_1\) and Bob will use prior p.d.f. \(f_2\). This means that Alice constructs a model with sampling distribution
\[f_{X_1}(x)=\int _\Pi f_{M_\theta }(x)f_1(\theta )\,d\theta \]
and Bob constructs a model with sampling distribution
\[f_{X_2}(x)=\int _\Pi f_{M_{h(\theta )}}(x)f_2(\theta )\,d\theta .\]
Alice and Bob have never met each other, and in fact they do not even know that each other exists. Neither of them knows the function \(h\).
This is where we come in. We write the statistics textbook, that both Alice and Bob will both read. They will choose their prior based on our instructions – the same instructions, for both people. Can we provide Alice and Bob with a way to choose their individual priors that will make their models equal i.e. so that \(f_{X_1}(x)=f_{X_2}(x)\)?
-
Remark 5.3.1 If Alice and Bob did meet each other, then Alice could tell Bob what her prior \(\Theta \) was and by comparing notes they could work out the function \(h\). Bob could then choose his prior to be the p.d.f. of \(h(\Theta )\), where \(\Theta \) is Alice’s prior. This choice makes \(f_{X_1}(x)=f_{X_2}(x)\), see Exercise 5.7.
Returning to the situation where Alice and Bob do not meet, the surprising answer to the problem is: yes, this is possible. The solution is that we should tell them both to use the prior
\(\seteqnumber{0}{5.}{0}\)\begin{equation} \label {eq:jeffreys_prior} f(\theta )\propto \E \l [\l (\frac {d}{d\lambda }\log (L_{M_{\lambda }}(X))\r )^2\r ]^{1/2} \quad \text { where }(M_\lambda )\text { is their chosen model family and }X\sim M_\lambda . \end{equation}
Let us not worry about how Jeffreys found this solution, and let us just show that it really works.
Proof that the solution works: \(\offsyl \) Alice writes down her prior \(f_1(\theta )\propto \E [(\frac {d}{d\theta }\log (L_{M_{\theta }}(X)))^2]^{1/2}\) and Bob writes down his prior, \(f_2(\varphi )\propto \E [(\frac {d}{d\varphi }\log (L_{M_{h(\varphi )}}(X)))^2]^{1/2}\). Then Alice’s model is
\[f_{X_2}(x) \propto \int _\Pi f_{M_{\theta }}(x)f_1(\theta )\,d\theta . \]
Alice doesn’t know the function \(h\), but substituting \(\theta =h(\lambda )\), her model is equal to
\(\seteqnumber{0}{5.}{1}\)\begin{align*} f_{X_1}(x) \propto \int _\Pi f_{M_{h(\lambda )}}(x)f_1(h(\lambda ))h'(\lambda )\,d\lambda \\ \propto \int _\Pi f_{M_{h(\lambda )}}(x)f_1(h(\lambda ))h'(\lambda )\,d\lambda \\ \propto \int _\Pi f_{M_{h(\theta )}}(x)f_1(h(\theta ))h'(\theta )\,d\theta . \end{align*} In the last line we have simply changed notation by setting \(\lambda =\theta \). Meanwhile, Bob’s model is equal to
\[f_{X_1}(x) \propto \int _\Pi f_{M_{h(\theta )}}(x)f_2(\theta )\,d\theta . \]
It follows that \(f_{X_1}(x)=f_{X_2}(x)\) if we have \(f_2(\theta )\propto f_1(h(\theta ))h'(\theta )\). We will now show that this equation holds, for any strictly monotone function \(h\). Writing \(\varphi =h(\theta )\),
\(\seteqnumber{0}{5.}{1}\)\begin{align*} f_2(\varphi )^2 &\propto \E \l [\l (\frac {d}{d\varphi }\log (L_{M_{\varphi }}(X))\r )^2\r ] \\ &\propto \E \l [\l (\frac {d}{d\theta }\log (L_{M_{h(\theta )}}(X))\times \frac {d\varphi }{d\theta }\r )^2\r ] \\ &\propto \E \l [\l (\frac {d}{d\theta }\log (L_{M_{h(\theta )}}(X))\times h'(\theta )\r )^2\r ] \\ &=f_1(h(\theta ))^2h'(\theta )^2. \end{align*} To reach the second line we use the chain rule. Taking square roots, we obtain that \(f_{X_1}(x)=f_{X_2}(x)\) as required. ∎
In the earlier half of the 20th the argument in Section 5.2.1 was taken quite seriously, and treated as a major philosophical reason to question the reliability of Bayesian statistics. In particular, the objection was that if both Alice and Bob tried to use the same uninformative prior but used models that were parametrized differently then they would obtain different results, despite having the same intentions and, from their own perspectives, the same methodology. Jeffreys showed that this difficulty could be entirely avoided with a particular choice of prior.
These arguments took place before modern computers, when it was difficult to test how well Bayesian methods worked in practice (except for conjugate priors). We are now better able to test how much different modelling errors matter. Statisticians today no longer attach much weight to this objection.
Starting from the ideas above and those in Remark 5.2.2, there is a modern branch of statistics that investigates uninformative priors with particular theoretical properties. A modern approach is to use a prior that tries to maximise the difference (in some sense) between the prior and posterior distribution, essentially seeking to maximise the influence of the data. Priors with this property are known as reference priors. Their theory is beyond what we can cover here, but it turns out that if we have only one parameter then the reference prior is the same as the prior proposed by Jeffreys – and for multi-parameter models, it is not. We will only study the one-parameter case here. Let us investigate what the reference prior looks like for some particular choices of one-parameter model.
-
Definition 5.3.2 Suppose that \((M_\theta )\) is a family of distributions with parameter space \(\Pi \sw \R \). The reference prior \(\Theta \) associated \((M_\theta )\) has p.d.f. given by
\(\seteqnumber{0}{5.}{1}\)\begin{equation} \label {eq:reference_prior_d1} f_{\Theta }(\theta )\propto \E \l [\l (\frac {d}{d\theta }\log (L_{M_{\theta }}(X))\r )^2\r ]^{1/2}. \end{equation}
There are caveats to this definition. The reference prior might be an improper prior, or if the expectation in (5.2) is not finite then the reference prior may not exist.
Sometimes the reference prior \(\Theta \) for \((M_\theta )\) is easier to find via the equation
\(\seteqnumber{0}{5.}{2}\)\begin{equation} \label {eq:reference_prior_d2} f_{\Theta }(\theta )\propto \E \l [-\frac {d^2}{d\theta ^2}\log (L_{M_{\theta }}(X))\r ]^{1/2}. \end{equation}
-
Remark 5.3.3 \(\offsyl \) To deduce (5.3) from (5.2), use the partial differentiation identity \(\frac {\partial ^2}{\partial \theta ^2} \log f(x,\theta ) = \frac {1}{f(x,\theta )}\frac {\partial ^2}{\partial \theta ^2} f(x,\theta ) - \big ( \frac {\partial }{\partial \theta } \log f(x,\theta )\big )^2\) and that \(\E \big [ \frac {1}{f(X;\theta )}\frac {\partial ^2}{\partial \theta ^2}f(X, \theta ) \, \big | \, \theta \big ] = \frac {\partial ^2}{\partial \theta ^2} \int _{\mathbb {R}} f(x,\theta )\,dx = 0\). We omit the details.
-
Example 5.3.4 For the Bernoulli model family \((M_p)_{p\in [0,1]}\) where \(M_p\sim \Bern (p)\), the likelihood is
\[L_{M_p}(x)= \begin {cases} p^x(1-p)^{1-x} & \text { for }p\in [0,1] \\ 0 & \text { otherwise} \end {cases} \]
for \(x\in \{0,1\}\). Hence \(\frac {d}{dp}\log (L_{M_p}(x))=\frac {p}{dp}(x\log p + (1-x)\log (1-p))=\frac {x}{p}+\frac {1-x}{1-p}\). For \(p\in [0,1]\), the density function of the reference prior \(P\) is given by
\(\seteqnumber{0}{5.}{3}\)\begin{align*} f_P(p) &\propto \E \l [\l (\frac {d}{dp}\log (L_{M_{p}}(X))\r )^2\r ]^{1/2} \\ &\propto \E \l [\l (\frac {X}{p}+\frac {1-X}{1-p}\r )^2\r ]^{1/2} \\ &\propto \l (p\l (\frac {1}{p}+\frac {1-1}{1-p}\r )^2+(1-p)\l (\frac {0}{p}+\frac {1-0}{1-p}\r )^2\r )^{1/2} \\ &\propto \l (\frac {1}{p}+\frac {1}{1-p}\r )^{1/2} \\ &\propto p^{-1/2}(1-p)^{-1/2}. \end{align*} Using Lemma 1.2.5 we recognize that \(P\sim \Beta (\frac 12,\frac 12)\).
A useful fact is that the reference prior for \(M_\theta \) and \(M_\theta ^{\otimes n}\) are identical, in the sense that they are \(\propto \) to each other. This is shown in Exercise 5.8.