Bayesian Statistics

6.3 Exercises on Chapter 6

• 6.1 $\star$ Show that the mode of the $\mathrm{Gamma}(\alpha ,\beta )$ distribution is $\frac{\alpha -1}{\beta }$, where $\alpha \ge 1$. What about $\alpha \in (0,1)$?

• 6.2 $\star \star$ The following equations, written in Bayesian shorthand, are the key conclusions from results in earlier chapters of these notes. Which results are they from?

  • (a) $f(x|y)=\frac{f(y,x)}{f(y)}$.

  • (b) If $\theta \sim \mathrm{Beta}(\alpha ,\beta )$ and $x|\theta \sim \mathrm{Bernoulli}(\theta {)}^{\otimes n}$ then $\theta |x\sim \mathrm{Beta}(\alpha +k,\beta +n-k)$, where $x=(x_i{)}_1^n$ and $k=\sum _1^n{x}_i$.

  Write the following results in Bayesian shorthand, using similar notation to that in parts (a) and (b).

  • (c) Lemma 4.2.1.

  • (d) From Section 4.5, the two facts above Lemma 4.5.2 concerning marginal and conditional distributions of the $\mathrm{NGamma}$ distribution.

• 6.3 $\star \star$ The following results are written in Bayesian shorthand.

  • (a) If $x\sim N(0,1)$ then $x|\{x>0\}\sim |x|$.

  • (b) If $x$ and $y$ are independent then $x|y\sim x$.

  In each case, write a version of the results in precise mathematical notation. Which parts of Chapter 1 are they closely related to?

• 6.4 $\star \star$ Suppose that we model $x|\theta \sim \mathrm{NegBin}(m,\theta {)}^{\otimes n}$, where $m\in \mathbb{N}$ is fixed and $\theta \in (0,1)$ is an unknown parameter.

  • (a) Show that $f(x|\theta )\propto {\theta }^{mn}(1-\theta {)}^{\sum _1^n{x}_i}.$

  • (b) Show that the prior $\theta \sim \mathrm{Beta}(\alpha ,\beta )$ is conjugate to $\mathrm{NegBin}(m,\theta {)}^{\otimes n}$, and find the posterior parameters.

  • (c)

    • (i) Show that the reference prior for $\theta$ is given by $f(\theta )\propto {\theta }^{-1}(1-\theta {)}^{-1/2}$.

    • (ii) Does $f(\theta )$ define a proper distribution?

    • (iii) Find the posterior density $f(\theta |x)$ arising from this prior.

  Hint: The setup given is a Bayesian model with model family $M_{\theta }\sim \mathrm{NegBin}(m,\theta {)}^{\otimes n}$.

• 6.5 Suppose that we model $x|\mu ,\tau \sim \mathrm{N}(\mu ,\frac{1}{\tau }{)}^{\otimes n}$, where both $\mu$ and $\tau$ are unknown parameters. We use the improper prior $f(\mu ,\tau )\propto \frac{1}{\tau }$ for $\tau >0$, and $f(\tau )=0$ elsewhere.

  • (a) $\star \star$ Show that for $\mu \in \mathbb{R}$ and $\tau >0$ the posterior distribution satisfies

    $$f(\mu ,\tau |x)\propto {\tau }^{\frac{n}{2}-1}\exp (-\frac{\tau }{2}\sum _{i=1}^n(x_i-\mu {)}^2).$$

  • (b) $\star \star \star$ Find the marginal p.d.f of $\tau |x$. Show that $(\mu ,\tau )|x$ is a proper distribution if and only if $n\ge 2$.

  Hint: The setup given is a Bayesian model with model family $M_{\mu ,\tau }\sim \mathrm{N}(\mu ,\frac{1}{\tau }{)}^{\otimes n}$. For part (b) use the sample-mean-variance identity (4.10).

• 6.6 $\star \star$ Let $(M_{\theta }{)}_{\theta \in \mathrm{\Pi }}$ be a continuous family of distributions. For $i=1,2,$ let ${\mathrm{\Theta }}_i$ be a continuous random variable with p.d.f. $f_{{\mathrm{\Theta }}_i}$, both taking values in ${\mathbb{R}}^d$. Let $\alpha ,\beta \in (0,1)$ be such that $\alpha +\beta =1$.

  • (a) Show that $f_{\mathrm{\Theta }}(\theta )=\alpha f_{{\mathrm{\Theta }}_1}(\theta )+\beta f_{{\mathrm{\Theta }}_2}(\theta )$ is a probability density function.

  • (b) Consider Bayesian models $(X_1,{\mathrm{\Theta }}_1)$ and $(X_2,{\mathrm{\Theta }}_2)$, with the same model family $(M_{\theta })$ and different prior distributions. Consider also a third Bayesian model $(X,\mathrm{\Theta })$ with model family $(M_{\theta })$ and prior $\mathrm{\Theta }$ with p.d.f. $f_{\mathrm{\Theta }}(\theta )=\alpha f_{{\mathrm{\Theta }}_1}(\theta )+\beta f_{{\mathrm{\Theta }}_2}(\theta )$.

    Show that the posterior distributions of these three models satisfy

    $$f_{\mathrm{\Theta }{|}_{\{X=x\}}}(\theta )={\alpha }^{\prime }{f}_{{\mathrm{\Theta }}_1{|}_{\{X_1=x\}}}(\theta )+{\beta }^{\prime }{f}_{{\mathrm{\Theta }}_2{|}_{\{X_2=x\}}}(\theta )$$

    where ${\alpha }^{\prime }=\frac{\alpha Z_1}{\alpha Z_1+\beta Z_2}$ and ${\beta }^{\prime }=\frac{\beta Z_2}{\alpha Z_1+\beta Z_2}$. Here $Z_1$ and $Z_2$ are the normalizing constants given in Theorem 3.1.2 for the posterior distributions of $(X_1,{\mathrm{\Theta }}_1)$ and $(X_2,{\mathrm{\Theta }}_2)$.

  • (c) Outline briefly how to modify your argument in (c) to also cover the case of discrete Bayesian models.

• 6.7 $\star \star \star$ This question explores the idea in Exercise 4.6 further, but except for (a)(ii) it does not depend on having completed that exercise.

  • (a) Let $(M_{\theta })$ be a discrete or absolutely continuous family with range $R$. Let $(X,\mathrm{\Theta })$ be a Bayesian model with model family $M_{\theta }^{\otimes n}$. Let $x\in R^n$ and write $x(1)=(x_1,\dots ,x_{n_1})$, $x(2)=(x_{n_1+1},\dots ,x_n)$. Let $(X_1,\mathrm{\Theta })$ and $(X_2,\mathrm{\Theta }{|}_{\{X_1=x(1)\}})$ be Bayesian models with model families $M_{\theta }^{\otimes n_1}$ and $M_{\theta }^{\otimes n_2}$, where $n_1+n_2=n$.

    • (i) Show that

      $$({\mathrm{\Theta }}_1{|}_{\{X_1=x(1)\}}){|}_{\{X_2=x(2)\}}\stackrel{\mathrm{d}}{=}\mathrm{\Theta }{|}_{\{X=x\}}.$$

      Use likelihood functions to write your argument in a way that covers both the discrete and absolutely continuous cases.

    • (ii) What is the connection between this fact and Exercise 4.6?

  • (b) Rewrite your solution to (a)(i) in a Bayesian shorthand notation of your choice.