Bayesian Statistics

$\newcommand{\footnotename}{footnote}$ $\def \LWRfootnote {1}$ $\newcommand {\footnote }[2][\LWRfootnote ]{{}^{\mathrm {#1}}}$ $\newcommand {\footnotemark }[1][\LWRfootnote ]{{}^{\mathrm {#1}}}$ $\let \LWRorighspace \hspace $ $\renewcommand {\hspace }{\ifstar \LWRorighspace \LWRorighspace }$ $\newcommand {\mathnormal }[1]{{#1}}$ $\newcommand \ensuremath [1]{#1}$ $\newcommand {\LWRframebox }[2][]{\fbox {#2}} \newcommand {\framebox }[1][]{\LWRframebox } $ $\newcommand {\setlength }[2]{}$ $\newcommand {\addtolength }[2]{}$ $\newcommand {\setcounter }[2]{}$ $\newcommand {\addtocounter }[2]{}$ $\newcommand {\arabic }[1]{}$ $\newcommand {\number }[1]{}$ $\newcommand {\noalign }[1]{\text {#1}\notag \\}$ $\newcommand {\cline }[1]{}$ $\newcommand {\directlua }[1]{\text {(directlua)}}$ $\newcommand {\luatexdirectlua }[1]{\text {(directlua)}}$ $\newcommand {\protect }{}$ $\def \LWRabsorbnumber #1 {}$ $\def \LWRabsorbquotenumber "#1 {}$ $\newcommand {\LWRabsorboption }[1][]{}$ $\newcommand {\LWRabsorbtwooptions }[1][]{\LWRabsorboption }$ $\def \mathchar {\ifnextchar "\LWRabsorbquotenumber \LWRabsorbnumber }$ $\def \mathcode #1={\mathchar }$ $\let \delcode \mathcode $ $\let \delimiter \mathchar $ $\def \oe {\unicode {x0153}}$ $\def \OE {\unicode {x0152}}$ $\def \ae {\unicode {x00E6}}$ $\def \AE {\unicode {x00C6}}$ $\def \aa {\unicode {x00E5}}$ $\def \AA {\unicode {x00C5}}$ $\def \o {\unicode {x00F8}}$ $\def \O {\unicode {x00D8}}$ $\def \l {\unicode {x0142}}$ $\def \L {\unicode {x0141}}$ $\def \ss {\unicode {x00DF}}$ $\def \SS {\unicode {x1E9E}}$ $\def \dag {\unicode {x2020}}$ $\def \ddag {\unicode {x2021}}$ $\def \P {\unicode {x00B6}}$ $\def \copyright {\unicode {x00A9}}$ $\def \pounds {\unicode {x00A3}}$ $\let \LWRref \ref $ $\renewcommand {\ref }{\ifstar \LWRref \LWRref }$ $ \newcommand {\multicolumn }[3]{#3}$ $\require {textcomp}$ $\newcommand {\intertext }[1]{\text {#1}\notag \\}$ $\let \Hat \hat $ $\let \Check \check $ $\let \Tilde \tilde $ $\let \Acute \acute $ $\let \Grave \grave $ $\let \Dot \dot $ $\let \Ddot \ddot $ $\let \Breve \breve $ $\let \Bar \bar $ $\let \Vec \vec $ $\require {colortbl}$ $\let \LWRorigcolumncolor \columncolor $ $\renewcommand {\columncolor }[2][named]{\LWRorigcolumncolor [#1]{#2}\LWRabsorbtwooptions }$ $\let \LWRorigrowcolor \rowcolor $ $\renewcommand {\rowcolor }[2][named]{\LWRorigrowcolor [#1]{#2}\LWRabsorbtwooptions }$ $\let \LWRorigcellcolor \cellcolor $ $\renewcommand {\cellcolor }[2][named]{\LWRorigcellcolor [#1]{#2}\LWRabsorbtwooptions }$ $\require {mathtools}$ $\newenvironment {crampedsubarray}[1]{}{}$ $\newcommand {\smashoperator }[2][]{#2\limits }$ $\newcommand {\SwapAboveDisplaySkip }{}$ $\newcommand {\LaTeXunderbrace }[1]{\underbrace {#1}}$ $\newcommand {\LaTeXoverbrace }[1]{\overbrace {#1}}$ $\newcommand {\LWRmultlined }[1][]{\begin {multline*}}$ $\newenvironment {multlined}[1][]{\LWRmultlined }{\end {multline*}}$ $\let \LWRorigshoveleft \shoveleft $ $\renewcommand {\shoveleft }[1][]{\LWRorigshoveleft }$ $\let \LWRorigshoveright \shoveright $ $\renewcommand {\shoveright }[1][]{\LWRorigshoveright }$ $\newcommand {\shortintertext }[1]{\text {#1}\notag \\}$ $\newcommand {\vcentcolon }{\mathrel {\unicode {x2236}}}$ $\renewcommand {\intertext }[2][]{\text {#2}\notag \\}$ $\newenvironment {fleqn}[1][]{}{}$ $\newenvironment {ceqn}{}{}$ $\newenvironment {darray}[2][c]{\begin {array}[#1]{#2}}{\end {array}}$ $\newcommand {\dmulticolumn }[3]{#3}$ $\newcommand {\LWRnrnostar }[1][0.5ex]{\\[#1]}$ $\newcommand {\nr }{\ifstar \LWRnrnostar \LWRnrnostar }$ $\newcommand {\mrel }[1]{\begin {aligned}#1\end {aligned}}$ $\newcommand {\underrel }[2]{\underset {#2}{#1}}$ $\newcommand {\medmath }[1]{#1}$ $\newcommand {\medop }[1]{#1}$ $\newcommand {\medint }[1]{#1}$ $\newcommand {\medintcorr }[1]{#1}$ $\newcommand {\mfrac }[2]{\frac {#1}{#2}}$ $\newcommand {\mbinom }[2]{\binom {#1}{#2}}$ $\newenvironment {mmatrix}{\begin {matrix}}{\end {matrix}}$ $\newcommand {\displaybreak }[1][]{}$ $ \def \offsyl {(\oslash )} \def \msconly {(\Delta )} $ $ \DeclareMathOperator {\var }{var} \DeclareMathOperator {\cov }{cov} \DeclareMathOperator {\Bin }{Bin} \DeclareMathOperator {\Geo }{Geometric} \DeclareMathOperator {\Beta }{Beta} \DeclareMathOperator {\Unif }{Uniform} \DeclareMathOperator {\Gam }{Gamma} \DeclareMathOperator {\Normal }{N} \DeclareMathOperator {\Exp }{Exp} \DeclareMathOperator {\Cauchy }{Cauchy} \DeclareMathOperator {\Bern }{Bernoulli} \DeclareMathOperator {\Poisson }{Poisson} \DeclareMathOperator {\Weibull }{Weibull} \DeclareMathOperator {\IGam }{IGamma} \DeclareMathOperator {\NGam }{NGamma} \DeclareMathOperator {\ChiSquared }{ChiSquared} \DeclareMathOperator {\Pareto }{Pareto} \DeclareMathOperator {\NBin }{NegBin} \DeclareMathOperator {\Studentt }{Student-t} \DeclareMathOperator *{\argmax }{arg\,max} \DeclareMathOperator *{\argmin }{arg\,min} $ \( \def \to {\rightarrow } \def \iff {\Leftrightarrow } \def \ra {\Rightarrow } \def \sw {\subseteq } \def \mc {\mathcal } \def \mb {\mathbb } \def \sc {\setminus } \def \wt {\widetilde } \def \v {\textbf } \def \E {\mb {E}} \def \P {\mb {P}} \def \R {\mb {R}} \def \C {\mb {C}} \def \N {\mb {N}} \def \Q {\mb {Q}} \def \Z {\mb {Z}} \def \B {\mb {B}} \def \~{\sim } \def \-{\,;\,} \def \qed {$\blacksquare $} \CustomizeMathJax {\def \1{\unicode {x1D7D9}}} \def \cadlag {c\`{a}dl\`{a}g} \def \p {\partial } \def \l {\left } \def \r {\right } \def \Om {\Omega } \def \om {\omega } \def \eps {\epsilon } \def \de {\delta } \def \ov {\overline } \def \sr {\stackrel } \def \Lp {\mc {L}^p} \def \Lq {\mc {L}^p} \def \Lone {\mc {L}^1} \def \Ltwo {\mc {L}^2} \def \toae {\sr {\rm a.e.}{\to }} \def \toas {\sr {\rm a.s.}{\to }} \def \top {\sr {\mb {\P }}{\to }} \def \tod {\sr {\rm d}{\to }} \def \toLp {\sr {\Lp }{\to }} \def \toLq {\sr {\Lq }{\to }} \def \eqae {\sr {\rm a.e.}{=}} \def \eqas {\sr {\rm a.s.}{=}} \def \eqd {\sr {\rm d}{=}} \def \approxd {\sr {\rm d}{\approx }} \def \Sa {(S1)\xspace } \def \Sb {(S2)\xspace } \def \Sc {(S3)\xspace } \)

5.4 Exercises on Chapter 5

5.1
- (a) $\color {blue}\star $ Let $X$ be the age in years of a person sampled uniformly at random from the UK population. Write down your best guess at $\P [X\geq x]$ for the values
  
  \[x=10,20,30,40,50,60,70,80,90.\]
- (b) Sketch the distribution that you obtained in (a) as a histogram. Does the histogram accurately represent your prior beliefs about the UK population? If not, make changes until you think it does.
- (c) In the solutions to this question you will find a table of these statistics, obtained in the UK Census 2021. For each value of $x$, calculate the value of $\frac {\text {your estimate}}{\text {census value}}$ (for example, if this value is $2$, your estimate was twice the true value). For which values of $x$ was your prior distribution least accurate?
5.2 $\color {blue}\star \,\star $ Let $\tau $ denote the maximum temperature that will occur outdoors in Sheffield tomorrow.
- (a) Sketch your prior density for $\tau $.
- (b)
  - (i) Perform the bisection method on yourself (or do this question with a friend) to elicit your 50th, 25th and 75th percentiles for $\tau $. Use the 25th and 75th percentiles to construct a Normal distribution $\Normal (\mu ,\sigma ^2)$ representing your beliefs. How close is $\mu $ to your 50th percentile?
    
    To help with this, the code used in Example 5.1.1 can be found within 5_elicitation_example.ipynb and 5_elicitation_example.Rmd.
  - (ii) Without using your answer to (i), write down your estimation of the 5th and 95th percentiles for $\tau $.
  - (iii) Compare your answers to (ii) to the implied probabilities of the distribution you found in (i). Is the Gaussian distribution a good fit for you beliefs?
- (c) Repeat part (b) using the Cauchy distribution instead of the Normal distribution. Which family of distributions better fits your beliefs?
5.3 $\color {blue}\star \,\star $ Let $(M_\lambda )_{\lambda \in (0,\infty )}$ be the Poisson model family, in which $M_\lambda \sim \Poisson (\lambda )$. Show that the reference prior of this model family is given by

\[f_{\Lambda }(\lambda )\propto \begin {cases} \lambda ^{-1/2} & \text { for }\lambda >0 \\ 0 & \text { otherwise.} \end {cases} \]

Does this define a proper prior or an improper prior?
5.4 $\color {blue}\star \,\star $ We will model the monthly occurrence of fires within a nameless small town using the $\Poisson (\lambda )$ model family. As very little is known prior to the data collection, we will use the reference prior $\Lambda $ obtained in Exercise 5.3.

The number of fires recorded during the past 12 months is $x=(x_1,\ldots ,x_{12})\in \{0,1,\ldots ,\}^{12}$.
- (a) Suppose that $x=(0,1,0,0,2,0,0,0,1,0,1,0)$. Find and sketch the distribution of the posterior $\Lambda |_{\{X=x\}}$.
- (b) Show that $\Lambda |_{\{X=x\}}$ is a proper distribution for all possible values of $x$.
5.5 $\color {blue}\star \,\star $ Let $(X,\Theta )$ be a Bayesian model with model family $M_\theta \sim \Unif (0,\theta )$, the continuous uniform distribution on $(0,\theta )$.
- (a) Given the prior $\Theta \sim \Exp (1)$, find the posterior density function $f_{\Theta |_{\{X=x\}}}(\theta )$.
  
  You should discover that $f_{\Theta |_{\{X=x\}}}(\theta )=0$ for $\theta <x$. Can you explain (without reference to your calculations) why this has happened?
- (b) Instead, let us now use the prior $\Theta \sim \Unif (1,2)$, and suppose that our data is $x=3$. Is the posterior distribution well defined? What has gone wrong here?
5.6 $\color {blue}\star \star \star $ We say that a random variable $U$ is uniformly distributed on an interval $I$ if, for all $a<b$ and $c>0$ such that both $[a,b]\sw I$ and $[a+c,b+c]\sw I$, we have $\P [U\in [a,b]]=\P [U\in [a+c,b+c]]$.

Show that there is no random variable $U$ that is uniformly distributed on $[0,\infty )$.
5.7 $\color {blue}\star \star \star $ Prove the claim in Remark 5.3.1. You should start by finding an expression for the p.d.f. of $h(\Theta )$, where $\Theta $ has p.d.f. $f_1$.
5.8 $\color {blue}\star \star \star $ Let $(M_\theta )$ be a family of distributions and let $n\in \N $. Let $f_{M_\theta }(\theta )$ denote its reference prior and let $f_{M_\theta ^{\otimes n}}(\theta )$ denote the reference prior of the family $(M_\theta ^{\otimes n})$. Show that $f_{M_\theta }(\theta )\propto f_{M_\theta ^{\otimes n}}(\theta )$.