Bayesian Statistics

$\newcommand{\footnotename}{footnote}$ $\def \LWRfootnote {1}$ $\newcommand {\footnote }[2][\LWRfootnote ]{{}^{\mathrm {#1}}}$ $\newcommand {\footnotemark }[1][\LWRfootnote ]{{}^{\mathrm {#1}}}$ $\let \LWRorighspace \hspace $ $\renewcommand {\hspace }{\ifstar \LWRorighspace \LWRorighspace }$ $\newcommand {\mathnormal }[1]{{#1}}$ $\newcommand \ensuremath [1]{#1}$ $\newcommand {\LWRframebox }[2][]{\fbox {#2}} \newcommand {\framebox }[1][]{\LWRframebox } $ $\newcommand {\setlength }[2]{}$ $\newcommand {\addtolength }[2]{}$ $\newcommand {\setcounter }[2]{}$ $\newcommand {\addtocounter }[2]{}$ $\newcommand {\arabic }[1]{}$ $\newcommand {\number }[1]{}$ $\newcommand {\noalign }[1]{\text {#1}\notag \\}$ $\newcommand {\cline }[1]{}$ $\newcommand {\directlua }[1]{\text {(directlua)}}$ $\newcommand {\luatexdirectlua }[1]{\text {(directlua)}}$ $\newcommand {\protect }{}$ $\def \LWRabsorbnumber #1 {}$ $\def \LWRabsorbquotenumber "#1 {}$ $\newcommand {\LWRabsorboption }[1][]{}$ $\newcommand {\LWRabsorbtwooptions }[1][]{\LWRabsorboption }$ $\def \mathchar {\ifnextchar "\LWRabsorbquotenumber \LWRabsorbnumber }$ $\def \mathcode #1={\mathchar }$ $\let \delcode \mathcode $ $\let \delimiter \mathchar $ $\def \oe {\unicode {x0153}}$ $\def \OE {\unicode {x0152}}$ $\def \ae {\unicode {x00E6}}$ $\def \AE {\unicode {x00C6}}$ $\def \aa {\unicode {x00E5}}$ $\def \AA {\unicode {x00C5}}$ $\def \o {\unicode {x00F8}}$ $\def \O {\unicode {x00D8}}$ $\def \l {\unicode {x0142}}$ $\def \L {\unicode {x0141}}$ $\def \ss {\unicode {x00DF}}$ $\def \SS {\unicode {x1E9E}}$ $\def \dag {\unicode {x2020}}$ $\def \ddag {\unicode {x2021}}$ $\def \P {\unicode {x00B6}}$ $\def \copyright {\unicode {x00A9}}$ $\def \pounds {\unicode {x00A3}}$ $\let \LWRref \ref $ $\renewcommand {\ref }{\ifstar \LWRref \LWRref }$ $ \newcommand {\multicolumn }[3]{#3}$ $\require {textcomp}$ $\newcommand {\intertext }[1]{\text {#1}\notag \\}$ $\let \Hat \hat $ $\let \Check \check $ $\let \Tilde \tilde $ $\let \Acute \acute $ $\let \Grave \grave $ $\let \Dot \dot $ $\let \Ddot \ddot $ $\let \Breve \breve $ $\let \Bar \bar $ $\let \Vec \vec $ $\require {colortbl}$ $\let \LWRorigcolumncolor \columncolor $ $\renewcommand {\columncolor }[2][named]{\LWRorigcolumncolor [#1]{#2}\LWRabsorbtwooptions }$ $\let \LWRorigrowcolor \rowcolor $ $\renewcommand {\rowcolor }[2][named]{\LWRorigrowcolor [#1]{#2}\LWRabsorbtwooptions }$ $\let \LWRorigcellcolor \cellcolor $ $\renewcommand {\cellcolor }[2][named]{\LWRorigcellcolor [#1]{#2}\LWRabsorbtwooptions }$ $\require {mathtools}$ $\newenvironment {crampedsubarray}[1]{}{}$ $\newcommand {\smashoperator }[2][]{#2\limits }$ $\newcommand {\SwapAboveDisplaySkip }{}$ $\newcommand {\LaTeXunderbrace }[1]{\underbrace {#1}}$ $\newcommand {\LaTeXoverbrace }[1]{\overbrace {#1}}$ $\newcommand {\LWRmultlined }[1][]{\begin {multline*}}$ $\newenvironment {multlined}[1][]{\LWRmultlined }{\end {multline*}}$ $\let \LWRorigshoveleft \shoveleft $ $\renewcommand {\shoveleft }[1][]{\LWRorigshoveleft }$ $\let \LWRorigshoveright \shoveright $ $\renewcommand {\shoveright }[1][]{\LWRorigshoveright }$ $\newcommand {\shortintertext }[1]{\text {#1}\notag \\}$ $\newcommand {\vcentcolon }{\mathrel {\unicode {x2236}}}$ $\renewcommand {\intertext }[2][]{\text {#2}\notag \\}$ $\newenvironment {fleqn}[1][]{}{}$ $\newenvironment {ceqn}{}{}$ $\newenvironment {darray}[2][c]{\begin {array}[#1]{#2}}{\end {array}}$ $\newcommand {\dmulticolumn }[3]{#3}$ $\newcommand {\LWRnrnostar }[1][0.5ex]{\\[#1]}$ $\newcommand {\nr }{\ifstar \LWRnrnostar \LWRnrnostar }$ $\newcommand {\mrel }[1]{\begin {aligned}#1\end {aligned}}$ $\newcommand {\underrel }[2]{\underset {#2}{#1}}$ $\newcommand {\medmath }[1]{#1}$ $\newcommand {\medop }[1]{#1}$ $\newcommand {\medint }[1]{#1}$ $\newcommand {\medintcorr }[1]{#1}$ $\newcommand {\mfrac }[2]{\frac {#1}{#2}}$ $\newcommand {\mbinom }[2]{\binom {#1}{#2}}$ $\newenvironment {mmatrix}{\begin {matrix}}{\end {matrix}}$ $\newcommand {\displaybreak }[1][]{}$ $ \def \offsyl {(\oslash )} \def \msconly {(\Delta )} $ $ \DeclareMathOperator {\var }{var} \DeclareMathOperator {\cov }{cov} \DeclareMathOperator {\Bin }{Bin} \DeclareMathOperator {\Geo }{Geometric} \DeclareMathOperator {\Beta }{Beta} \DeclareMathOperator {\Unif }{Uniform} \DeclareMathOperator {\Gam }{Gamma} \DeclareMathOperator {\Normal }{N} \DeclareMathOperator {\Exp }{Exp} \DeclareMathOperator {\Cauchy }{Cauchy} \DeclareMathOperator {\Bern }{Bernoulli} \DeclareMathOperator {\Poisson }{Poisson} \DeclareMathOperator {\Weibull }{Weibull} \DeclareMathOperator {\IGam }{IGamma} \DeclareMathOperator {\NGam }{NGamma} \DeclareMathOperator {\ChiSquared }{ChiSquared} \DeclareMathOperator {\Pareto }{Pareto} \DeclareMathOperator {\NBin }{NegBin} \DeclareMathOperator {\Studentt }{Student-t} \DeclareMathOperator *{\argmax }{arg\,max} \DeclareMathOperator *{\argmin }{arg\,min} $ \( \def \to {\rightarrow } \def \iff {\Leftrightarrow } \def \ra {\Rightarrow } \def \sw {\subseteq } \def \mc {\mathcal } \def \mb {\mathbb } \def \sc {\setminus } \def \wt {\widetilde } \def \v {\textbf } \def \E {\mb {E}} \def \P {\mb {P}} \def \R {\mb {R}} \def \C {\mb {C}} \def \N {\mb {N}} \def \Q {\mb {Q}} \def \Z {\mb {Z}} \def \B {\mb {B}} \def \~{\sim } \def \-{\,;\,} \def \qed {$\blacksquare $} \CustomizeMathJax {\def \1{\unicode {x1D7D9}}} \def \cadlag {c\`{a}dl\`{a}g} \def \p {\partial } \def \l {\left } \def \r {\right } \def \Om {\Omega } \def \om {\omega } \def \eps {\epsilon } \def \de {\delta } \def \ov {\overline } \def \sr {\stackrel } \def \Lp {\mc {L}^p} \def \Lq {\mc {L}^p} \def \Lone {\mc {L}^1} \def \Ltwo {\mc {L}^2} \def \toae {\sr {\rm a.e.}{\to }} \def \toas {\sr {\rm a.s.}{\to }} \def \top {\sr {\mb {\P }}{\to }} \def \tod {\sr {\rm d}{\to }} \def \toLp {\sr {\Lp }{\to }} \def \toLq {\sr {\Lq }{\to }} \def \eqae {\sr {\rm a.e.}{=}} \def \eqas {\sr {\rm a.s.}{=}} \def \eqd {\sr {\rm d}{=}} \def \approxd {\sr {\rm d}{\approx }} \def \Sa {(S1)\xspace } \def \Sb {(S2)\xspace } \def \Sc {(S3)\xspace } \)

4.7 Exercises on Chapter 4

You can find formulae for named distributions in Appendix A.

4.1
- (a) $\color {blue}\star $ Consider the Bayesian model $(X,\Theta )$ with model family $M_\theta \sim \Normal (\theta ,2^2)^{\otimes 3}$ and prior $\Theta \sim \Normal (0,1)$.
  - (i) Use Lemma 4.2.2 to find the posterior distribution given the data $x=(3.88,\ 2.34,\ 7.86)$, which satisfies $\sum _1^3 x_i=14.08$.
  - (ii) Write down the probability density functions of the sampling and predictive distributions given by this model for a single data point. Give your answers in the form $\int _\R f_{\Normal (\cdot ,\cdot )}(\cdot )f_{\Normal (\cdot ,\cdot )}(\cdot )\,d(\cdot )$.
- (b) Inside the files 2_dist_sketching.ipynb and 2_dist_sketching.Rmd, below the part corresponding to Exercise 3.1, you will find code for sketching the sampling p.d.f. of the Bayesian model $(X,\Theta )$ from Example 3.2.3, with model family $M_\theta \sim \Exp (\lambda )$ and prior $\Lambda \sim \Gamma (2,60)$. Modify the code given to sketch the sampling and predictive distributions from (a), on the same graph.
4.2 $\color {blue}\star \,\star $ Let $\alpha ,\beta >0$. Let $(X,\Theta )$ be a discrete Bayesian model with model family $M_\theta \sim \Geo (\theta )^{\otimes n}$ and parameter $\theta \in [0,1]$. Suppose that the prior is $\Theta \sim \Beta (\alpha ,\beta )$ and let $x=(x_1,\ldots ,x_n)$ where $x_i\in \{0,1,\ldots ,\}$. Show that the posterior is

\[\Theta |_{\{X=x\}}\sim \Beta \l (\alpha +n,\, \beta +\sum _{i=1}^n x_i\r ).\]
4.3 $\color {blue}\star \,\star $ Let $\alpha ,\beta >0$. Let $(X,\Theta )$ be a discrete Bayesian model with model family $M_\theta \sim \Poisson (\theta )^{\otimes n}$ and parameter $\theta \in (0,\infty )$. Suppose that the prior is $\Theta \sim \Gam (\alpha ,\beta )$ and let $x=(x_1,\ldots ,x_n)$ where $x_i\in \{0,1,\ldots ,\}$. Show that the posterior is

\[\Theta |_{\{X=x\}}\sim \Gam \l (\alpha +\sum _{i=1}^n x_i,\, \beta +n\r ).\]
4.4 $\color {blue}\star \,\star $ Let $\mu \in \R $ and $\alpha ,\beta >0$. Let $(X,T)$ be a discrete Bayesian model with model family $M_\theta \sim \Normal (\mu ,\frac {1}{\tau })^{\otimes n}$ and parameter $\tau \in (0,\infty )$. Suppose that the prior is $T\sim \Gam (\alpha ,\beta )$ and let $x=(x_1,\ldots ,x_n)$ where $x_i\in (0,\infty )$. Show that the posterior is

\[T|_{\{X=x\}}\sim \Gam \l (\alpha +\frac {n}{2},\, \beta +\frac 12\sum _{i=1}^n(x_i-\mu )^2\r ).\]
4.5 $\color {blue}\star $ Using the model from Exercise 4.4, with $\mu =0$ and prior $\Gam (2,2)$, use a computer package of your choice to produce a graph of the prior and posterior density functions, given the data

\[x=(0.22,\ -0.17,\ 1.22,\ -0.13,\ 0.05,\ 0.79,\ -0.45,\ -0.30,\ 0.09,\ -0.16).\]

This data satisfies $\sum _1^{10} x_i^2=2.53$. Then, draw a second graph of the sampling and predictive density functions for a single data point.
4.6 Let $u\in \R $. Let $(X,\Theta )$ be a continuous Bayesian model with model family $M_{\theta }\sim \Normal (\theta ,2^2)^{\otimes 10}$ and parameter $\theta \in \R $. Suppose that the prior is $\Theta \sim \Normal (0,2^2)$, and that we have data

\[x=(5.29,\ 1.20,\ 2.94,\ 6.72,\ 5.60,\ -2.93,\ 2.85,\ -0.45,\ -0.31,\ 1.23).\]

This data satisfies $\sum _1^{10} x_i= 133.19$.
- (a) $\color {blue}\star $ Use Lemma 4.2.2 to find the posterior distribution $\Theta |_{\{X=x\}}$.
- (b) $\color {blue}\star $ Using a computer package of your choice, implement the Bayesian update given by Lemma 4.2.2 with $n=1$, and use it to perform $10$ Bayesian update steps, one for each $x_i$, on the model $(X',\Theta )$ with model family $M_{\theta }\sim \Normal (\theta ,2)$. Write down the resulting posterior distribution.
- (c) $\color {blue}\star \,\star $ What do you notice? Investigate this as you vary the data and prior parameters. Would the same thing happen with the other families of conjugate pairs in this chapter?
4.7 $\color {blue}\star \,\star $ Let $a,b,\beta >0$. Let $(X,\Theta )$ be a discrete Bayesian model with model family $M_\theta \sim \Weibull (\theta ,\beta )^{\otimes n}$ and parameter $\theta \in (0,\infty )$. Suppose that the prior is $\Theta \sim \IGam (a,b)$ and let $x=(x_1,\ldots ,x_n)$ where $x_i\in \{0,1,\ldots ,\}$. Show that the posterior is

\[\Theta |_{\{X=x\}}\sim \IGam \l (a+n,\, b+\sum _{i=1}^nx_i^\beta \r ).\]
4.8 $\color {blue}\star $ Match each of Lemmas 4.1.5, 4.1.6, 4.2.1, 4.5.2 and Exercises 4.2, 4.3, 4.4, 4.7 to their corresponding rows on the reference sheet of conjugate pairs given in Appendix A
4.9 $\color {blue}\star $ Recall the relation $\propto $ from Definition 4.1.1. Let $f,g,h$ be functions with the same domain. Explain briefly why (a) $f\propto f$; (b) if $f\propto g$ then $g\propto f$; (c) if $f\propto g$ and $g\propto h$ then $f\propto h$.

$\offsyl $ Remark: These three properties are the definition of an equivalence relation.