last updated: October 24, 2024

Bayesian Statistics

\(\newcommand{\footnotename}{footnote}\) \(\def \LWRfootnote {1}\) \(\newcommand {\footnote }[2][\LWRfootnote ]{{}^{\mathrm {#1}}}\) \(\newcommand {\footnotemark }[1][\LWRfootnote ]{{}^{\mathrm {#1}}}\) \(\let \LWRorighspace \hspace \) \(\renewcommand {\hspace }{\ifstar \LWRorighspace \LWRorighspace }\) \(\newcommand {\mathnormal }[1]{{#1}}\) \(\newcommand \ensuremath [1]{#1}\) \(\newcommand {\LWRframebox }[2][]{\fbox {#2}} \newcommand {\framebox }[1][]{\LWRframebox } \) \(\newcommand {\setlength }[2]{}\) \(\newcommand {\addtolength }[2]{}\) \(\newcommand {\setcounter }[2]{}\) \(\newcommand {\addtocounter }[2]{}\) \(\newcommand {\arabic }[1]{}\) \(\newcommand {\number }[1]{}\) \(\newcommand {\noalign }[1]{\text {#1}\notag \\}\) \(\newcommand {\cline }[1]{}\) \(\newcommand {\directlua }[1]{\text {(directlua)}}\) \(\newcommand {\luatexdirectlua }[1]{\text {(directlua)}}\) \(\newcommand {\protect }{}\) \(\def \LWRabsorbnumber #1 {}\) \(\def \LWRabsorbquotenumber "#1 {}\) \(\newcommand {\LWRabsorboption }[1][]{}\) \(\newcommand {\LWRabsorbtwooptions }[1][]{\LWRabsorboption }\) \(\def \mathchar {\ifnextchar "\LWRabsorbquotenumber \LWRabsorbnumber }\) \(\def \mathcode #1={\mathchar }\) \(\let \delcode \mathcode \) \(\let \delimiter \mathchar \) \(\def \oe {\unicode {x0153}}\) \(\def \OE {\unicode {x0152}}\) \(\def \ae {\unicode {x00E6}}\) \(\def \AE {\unicode {x00C6}}\) \(\def \aa {\unicode {x00E5}}\) \(\def \AA {\unicode {x00C5}}\) \(\def \o {\unicode {x00F8}}\) \(\def \O {\unicode {x00D8}}\) \(\def \l {\unicode {x0142}}\) \(\def \L {\unicode {x0141}}\) \(\def \ss {\unicode {x00DF}}\) \(\def \SS {\unicode {x1E9E}}\) \(\def \dag {\unicode {x2020}}\) \(\def \ddag {\unicode {x2021}}\) \(\def \P {\unicode {x00B6}}\) \(\def \copyright {\unicode {x00A9}}\) \(\def \pounds {\unicode {x00A3}}\) \(\let \LWRref \ref \) \(\renewcommand {\ref }{\ifstar \LWRref \LWRref }\) \( \newcommand {\multicolumn }[3]{#3}\) \(\require {textcomp}\) \(\newcommand {\intertext }[1]{\text {#1}\notag \\}\) \(\let \Hat \hat \) \(\let \Check \check \) \(\let \Tilde \tilde \) \(\let \Acute \acute \) \(\let \Grave \grave \) \(\let \Dot \dot \) \(\let \Ddot \ddot \) \(\let \Breve \breve \) \(\let \Bar \bar \) \(\let \Vec \vec \) \(\require {colortbl}\) \(\let \LWRorigcolumncolor \columncolor \) \(\renewcommand {\columncolor }[2][named]{\LWRorigcolumncolor [#1]{#2}\LWRabsorbtwooptions }\) \(\let \LWRorigrowcolor \rowcolor \) \(\renewcommand {\rowcolor }[2][named]{\LWRorigrowcolor [#1]{#2}\LWRabsorbtwooptions }\) \(\let \LWRorigcellcolor \cellcolor \) \(\renewcommand {\cellcolor }[2][named]{\LWRorigcellcolor [#1]{#2}\LWRabsorbtwooptions }\) \(\require {mathtools}\) \(\newenvironment {crampedsubarray}[1]{}{}\) \(\newcommand {\smashoperator }[2][]{#2\limits }\) \(\newcommand {\SwapAboveDisplaySkip }{}\) \(\newcommand {\LaTeXunderbrace }[1]{\underbrace {#1}}\) \(\newcommand {\LaTeXoverbrace }[1]{\overbrace {#1}}\) \(\newcommand {\LWRmultlined }[1][]{\begin {multline*}}\) \(\newenvironment {multlined}[1][]{\LWRmultlined }{\end {multline*}}\) \(\let \LWRorigshoveleft \shoveleft \) \(\renewcommand {\shoveleft }[1][]{\LWRorigshoveleft }\) \(\let \LWRorigshoveright \shoveright \) \(\renewcommand {\shoveright }[1][]{\LWRorigshoveright }\) \(\newcommand {\shortintertext }[1]{\text {#1}\notag \\}\) \(\newcommand {\vcentcolon }{\mathrel {\unicode {x2236}}}\) \(\renewcommand {\intertext }[2][]{\text {#2}\notag \\}\) \(\newenvironment {fleqn}[1][]{}{}\) \(\newenvironment {ceqn}{}{}\) \(\newenvironment {darray}[2][c]{\begin {array}[#1]{#2}}{\end {array}}\) \(\newcommand {\dmulticolumn }[3]{#3}\) \(\newcommand {\LWRnrnostar }[1][0.5ex]{\\[#1]}\) \(\newcommand {\nr }{\ifstar \LWRnrnostar \LWRnrnostar }\) \(\newcommand {\mrel }[1]{\begin {aligned}#1\end {aligned}}\) \(\newcommand {\underrel }[2]{\underset {#2}{#1}}\) \(\newcommand {\medmath }[1]{#1}\) \(\newcommand {\medop }[1]{#1}\) \(\newcommand {\medint }[1]{#1}\) \(\newcommand {\medintcorr }[1]{#1}\) \(\newcommand {\mfrac }[2]{\frac {#1}{#2}}\) \(\newcommand {\mbinom }[2]{\binom {#1}{#2}}\) \(\newenvironment {mmatrix}{\begin {matrix}}{\end {matrix}}\) \(\newcommand {\displaybreak }[1][]{}\) \( \def \offsyl {(\oslash )} \def \msconly {(\Delta )} \) \( \DeclareMathOperator {\var }{var} \DeclareMathOperator {\cov }{cov} \DeclareMathOperator {\Bin }{Bin} \DeclareMathOperator {\Geo }{Geometric} \DeclareMathOperator {\Beta }{Beta} \DeclareMathOperator {\Unif }{Uniform} \DeclareMathOperator {\Gam }{Gamma} \DeclareMathOperator {\Normal }{N} \DeclareMathOperator {\Exp }{Exp} \DeclareMathOperator {\Cauchy }{Cauchy} \DeclareMathOperator {\Bern }{Bernoulli} \DeclareMathOperator {\Poisson }{Poisson} \DeclareMathOperator {\Weibull }{Weibull} \DeclareMathOperator {\IGam }{IGamma} \DeclareMathOperator {\NGam }{NGamma} \DeclareMathOperator {\ChiSquared }{ChiSquared} \DeclareMathOperator {\Pareto }{Pareto} \DeclareMathOperator {\NBin }{NegBin} \DeclareMathOperator {\Studentt }{Student-t} \DeclareMathOperator *{\argmax }{arg\,max} \DeclareMathOperator *{\argmin }{arg\,min} \) \( \def \to {\rightarrow } \def \iff {\Leftrightarrow } \def \ra {\Rightarrow } \def \sw {\subseteq } \def \mc {\mathcal } \def \mb {\mathbb } \def \sc {\setminus } \def \wt {\widetilde } \def \v {\textbf } \def \E {\mb {E}} \def \P {\mb {P}} \def \R {\mb {R}} \def \C {\mb {C}} \def \N {\mb {N}} \def \Q {\mb {Q}} \def \Z {\mb {Z}} \def \B {\mb {B}} \def \~{\sim } \def \-{\,;\,} \def \qed {$\blacksquare $} \CustomizeMathJax {\def \1{\unicode {x1D7D9}}} \def \cadlag {c\`{a}dl\`{a}g} \def \p {\partial } \def \l {\left } \def \r {\right } \def \Om {\Omega } \def \om {\omega } \def \eps {\epsilon } \def \de {\delta } \def \ov {\overline } \def \sr {\stackrel } \def \Lp {\mc {L}^p} \def \Lq {\mc {L}^p} \def \Lone {\mc {L}^1} \def \Ltwo {\mc {L}^2} \def \toae {\sr {\rm a.e.}{\to }} \def \toas {\sr {\rm a.s.}{\to }} \def \top {\sr {\mb {\P }}{\to }} \def \tod {\sr {\rm d}{\to }} \def \toLp {\sr {\Lp }{\to }} \def \toLq {\sr {\Lq }{\to }} \def \eqae {\sr {\rm a.e.}{=}} \def \eqas {\sr {\rm a.s.}{=}} \def \eqd {\sr {\rm d}{=}} \def \approxd {\sr {\rm d}{\approx }} \def \Sa {(S1)\xspace } \def \Sb {(S2)\xspace } \def \Sc {(S3)\xspace } \)

1.8 Exercises on Chapter 1

  • 1.1

    • (a) \(\color {blue}\star \) Consider a continuous random variable \(X\) with range \(R_X=(-10,10)\) and p.d.f. \(f_X\) sketched as follows:

      (image)

      In words, explain which parts of \(R_X\) are the most likely locations for a random sample of \(X\) to be.

    • (b) Let \(\theta >1\). Check that the function \(f:\R \to [0,\infty )\) by \(f(x) =\Big \{\begin {smallmatrix} (\theta -1)x^{-\theta } & \text { for } x\geq 1 \hfill \\ 0 & \text { otherwise} \hfill \end {smallmatrix}\) is a probability density function i.e. check that integrating over its range gives \(1\).

  • 1.2 \(\color {blue}\star \) Let \(X\sim \Normal (0,1)\) and let \(Y\) be given by \(\P [Y=0]=\P [Y=1]=\frac 12\), independently of \(Z\). Define a random variable

    \[Z=\begin {cases} 0 & \text { if }Y=0, \\ X & \text { if }Y=1. \\ \end {cases} \]

    Is \(Z\) a discrete random variable, a continuous random variable, or neither?

  • 1.3 \(\color {blue}\star \,\star \) Let \(a<b<c\), all elements of \(\R \). Suppose that \(U\) has the continuous uniform distribution on \([a,c]\) and let \(U'\eqd U|_{\{U\in [a,b]\}}\).

    • (a) Write out the meaning of \(U'\eqd U|_{\{U\in [a,b]\}}\), in words.

    • (b) Show that \(U'\) has the continuous uniform distribution on \([a,b]\).

    • (c) Plot the probability density functions of \(U\) and \(U'\) on the same axis, for the case \(a=1,b=2,c=4\).

  • 1.4 \(\color {blue}\star \,\star \) Suppose that \(G\) has the \(\Geo (p)\) distribution, that is \(\P [G=g]=p^{g}(1-p)\) for \(g\in \{0,1,\ldots \}\), where \(p\in [0,1]\). Let \(G'\eqd G|_{\{G\geq n\}}\), where \(n\in \N \).

    • (a) Find the distribution of \(G'\).

    • (b) Consider the following claim.

      Recall that \(G\) represents the time of the first failure in a sequence of independent trials, each of which has failure probability \(p\). Conditioning \(G\) to be greater than or equal to \(n\) has the effect of forcing the first \(n\) trials to be successful, without changing the distribution of the remaining trials.

      Do you agree with this statement? Give brief reasons for your answer.

  • 1.5 \(\color {blue}\star \) Let \(X\sim \Cauchy (0,1)\) and let \(x\in \R \). In the files 1_rejection_sampling.ipynb and 1_rejection_sampling.Rmd you can find a rejection sampling algorithm (as described in our comments below Lemma 1.4.1 or within the proof of that lemma) that obtains samples from \(X|_{\{X\geq k\}}\).

    Record the time it takes for the code to obtain some samples, for \(k=2^n\) for various values of \(n\in \{1,2,\ldots \}\). What do you notice? Repeat the experiment with some other distributions.

  • 1.6 \(\color {blue}\star \,\star \) Let \((X,Y)\) be a discrete random variable with values in \(\R ^2\) (so \(n=d=1\)) with distribution given by

    \begin{equation} \label {eq:cond_corr_ps} \P [(X,Y)=(x,y)]=\begin{cases} 2^{-xy}(1-2^{-y}) & \text { for }x,y\in \N \\ 0 & \text { otherwise.} \end {cases} \end{equation}

    Here is a plot of this function, with shading to indicate regions for which \(x\geq 5\) and/or \(y=5\).

    (image)

    • (a) Check that the function given in (1.14) is a probability mass function with range \(\N ^2\) i.e. check that summing over its range gives \(1\). Are \(X\) and \(Y\) independent?

    • (b)

      • (i) Show that for \(y\in \N \) we have \(\P [Y=y]=\l (\frac 12\r )^y\).

      • (ii) Show that for \(x\in \N \) we have \(\P [X|_{\{Y=5\}}=x]=\l (1-\frac {1}{2^5}\r )\l (\frac {1}{2^5}\r )^{x-1}.\)

      • (iii) Show that for \(y\in \N \) we have \(\P [Y|_{\{X\geq 5\}}=y]=\l (1-\frac {1}{2^5}\r )\l (\frac {1}{2^5}\r )^{y-1}.\)

  • 1.7 \(\color {blue}\star \,\star \) Let \(X\sim \Normal (0,1)\) and set \(A=[0,\infty )\), as in Example 1.4.3. Let \(Y'=|X|\). Show that \(Y'\eqd X|_{\{X\in A\}}\).

  • 1.8 \(\color {blue}\star \star \star \) Let \(X\) be a continuous random variable and let \(A\sw R_X\) with \(\P [X\in A]>0\). Show that \(X|_{\{X\in A\}}\) is a continuous random variable with p.d.f.

    \[f_{X|_{\{X\in A\}}}(x)= \begin {cases} \frac {f_X(x)}{\P [X\in A]} & \text { if } x\in A \\ 0 & \text {otherwise.} \end {cases} \]

  • 1.9 \(\color {blue}\star \star \star \) Prove Lemma 1.2.3.

  • 1.10 \(\color {blue}\star \star \star \) Suppose that \(X\) and \(Y\) are independent random variables, and that \(\P [X\in A]>0\). Show that \(Y\eqd Y|_{\{X\in A\}}\).