Bayesian Statistics
1.2 Equality in distribution
Let \(X\) be a random variable taking values in \(\R ^d\). The law or distribution of \(X\) is the function \(A\mapsto \P [X\in A]\), which tells us how likely the value of \(X\) is to be within the set \(A\sw \R ^d\).
In the case \(d=1\) we also have the cumulative distribution function \(F_X(x)=\P [X\leq x]\) which tells us how likely the value of \(X\) is to be less than or equal to \(x\in \R \). For random variables \(X\) and \(Y\) taking values in \(\R \),
\(\seteqnumber{0}{1.}{1}\)\begin{equation} \label {eq:cdf_df_d=1} F_X=F_Y\text { if and only if }X\eqd Y. \end{equation}
We won’t prove (1.2) within this course, although it is hopefully not surprising to you.
-
Example 1.2.2 It is important to understand that Definition 1.2.1 is not the same thing as equality. For example, let \(X\sim \Normal (0,1)\) and let \(Y=-X\). Then
\[\P [Y\leq x] =\P [-x\leq X] =\int _{-x}^\infty \frac {1}{\sqrt {2\pi }}e^{-y^2/2}\,dy =\int _{-\infty }^x \frac {1}{\sqrt {2\pi }}e^{-z^2/2}\,dz =\P [X\leq x], \]
where we have made the substitution \(z=-y\). Hence \(F_X=F_Y\) so from (1.2) we have \(X\eqd Y\). But \(X=Y\) only happens when \(X=Y=0\), which has probability zero.
A perhaps simpler example: if \(X\) and \(Y\) are independent \(\Normal (0,1)\) random variables then \(X\eqd Y\), but \(\P [X=Y]=\P [X-Y=0]\) and \(X-Y\sim \Normal (0,1+1)\eqd \Normal (0,2)\) so \(\P [X=Y]=0\).
Note that we have used the notation \(\sim \Normal (0,1)\) in Example 1.2.2. We might wonder what the difference between the symbols \(\sim \) and \(\eqd \) is. Formally, they have the same meaning, but we tend to use \(\sim \) when we are referring to a named distribution, and \(\eqd \) when we are comparing two existing random variables. That is a convention and not a rule, so you can use \(\sim \) and \(\eqd \) interchangeably if you wish.
1.2.1 Identifying distributions
In this section we give some results that help to identify the relationship \(X\eqd Y\). Note that this also helps us identify when random variables have named distributions. The discrete case is easily dealt with.
Note that the statement \(p_X=p_Y\) means that the functions \(p_X\) and \(p_Y\) are equal, that is \(p_X(x)=p_Y(x)\) for all \(x\in \R ^d\). The proof is left for you in Problem 1.9.
The situation for continuous random variables is a bit more complicated. If \(X\) and \(Y\) are continuous random variables with \(f_X=f_Y\), then it is clear from (1.1) that \(X\eqd Y\), but it is possible to have \(X\eqd Y\) and for \(f_X\) and \(f_Y\) to be ‘different in an unimportant way’. You should have already seen examples of this situation, like the following.
-
Example 1.2.4 The probability density functions
\[ f_X(x)= \begin {cases} 1 & \text { for }x\in (0,1) \\ 0 & \text { otherwise,} \end {cases} \qquad f_Y(y)= \begin {cases} 1 & \text { for }y\in [0,1] \\ 0 & \text { otherwise,} \end {cases} \]
define random variables \(X\) and \(Y\). Note that \(f_X(x)=f_Y(x)\) for all \(x\in \R \) except for \(x=0\) and \(x=1\), so \(f_X\) and \(f_Y\) are different, but only very slightly! You might think of these as the continuous uniform distributions \(X\sim \Unif ((0,1))\) and \(Y\sim \Unif ([0,1])\), but they are really the same distribution because \(\P [X=0]=\P [X=1]=\P [Y=0]=\P [Y=1]=0\).
We need to handle this point carefully because, in this course, we don’t assume enough mathematical background to explain precisely what we mean by ‘different in an unimportant way’. We do need to know the following facts, however:
-
• For a random variable \(X\) with range in \(\R \), changing the value of \(f_X(x)\) on a finite set of \(x\in \R \) will not change the distribution of \(X\) (as in Example 1.2.4).
-
• For a random variable \(X\) with range in \(\R ^2\), the same is true, but we can also change the value of \(f_X(x)\) on a finite set of lines (in \(\R ^2\)) without changing the distribution of \(X\).
Similar things work in higher dimensions too, but we won’t need those.
We sometimes think of random variables as being defined by probability mass functions or probability density functions. This is a slight abuse of terminology: as we have discussed above, the p.m.f. and p.d.f. specify the distribution. If you are asked to ‘find’ the random variable \(X\), or to find the distribution of \(X\), then a statement of the p.m.f. or p.d.f. will suffice. You should always specify the range of values for which the p.m.f. of p.d.f. is non-zero.
1.2.2 Normalizing constants
Often you will find that the p.m.f. or p.d.f of some random variable \(X\) appears in the form
\(\seteqnumber{0}{1.}{2}\)\begin{equation} \label {eq:normalizing_constants} \P [X=x]=\frac {1}{Z}g(x)\qquad \text { or }\qquad f_X(x)=\frac {1}{Z}g(x) \end{equation}
where \(Z\) does not depend on \(x\). In such cases \(Z\) is known as a normalizing constant. Its role is to make sure that p.m.f. sums (over \(x\in R_X\)) to one, and the p.d.f. integrates (again, over \(x\in R_X\)) to one. We have written \(\frac {1}{Z}\) because normalizing constants often appear in a denominator e.g. \(\frac {1}{\sqrt {2\pi }}\) in \(f_{\Normal (0,1)}(x)=\frac {1}{\sqrt {2}\pi }e^{-x^2/2}\), but they don’t have to appear that way up e.g. \(\lambda \) in \(f_{\Exp (\lambda )}(x)=\lambda e^{-\lambda x}\).
-
Lemma 1.2.5 Suppose that \(X\) and \(Y\) are random variables.
-
1. If \(X\) and \(Y\) are discrete, with probability mass functions in the form \(p_X(x)=\frac {1}{Z}g(x)\) and \(p_Y(x)=\frac {1}{Z'}g(x)\) then \(X\eqd Y\).
-
2. If \(X\) and \(Y\) are continuous, with probability density functions in the form \(f_X(x)=\frac {1}{Z}g(x)\) and \(f_Y(x)=\frac {1}{Z'}g(x)\) then \(X\eqd Y\).
-
Proof: \(\offsyl \) Note that in (1.3) the normalizing constant \(Z\) is determined by \(g(x)\); in the discrete case we have \(Z=\sum _{x\in R}g(x)\) and in the continuous case we have \(Z=\int _\R g(x)\,dx\). Hence in both cases, the fact that \(X\) and \(Y\) are random variables implies that \(Z=Z'\). The lemma now follows from Lemma 1.2.3 for the discrete case, and from our discussion below Lemma 1.2.3 for the continuous case. ∎
-
Example 1.2.6 If \(X\sim \Gam (\alpha ,\beta )\) then \(f_X(x)=\frac {\beta ^\alpha }{\Gamma (a)}x^{\alpha -1}e^{-\beta x}\) for \(x>0\), where \(\Gamma :(0,\infty )\to (0,\infty )\) is the \(\Gamma \)-function. By Lemma 1.2.5, if \(Y\) is any other random variable in the form \(f_Y(y)=(\text {constant})\times x^{\alpha -1}e^{-\beta x}\), then we have \(Y\sim \Gam (\alpha ,\beta )\).
When we have \(p_X(x)=\frac {1}{Z}g(x)\) and \(p_Y(x)=\frac {1}{Z'}g(x)\), as in part 1 of Lemma 1.2.5, it is common to summarize this relationship as \(p_X \propto p_Y\). In words, \(p_X\) is proportional to \(p_Y\). The same applies to part 2 of Lemma 1.2.5. This notation can save time, but we will avoid using it while we are focused on understanding conditioning and Bayesian models in Chapters 1-3. We will begin to use it in Section 4.1, where it will become very helpful in keeping our calculations tidy, and we will discuss it further at that point.