1. Statistics And Probability - Expected Value

Sub-section with a list of some derivations of nice properties of Expected Value (\(\mathbb{E}\)). All these equivalences are added in this personal binacle as a reminder of their origins, just not taking it as given all the times when I need to use thereof. Built as a post to recall some properties.

1.1 Expected Value of \(e\) power a random rariable

Consider a random variable \(Z\sim N(0,\sigma^2)\), namely, following a normal distribution with mean equal to zero and variance \(\sigma^2\). The following equivalence requires to remember that the expected value for a continuous variable can be described as the integral of the function multipyed by its probability distribution function (PDF).

\[ \begin{equation} \mathbb{E}\left[e^{(a-h)\cdot Z}\right]=\int_{-\infty}^\infty e^{(a-h) z}f_Z(z)dz \label{eq:expected_equivalence} \end{equation} \]

where \(f_Z(z)\) representes the PDF for a normal random variable \(Z\), and \(z \in Z\), as shown in \(\ref{eq:pdf_normal_distribution}\)

\[ \begin{equation} f_Z(z) = \frac{1}{\sqrt{2 \pi \sigma^2}}\times \exp\left(-\frac{z^2}{2\sigma^2}\right) \label{eq:pdf_normal_distribution} \end{equation} \]

Then, by replacing \(\ref{eq:pdf_normal_distribution}\) in \(\ref{eq:expected_equivalence}\) we obtain

\[ \begin{equation} \mathbb{E}\left[e^{(a-h)\cdot Z}\right]=\int_{-\infty}^\infty \exp((a-h) z) \times \frac{1}{\sqrt{2 \pi \sigma^2}}\times \exp\left(-\frac{z^2}{2\sigma^2}\right) dz \label{eq:expected_equivalence_II} \end{equation} \]

Therefore grouping the exponent

\[ \begin{equation} \mathbb{E}\left[e^{(a-h)\cdot Z}\right]= \frac{1}{\sqrt{2 \pi \sigma^2}} \times \int_{-\infty}^\infty \exp\left((a-h) z -\frac{z^2}{2\sigma^2} \right) dz \label{eq:expected_equivalence_III} \end{equation} \]

Now, lets hold a moment to take a breath and develop the power of the exponent as shown in \(\ref{eq:expected_equivalence_internal_comp}\)

\[ \begin{equation} (a-h) z -\frac{z^2}{2\sigma^2} = \frac{2\sigma^2(a-h)z-z^2}{2\sigma^2} =-\frac{1}{2\sigma^2}\times(z^2-2\sigma^2(a-h)z) \label{eq:expected_equivalence_internal_comp} \end{equation} \]

1.1.1 Completing the quadratic binomial

💊 Trick Number 1

The second factor in \(\ref{eq:expected_equivalence_internal_comp}\) seems to be pretty similar to an incomplete quadratic binomial. Hence, the next step may be to complete it. Lets use \((\sigma^2(a-h)z)^2\) to complete the quadratic binomial.

\[ \begin{equation} -\frac{1}{2\sigma^2}\times\left[z^2-2\sigma^2(a-h)z + \sigma^2(a-h)z^2 - (\sigma^2(a-h)z)^2\right] \label{eq:expected_equivalence_completing} \end{equation} \]

Aforementioned yields the equation \(\ref{eq:expected_equivalence_completing_grouped}\) when grouping the quadratic binomial.

\[ \begin{equation} -\frac{1}{2\sigma^2}\times\left[(z^2-\sigma^2(a-h)z)^2 - \sigma^4(a-h)^2 z^2\right] \label{eq:expected_equivalence_completing_grouped} \end{equation} \]

Then, with a bit of arithmetic in \(\ref{eq:expected_equivalence_completing_grouped}\) and pluging back in \(\ref{eq:expected_equivalence_II}\) we obtain \(\ref{eq:expected_equivalence_IV}\).

\[ \begin{equation} \mathbb{E}\left[e^{(a-h)\cdot Z}\right]=\frac{1}{\sqrt{2 \pi \sigma^2}}\times\int_{-\infty}^\infty exp\left( \frac{\sigma^2(a-h)^2}{2} - \frac{(z-\sigma^2(a-h))^2}{2\sigma^2} \right) dz \label{eq:expected_equivalence_IV} \end{equation} \]

Now, since there is a term within the integral that does not depend on \(z\), it is took out as an additional factor.

\[ \begin{equation} exp\left( \frac{\sigma^2(a-h)^2}{2} \right) \times \frac{1}{\sqrt{2 \pi \sigma^2}}\times \int_{-\infty}^\infty exp\left(- \frac{(z-\sigma^2(a-h))^2}{2\sigma^2}\right) dz \label{eq:expected_equivalence_V} \end{equation} \]

1.1.2 The normal distribution CDF in the solution, again

💊 Trick Number 2

Do you see something similar to \(\ref{eq:pdf_normal_distribution}\)?. You're right, equation \(\ref{eq:expected_equivalence_V}\) is comprised by the exponent that was took out the integral times the Cummulative Distribution Function (CDF) of a normal distribution with mean \(\sigma^2(a-h)\) and variance \(\sigma^2\), namely, \(N(\sigma^2(a-h), \sigma^2)\).

Moreover, it's known that integral s equal to \(\sqrt{2\pi\sigma^2}\), see Gaussian Integral. Therefore, the resulting expresion verses like

\[ \begin{equation} \mathbb{E}\left[e^{(a-h)\cdot Z}\right]=\exp\left( \frac{\sigma^2(a-h)^2}{2} \right) \times \frac{1}{\sqrt{2 \pi \sigma^2}}\times \sqrt{2 \pi \sigma^2} \label{eq:expected_equivalence_VI} \end{equation} \]

Then, concluding that

\[ \begin{equation} \mathbb{E}\left[e^{(a-h)\cdot Z}\right]=\exp\left( \frac{1}{2} \sigma^2(a-h)^2 \right) \label{eq:enlusion} \end{equation} \]

1.2 Expected value of \(e^{aX + bY}\) for correlated Gaussians

Let \((X, Y)\) be jointly normal with means \(\mu_X, \mu_Y\), variances \(\sigma_X^2, \sigma_Y^2\), and correlation \(\rho\) (so \(\mathrm{Cov}(X,Y)=\rho\sigma_X\sigma_Y\)). Then for any real \(a,b\),

\[ \begin{equation} \mathbb{E}\left[e^{aX + bY}\right]=\exp\left(a\mu_X + b\mu_Y + \frac{1}{2}\left(a^2\sigma_X^2 + b^2\sigma_Y^2 + 2ab\rho\sigma_X\sigma_Y\right)\right) \label{eq:expected_value_correlated_gaussians} \end{equation} \]

where is \(\ref{eq:expected_value_correlated_gaussians}\) comming from?

This comes directly from the moment generating function of a multivariate normal. Let \(\mathbf{Z}=(X,Y)^\top\), \(\mathbf{t}=(a,b)^\top\), mean \(\boldsymbol{\mu}=(\mu_X,\mu_Y)^\top\), and covariance

\[ \begin{equation} \Sigma=\begin{bmatrix} \sigma_X^2 & \rho\sigma_X\sigma_Y\\ \rho\sigma_X\sigma_Y & \sigma_Y^2 \end{bmatrix} \label{eq:covariance_matrix_correlated_gaussians} \end{equation} \]

The MGF is \(M_{\mathbf{Z}}(\mathbf{t})=\exp\left(\mathbf{t}^\top\boldsymbol{\mu}+\frac{1}{2}\mathbf{t}^\top\Sigma\mathbf{t}\right)\). Therefore,

\[ \begin{equation} \mathbb{E}\left[e^{aX+bY}\right]=\exp\left(\mathbf{t}^\top\boldsymbol{\mu}+\frac{1}{2}\mathbf{t}^\top\Sigma\mathbf{t}\right) \label{eq:expected_value_correlated_gaussians_mgf} \end{equation} \]

and expanding the quadratic form yields

\[ \begin{equation} \mathbf{t}^\top\Sigma\mathbf{t}=a^2\sigma_X^2+b^2\sigma_Y^2+2ab\rho\sigma_X\sigma_Y. \label{eq:expected_value_correlated_gaussians_expand} \end{equation} \]

Building confidence through rigorous validation