1. Statistics And Probability - Moment Generating Functions

This entry builds intuition and detailed derivations for Moment Generating Functions (MGFs). Since many statistical derivations rely on MGFs throughout the field, we present the concepts and steps in an explicit, accessible way so you (and me) can return to this reference whenever needed and quickly recall the essential techniques.

1.1 What is an MGF?

Let $X$ be a random variable. The moment generating function is defined as

\[ \begin{equation} M_X(t)=\mathbb{E}\left[e^{tX}\right],\quad \text{for values of $t$ where the expectation exists.} \label{eq:mgf_definition} \end{equation} \]

Why this definition?

Because if you expand $e^{tX}$ into a series, you will see the moments of $X$ appear one by one. This property makes MGFs a powerful tool for deriving and manipulating moments without computing integrals or sums directly. By taking derivatives of $M_X(t)$ at $t=0$, you can extract any moment of the distribution.

1.2 How moments appear inside the MGF

Start with the Taylor expansion of $e^{tX}$:

\[ \begin{equation} \exp(tX)=\sum_{n=0}^{\infty}\frac{t^n X^n}{n!} \label{eq:mgf_taylor_expansion} \end{equation} \]

How and Why of $\ref{eq:mgf_taylor_expansion}$

The Taylor Series Foundation:

Here $X$ is a scalar random variable, and we use the standard Taylor series for the exponential function:

\[e^u = \sum_{n=0}^{\infty}\frac{u^n}{n!} = 1 + u + \frac{u^2}{2!} + \frac{u^3}{3!} + \cdots\]

Applying it to $e^{tX}$:

When we set $u = tX$, we get:

\[\exp(tX) = \sum_{n=0}^{\infty}\frac{(tX)^n}{n!} = \sum_{n=0}^{\infty}\frac{t^n X^n}{n!} = 1 + tX + \frac{t^2 X^2}{2!} + \frac{t^3 X^3}{3!} + \cdots\]

When $t=0$: $\exp(0) = 1$
When $X$ is a random variable, each term $X^n$ is also a random variable
Taking expectations term-by-term (assuming we can swap $\mathbb{E}$ and $\sum$) gives us the moments of $X$

Take expectation on both sides:

\[ \begin{equation} M_X(t)=\mathbb{E}\left[e^{tX}\right]=\sum_{n=0}^{\infty}\frac{t^n}{n!}\mathbb{E}\left[X^n\right] \label{eq:mgf_series_moments} \end{equation} \]

Jensen could be really mad if you forget that ...

... the assumtion in $\ref{eq:mgf_series_moments}$ applies exclusively to linear functions. In another cases where the function is non-linear, the expression requires an adjustment given the Jensen's Inequality.

That means the first derivative at $t=0$ gives the mean, while the second derivative at $t=0$ gives the second moment, and the same pattern continues for higher moments.

Formally,

\[ \begin{equation} M_X^{(n)}(0)=\mathbb{E}\left[X^n\right] \label{eq:mgf_derivatives_moments} \end{equation} \]

1.3 Examples

The expected value $\mathbb{E}[X]$ is defined differently depending on whether $X$ is discrete or continuous, but the concept is the same: a weighted average of all possible values.

For discrete $X$:

\[\mathbb{E}[X]=\sum_{k} k \cdot \mathbb{P}(X=k)\]

Each possible value $k$ is weighted by its probability $\mathbb{P}(X=k)$. The sum accumulates these weighted contributions.

For continuous $X$:

\[\mathbb{E}[X]=\int_{-\infty}^{\infty} x \cdot f_X(x)\,dx\]

Each infinitesimal value $x$ is weighted by its probability density $f_X(x)$. The integral accumulates these weighted contributions over all real numbers.

The connection: Both operations, the sum and the integral. are computing a weighted average. The sum groups discrete outcomes; the integral groups infinitely many continuous outcomes. When you see $\mathbb{E}[g(X)]$, replace the sum or integral with the same pattern: apply the function $g$ to each outcome, weight it by probability (or density), then sum or integrate.

1.3.1 Bernoulli($p$)

Let $X\in\{0,1\}$ with $\mathbb{P}(X=1)=p$ and $\mathbb{P}(X=0)=1-p$.

Compute the MGF directly from the definition:

\[ \begin{equation} M_X(t)=\mathbb{E}\left[e^{tX}\right]=(1-p)\,e^{t\cdot 0}+p\,e^{t\cdot 1}=1-p+p\,e^{t} \label{eq:mgf_bernoulli} \end{equation} \]

Why this formula for $\mathbb{E}[e^{tX}]$?

For a discrete random variable, the expected value is computed by summing the function values weighted by their probabilities. In general, $\mathbb{E}[g(X)]=\sum_{k} g(k)\mathbb{P}(X=k)$. Here $g(X)=e^{tX}$, and $X$ takes only two values: $0$ and $1$. So we get

\[ \mathbb{E}[e^{tX}]=e^{t\cdot 0}\mathbb{P}(X=0)+e^{t\cdot 1}\mathbb{P}(X=1)=(1-p)\,e^{0}+p\,e^{t}=1-p+p\,e^{t}. \]

This is the definition of expected value for discrete distributions applied to the Bernoulli case.

Check the mean from the derivative:

\[ \begin{equation} M_X'(t)=p\,e^{t}\quad\Rightarrow\quad M_X'(0)=p=\mathbb{E}[X] \label{eq:mgf_bernoulli_mean} \end{equation} \]

Why does $M_X'(0)$ equal the mean?

Recall from $\ref{eq:mgf_series_moments}$ that the MGF can be written as

\[ M_X(t)=\mathbb{E}\left[X^0\right]+t\mathbb{E}\left[X^1\right]+\frac{t^2}{2!}\mathbb{E}\left[X^2\right]+\cdots \]

When we take the derivative with respect to $t$, we get

\[ M_X'(t)=\mathbb{E}\left[X^1\right]+t\mathbb{E}\left[X^2\right]+\frac{t^2}{2!}\mathbb{E}\left[X^3\right]+\cdots \]

Evaluating at $t=0$ makes all terms with $t$ vanish, leaving only $M_X'(0)=\mathbb{E}[X]$, the first moment (mean).

For the Bernoulli case specifically, we can verify this directly: $\mathbb{E}[X]=0\cdot(1-p)+1\cdot p=p$, which matches $M_X'(0)=p$.

1.3.2 Normal $N(\mu,\sigma^2)$

Let $X\sim N(\mu,\sigma^2)$. We compute

\[ \begin{equation} M_X(t)=\mathbb{E}\left[e^{tX}\right]=\int_{-\infty}^{\infty} e^{tx}\,\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)dx \label{eq:mgf_normal_integral} \end{equation} \]

Why this integral for $\mathbb{E}[e^{tX}]$?

For a continuous random variable with probability density function (PDF) $f_X(x)$, the expected value of any function $g(X)$ is computed by integrating the function times the PDF:

\[ \mathbb{E}[g(X)]=\int_{-\infty}^{\infty}g(x)f_X(x)\,dx. \]

In our case, $g(X)=e^{tX}$ and the PDF of $X\sim N(\mu,\sigma^2)$ is

\[ f_X(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right). \]

So the MGF becomes the integral shown below: we multiply $e^{tx}$ by the Gaussian PDF and integrate over all $x$.

Combine the exponent terms and complete the square:

\[ \begin{equation} -\frac{(x-\mu)^2}{2\sigma^2}+tx=-\frac{1}{2\sigma^2}\left(x-(\mu+\sigma^2 t)\right)^2+\mu t+\frac{1}{2}\sigma^2 t^2 \label{eq:mgf_normal_complete_square} \end{equation} \]

Now the integral becomes the integral of a normal PDF, which equals 1. So,

\[ \begin{equation} M_X(t)=\exp\left(\mu t+\frac{1}{2}\sigma^2 t^2\right) \label{eq:mgf_normal_result} \end{equation} \]

Extracting the mean from $M_X(t)$

We have $M_X(t)=\exp\left(\mu t+\frac{1}{2}\sigma^2 t^2\right)$. To find the mean, take the derivative:

\[ M_X'(t)=\left(\mu+\sigma^2 t\right)\exp\left(\mu t+\frac{1}{2}\sigma^2 t^2\right) \]

Evaluating at $t=0$ gives

\[ M_X'(0)=\mu\cdot\exp(0)=\mu \]

This confirms that the first derivative at $t=0$ recovers the mean parameter $\mu$ of the normal distribution. This follows the general principle from $\ref{eq:mgf_derivatives_moments}$: the first derivative at zero always gives $\mathbb{E}[X]$.

1.4 Sum of independent variables

If $X$ and $Y$ are independent, then

\[ \begin{equation} M_{X+Y}(t)=\mathbb{E}\left[e^{t(X+Y)}\right]=\mathbb{E}\left[e^{tX}\right]\mathbb{E}\left[e^{tY}\right]=M_X(t)\,M_Y(t) \label{eq:mgf_sum_independent} \end{equation} \]

This is one of the most useful properties of MGFs. It lets you find the distribution of sums by multiplying MGFs.

1.5 Why MGFs are useful

MGFs encode all moments in a single function, which makes them a compact summary of the distribution. They also make sums easy to handle in the independent case, because multiplying MGFs corresponds to adding random variables. Finally, they can identify distributions: if two variables share the same MGF in a neighborhood of zero, they have the same distribution.

1.6 Quick checklist for practice

Try to compute $M_X(t)$ for a discrete variable using a sum and for a continuous variable using an integral, and then recover the mean and variance by differentiating $M_X(t)$ at $t=0$. If you can do those three tasks, the core mechanics of MGFs will feel natural.

Building confidence through rigorous validation