Products and expectation values

by Willie Wong

Let us start with an instructive example (modified from one I learned from Steven Landsburg). Let us play a game:

I show you three identical looking boxes. In the first box there are 3 red marbles and 1 blue one. In the second box there are 2 red marbles and 1 blue one. In the last box there is 1 red marble and 4 blue ones. You choose one at random. What is …

  • The expected number of red marbles you will find?
  • The expected number of blue marbles you will find?
  • The expected number of marbles, irregardless of colour, you will find?
  • The expected percentage of red marbles you will find?
  • The expected percentage of blue marbles you will find?

Answer below the cut…

You have equal chances (1/3) of choosing each of the three boxes. So the expected number of red marbles is \dfrac{1}{3}(3 + 2 + 1) = 2. The expected number of blue marbles is \dfrac{1}{3}(1 + 1 + 4) = 2. So far so good. The expected number of marbles you get is \dfrac{1}{3}[(3 + 1) + (2 + 1) + (4 + 1)] = 4.

So the expected numbers of red and blue marbles are the same. This means that we expect half of the marbles to be red, on average, right?

Wrong! The expected percentage of red marbles is

\dfrac{1}{3}\left( \dfrac{3}{1+3} + \dfrac{2}{1 + 2} + \dfrac{1}{1+4}\right) = \dfrac{1}{3} \left( \dfrac{3}{4} + \dfrac{2}{3} + \dfrac{1}{5}\right) = \dfrac{97}{180} > \dfrac{1}{2}

and the expected percentage of blue marbles is

\dfrac{1}{3}\left( \dfrac{1}{4} + \dfrac{1}{3} + \dfrac{4}{5}\right) = \dfrac{83}{180} < \frac{1}{2}

So what gives?

Expectation values and random variables
Let us formalize the discussion a little bit. Let us suppose that in the game, I give you n identical boxes, but for the purpose of computation we’ll call the boxes 1, \ldots, n. The box number k contains R(k) red marbles and B(k) blue marbles, so R and B are functions whose domain is the set \{1, \ldots, n\} = \Omega and who take values in the non-negative integers. (Of course, we can generalize even further by having more than two colours and fractional pieces of marbles.) By the strategy in which you play the game, we derive a probability measure P, with the property that

  • The domain of P is \{1, \ldots, n\} = \Omega, the set of possible boxes to choose from.
  • The value of P is between 0 and 1.
  • The total probability, that is to say \sum_{k = 1}^n P(k), add up to 1.

In the random selection case each P(k) = 1/n.

Given a set of outcomes \Omega and a probability P, a function whose domain is \Omega is called a random variable. In our example both R and B are random variables. The expected value of the random variable is called its expectation (value) and written, and computed, as follows

\displaystyle \mathbb{E}(R) = \sum_{k\in \Omega} P(k) R(k)

From the elementary rules of arithmetic, using that the expectation is a sum, and hence is linear in its argument, we have that

\mathbb{E}(x R + y B) = x \mathbb{E}(R) + y \mathbb{E}(B)

where x,y are real numbers.

For example, the total number of marbles in a box, which we can write T, satisfies T(k) = R(k) + B(k). So we have that \mathbb{E}(T) = \mathbb{E}(R) + \mathbb{E}(B).

Products of random variables
Unlike sums and differences of random variables, products of random variables does, in general, not commute with operation of taking an expectation. That is to say, in general the expressions \mathbb{E}(RB) and \mathbb{E}(R)\cdot \mathbb{E}(B) take different values. There are, however, cases where the two do agree. First recall that \mathbb{E}(R) is a real number. So \mathbb{E}( \mathbb{E}(B) R) = \mathbb{E}(R) \mathbb{E}(B) by the linearity of expectations. So we can compute

\displaystyle \mathbb{E}\left[ (R - \mathbb{E}(R))\cdot(B - \mathbb{E}(B))\right] = \mathbb{E}\left[ RB + \mathbb{E}(R)\cdot\mathbb{E}(B) - \mathbb{E}(R)\cdot B - \mathbb{E}(B)\cdot R\right]

The third and fourth terms on the RHS both evaluate to -\mathbb{E}(R)\cdot \mathbb{E}(B), as does the second term (the expectation value of a constant is the constant itself). So we conclude that

\displaystyle \mathbb{E}\left[ (R - \mathbb{E}(R))\cdot(B - \mathbb{E}(B))\right] = \mathbb{E}(RB) - \mathbb{E}(R)\cdot \mathbb{E}(B)

The quantity in the above equation is known as the covariance of the two random variables, and provides a measure of how the two variables change together. A tautological statement is then: “the expectation of a product of random variables is equal to the product of their expectation, if and only if their covariance is zero”.

An immediate consequence is that

Let X be a non-vanishing random variable. Then \mathbb{E}(1/X) = 1/ \mathbb{E}(X) if and only if X is constant over all outcomes with positive probability.

Proof: If X is a constant, then it is clear that the desired equality holds. So it suffices to prove the other implication. The desired equality is equivalent to \mathbb{E}(X) \mathbb{E}(1/X) = 1 = \mathbb{E}(1) = \mathbb{E}(X \cdot 1/X). Assuming this holds, we have that the covariance

\displaystyle \mathbb{E}\left[ (X - \mathbb{E}(X)) (\frac{1}{X} - \mathbb{E}(\frac{1}{X}))\right] = \mathbb{E}\left[ (X - \mathbb{E}(X)) ( \frac{1}{X} - \frac{1}{\mathbb{E}(X)})\right]

On the other hand, observe that X > \mathbb{E}(X) \iff 1/X < 1 / \mathbb{E}(X). This implies that the random variable (X - \mathbb{E}(X))(1/X - 1 / \mathbb{E}(X)) is non-positive. For its expectation to be exactly 0 requires that the random variable itself vanishes on all events with positive probability, which shows the claim.

Moral of the story
In general,

\displaystyle \mathbb{E}\left( \frac{R}{R + B}\right) \neq \frac{ \mathbb{E}(R) }{\mathbb{E}(R) + \mathbb{E}(B)}

and so one should be very careful when trying to draw conclusions about the expected “proportion” from the expected “numbers”.