Imagine the following game. You're given $1. Then, you keep flipping a fair coin, and every time the coin lands on heads, your money is doubled. The game ends the first time you see tails. A little thought reveals that your expected payoff is $1 + (1/2)$2 + (1/4)$4 + (1/8)$8 + ... = infinity! No matter how much you're willing to pay per game, if you play enough times, you'll eventually come out ahead. But how much would you pay to play this game once?
Intuitively, you'd be crazy to pay more than $20, and you'd be right not to. One reason is that your utility for money isn't linear: you just don't value 10 billion dollars 10 times more than 1 billion. Another reason is if you got lucky enough, there wouldn't be enough money in the world to pay your winnings -- capping the game at, say, a trillion dollars makes its expected value only around $20. But you wouldn't get lucky enough -- because one-in-a-trillion events just don't occur in real life. That you shouldn't pay more than $20 to play this infinitely profitable game is known as the St. Petersburg paradox.
This paradox is related to the two envelopes problem in my previous post (required reading for the next part). The way I set it up, the two envelopes problem is also a game of infinite expectation. Whether you exchange envelopes or not, your expected payoff a priori is infinite. That's why you can swap envelopes without even looking at what's in yours and expect to gain about 22% -- 1.22 times infinity is still infinity! And the problem with both the St. Petersburg game and the two envelopes problem is that no matter how much money you get, it's less than the expectation. In both games, you get a finite amount with probability 1, but what's that compared to infinity? It's also related to why you always gain in expectation by switching. So, can we make the game's expected value finite and still get the two envelopes paradox? The answer is no! Paradox lost.
I'm happy enough with this explanation, but I'd love to see other ideas about how to resolve these problems. The two envelopes problem and others like it bothered me for a long time, but I'm trying to accept that often our intuitions simply break when we deal with infinity.