Chance vs. Skill?

Question

How does one mathematically analyze situations that involve chance and skill? Let's take the coin flip as a simple example. Assume that it possible to skillfully flip a coin to get the landing you want. Also assume zero cheating. ### FIRST SCENARIO The world's 5 most talented coin flippers gather to compete. The results: > **Person 1** > > - *Coin Flips*: 100 > > - *Success Rate*: 100% > > **Person 2** > > - *Coin Flips*: 10,000 > > - *Success Rate*: 90% > > **Person 3** > > - *Coin Flips*: 1,000,000 > > - *Success Rate*: 80% > > **Person 4** > > - *Coin Flips*: 100,000,000 > > - *Success Rate*: 70% > > **Person 5** > > - *Coin Flips*: 10,000,000,000 > > - *Success Rate*: 60% Each person claims he is the best coin flipper. How would you analyze the results? ### SECOND SCENARIO A man claims he is so skilled at coin flipping, he can always land heads. > He flips one coin. Sure enough, heads. > > He flips again. Heads again. > > He flips 100 times. All heads. > > 1000 times. Still all heads. > > After 100,000,000,000,000 flips, every single one heads, he stops and > says "I told you so." When do we go from thinking "He's *lucky*!" to "He's *good*!"?

Answer 1

Hmm...it sounds like one of them is named "Rosencrantz". Have you seen the play or movie "Rosencrantz & Guildenstern Are Dead"? Both begin with someone on a horse spotting a coin along a dusty path in the country. He flips it. Heads. Again? Heads. He and his friend begin riding again, and the coin comes up "heads" hundreds of times without failure, while they begin to wonder the sorts of things you do too.

The obvious thing to do first is see if the coins have a built-in bias rather than assuming these people are truly skilled.Design a machine that can randomly flip a coin and see if the coin is potentially biased. Did you know someone has tried something similar already? It turns out that unless you are careful, a coin doesn't actually "flip" in the first place; instead, it spins at an angle but keeps the same faces up and down the entire time.

Still, lets set that aside and focus on your intent. We have built our machine and gotten close to 50% heads and 50% tails after thousands of flips (the Law of Large Numbers says that you need not have an equal number of sides, but that in the long run, you will tend to average out that way). If the machine proves the coin and the flipping process is truly random without human interference, there must be skill at work for your contestants to deviate so dramatically from the LoLN. You could try a runs test for randomness, which enables you to prove or disprove randomness in a string of binomial events. Perhaps we are seeing a rare occurrence of odd but random behavior.

Of course, there is also the simple expedient of performing a hypothesis test on each coin flipper by assuming the heads = 50% and then testing the claim that the experimental results deviate significantly from that. It is a simple calculation, performed in just a few minutes (once you have logged those thousands, that is...).