How to quantify the differencen between 2/4 and 20/40?

Question

Assume I have two methods to do prediction. The first method makes 4 predictions and 2 out of 4 are correct. The second method makes 40 predictions and 20 out of 40 are correct. The prediction precisions of both methods are the same, which is 2/4=20/40=0.5. But I think the second method is better than the first one, because it makes more correct predictions. Is there a measure to quantify this? Any suggestion may help:) Thanks in advance.

Answer 1

The standard deviation is probably smaller for the 20 out of 40 than the 2 out of 4.

so for example the first method, 2 of 4 has correct prediction of 0.5 but possibly with

say a standard deviation of .25. So you're within 2 standard deviations of 0.5 100% of

the time, which really doesn't say much of anything. No matter what the true parameter proportion

is, it's always somewhere between 0 and 100% correct, somewhere between 0.0 and 1.0

But for 20 out of 40, the standard deviation is probably much smaller, maybe say .10

Then 2 standard deviations from the mean of 20/40 is .4 or .6. 95% of the time

the true parameter mean is between .4 and .6. That gives you more confidence

you're nearer to the true parameter mean of .5 with a sample size of 40 than of 4.

In both cases the most likely estimate of the mean is 0.5, but the larger the sample,

the smaller the likely standard deviation. You've narrowed the likely parameter to

a smaller range of values.

To get the standard deviation, you're dividing the sum of squared differences from the sample

mean by the square root of n-1 where n=the sample size. 4-1 for the 1st method, 40-1 for

the 2nd method. dividing by a larger number gives a smaller number for the standard deviation.