Statistics question- Testing a Hypothesis (Type I and Type II errors)

Question

Remember our bank that’s sending out DVDs to try to get customers to make payments on delinquent loans? It is looking for evidence that the costlier DVD strategy produces a higher success rate than the letters it has been sending. 

a. Explain what a Type I error is in this context and what the consequences would be to the bank.

b. What’s a Type II error in the bank experiment context, and what would the consequences be?

c. For the bank, which situation has higher power: a strategy that works really well, actually getting 60% of people to pay off their balances, or a strategy that barely increases the payoff rate to 32%? Explain briefly

Answer 1

Statistical hypothesis testing implies that no test is ever 100% certain: that’s because we rely on probabilities to experiment.

When online marketers and scientists run hypothesis tests, both seek out statistically relevant results. This means that the results of their test have to be true within a range of probabilities (typically 95%).

Even though hypothesis tests are meant to be reliable, there are two types of errors that can occur.

These errors are known as type 1 and type 2 errors.

a. Type one error in this scenario means the null hypothesis that sending DVDs does not produce higher success rate is true but rejected. This means that bank wrongfully assume that their hypothesis testing has worked even though it hasn’t, thus developing overconfidence in the experiment and make a bad investment.

b. Type 2 error is the opposite. It is a false negative. The null hypothesis is accepted when it should be rejected. In this example the bank wrongfully assume that their hypothesis testing did not work when actually it did. They will not have confidence in their experiment and also make a bad decision of assuming the null hypothesis is true i.e. DVD strategy does not provide higher success rate.

c. Based on the information we cant conclude whether getting 60% of people to pay is better or increasing the payoff rate to 32% is a better strategy, because

1) we don't know the absolute amount of debt

2) the 40% who don't pay may have the bigger debt

3) we can't really compare the amount of debt paid off in the 2 situations.

Answer 2

Answering only part C since parts A & B were answered so well by Kenny S above:

Assuming the word "power" is intended to mean statistical power and that we're comparing the power for the same proposed sample size across both outcomes (big improvement, small improvement), then the power is higher for the larger change. Statistical power is officially one minus the probability of type 2 error. Essentially higher power means you're more likely to reject the null hypothesis correctly (the upper left box of Kenny's diagram). When are you more likely to reject the null hypothesis? When the observed effect is larger. When is the observed effect likely to be larger? When the true effect is larger. So, the bigger the relevant effect size, the bigger the statistical power.

By analogy, it's much easier to tell that an elephant is charging you than it is to notice that an ant is charging you. Side note: it's also more harmful when an elephant charges you. That's where Kenny's part c answer comes in by noting that it's not just success/failure of debt repayment, but whether the underlying debt (i.e. elephant) is larger/smaller. Since the question doesn't specifically ask about statistical power, both Kenny's and my answers are reasonable.