alpha levels may be different

Question

Decisions about alpha level may be different, especially as it relates from hard sciences to social sciences. For example, a medical trial for cancer treatments conducts their statistical tests at .0001 – so for every 1 out of 10,000 patients, there may be issues, sickness or even death. For social science, we use alpha .05. We are comfortable with performing research, for example, on students. So we are satisfied with losing 5 out of 100 students or having our results being incorrect 5 out of 100 times. Do you agree with these alpha levels? Why or why not? What if your child’s education and the teacher assigned to him/her would be successful 95 out of 100 times?

Answer 1

I will address your questions in a moment, but first, let's talk a little about what an alpha level is.

An alpha value is the probability that a given test statistic would have been observed under the null hypothesis if it was true. Setting your alpha value to 0.05 is to say that "If the null hypothesis is true and I ran this test an infinite number of times, the proportion of times that I would get a test statistic that is 'rejectable' would happen 5% of the time." Your examples of "1 out of 10,000 patients" and "5 out of 100 students" are not what the alpha value is talking about. Your example of "being incorrect 5 out of 100 times" is more appropriate to what the alpha level is trying to get to.

To address your question about if I agree with "these alpha levels," I would need to know what studies you are referring to. The biggest problem with alpha being 0.05 is not that it's wrong 5% of the time, the biggest problem is that studies that don't have a significant test statistic don't get published. Let's say that we were testing a new method of math instruction. 1000 different researchers did independent and correctly run studies. 50 of them found that the new method showed a dramatic increase in the students abilities. 950 of the studies did not show much or any increase in the students abilities. If you knew this, you would probably say that the 50 that show a dramatic increase were somewhat of a random fluke since 950 to 50 is not a good ratio of success to failure.

However, when it comes to academics and policy making, people tend to look for peer reviewed articles in journals (those are the studies people tend to refer to in news articles.) If the journal only publishes the studies that had a significant result (the ones that showed a dramatic increase in the students success,) then there would be 50 peer reviewed articles about the successful studies and no mention of the 950 studies that were not successful. This is an unfortunate bias that currently exists called Publication Bias.

A more direct answer to your question is that your child's education is not a single event. It is made up of very many small events. If an idea or method was correct 95% of the time, you could look at it as everyday, there is a 95% chance that the method will work for the student on that individual day. If you were to look at it that way, then one would expect students to get 95% of the material and they would get a 95% in the class. There would be some students that would be above that and some that would be below. But that kind of success doesn't sound too bad to me.