Validating Survey Questions

Question

I have a thesis ongoing and I was advised to test my survey in a reliability test. The following are the type of questions found in my survey:

1) Likert scale

2) Yes or No

3) Multiple choice questions (nominal choice)

4) Multiple choice questions (Select all that apply)

I am aware that I can do Cronbach's alpha for the Likert scale. Will it also be applicable to the rest? If not, what reliability test can you suggest to each?

When I browse the internet, I am grasping that in #2 can use Guttman's Lambda due to its dichotomous choice. #3 & 4 can use Krippendorff's alpha. Do you agree with this? Or are there other test can you suggest? Or Cronbach can be applicable to all of them?

Your help will be highly appreciated! Thank you!

Answer 1

Hello Jared!

The choice of reliability test depends on the type of data and measurement scales used in your survey. Here are some suggestions for reliability tests for the different question types in your survey:

1. Likert Scale (Question #1):
2. Reliability Test: Cronbach's alpha is commonly used for Likert scale questions. It measures internal consistency.
3. Yes or No (Question #2):
4. Reliability Test: Guttman's Lambda is indeed suitable for dichotomous (binary) choices like Yes or No. It assesses the reliability of dichotomous items.
5. Multiple Choice Questions (Nominal Choice) (Question #3):
6. Reliability Test: Krippendorff's alpha is a good choice for nominal data, especially when dealing with multiple raters. It can be used to assess the reliability of nominal categorical items.
7. Multiple Choice Questions (Select All That Apply) (Question #4):
8. Reliability Test: Krippendorff's alpha can also be applicable here, especially if your multiple-choice questions involve more than two categories. It is suitable for assessing reliability in cases of multiple raters and multiple categories.

In summary:

1. Use Cronbach's alpha for Likert scale questions.
2. Use Guttman's Lambda for Yes or No questions.
3. Use Krippendorff's alpha for both nominal choice multiple-choice questions and select all that apply questions.

Ensure that the selected reliability tests align with the nature of your data and the measurement scales used in each question type. Always consider the specific characteristics of your survey when choosing reliability tests.

Answer 2

Hello! So, is this for an undergraduate or a graduate thesis? That might matter.

Next, are you looking for reliability of the whole survey itself, or are looking to see if your measures are reliable? If you're looking to determine whether the "whole survey" is reliable, none of the above methods world work. For this, you'd actually have to deliver the survey. There are two ways to do this.

1) Test/retest -- you give the survey to a sample. You then give the survey to the same group again some time later, and then compare the two tests. If the responses are consistent across each, then you're good!

2) Parallel form reliability -- you create two surveys measuring the same concepts using different questions.

However, you most likely are looking to see if your actual measures/concepts are reliable. It's tough to do this with "individual" items, though something like a split-half reliability test can help here (Google that if you have to). To determine the reliability of scales and created variables, Chronbach's alpha will be fine for testing reliability.