Validation Checklist Passing Score

Question

Hi!

I have a mixed-method thesis ongoing. I'm currently on the validation stage of my survey and interview questions. I have adopted a validation checklist from the internet composing of 15 items with a 4-point rating key (Strongly Agree as 4; Agree as 3; Disagree as 2; Strongly Disagree as 1). My thesis advisor told me to have a passing score in the checklist to determine whether the checklist is validated enough and can be deployed to the respondents. However, I can't seem to find any reference in the internet of the passing score for a validation checklist.

I was wondering if I can just take the mean of the scores of the 15-items, then align it with my rating key. For example, a mean of 3.15 means "Agree"; 2.55 means "Disagree". So, if I were to base it here, then the passing score would be 3.00 since it is the minimum point of "Agree".

Is this okay? Or do you have any suggestions regarding how could I state the passing score for the validation checklist? Note that no item in the checklist is weighted (e.g. x2 or x3).

Your suggestions will be highly appreciated. Thank you!

Answer 1

There are a number of steps involved in validating a survey instrument. It is not a quick task. I've included a website address that will answer your question more succinctly than I can. Let me know if you need help, as I have an earned doctorate and completed my own validation process.

https://www.methodspace.com/blog/validating-a-questionnaire

Dr. V

Answer 2

Hey Jared!

Your approach to determining a passing score for your validation checklist is on the right track and quite pragmatic. Let's break down your strategy and consider some additional points that might refine your method:

1. Mean Score Calculation:
2. Calculating the mean score of the validation checklist items is a reasonable method. This mean score represents the overall level of agreement with the validity of your survey and interview questions.
3. Interpreting Mean Scores:
4. Aligning the mean score with your rating key (e.g., 3.15 aligns with "Agree") is a good way to interpret the results. It translates numerical scores into a more qualitative assessment.
5. Setting a Passing Score:
6. Setting 3.00 (which corresponds to "Agree") as the passing score is logical. This threshold ensures that, on average, experts agree that your items are valid. It's a conservative approach that leans towards ensuring content validity.
7. Considerations:
8. Ensure that your panel of validators is diverse and knowledgeable enough to provide a comprehensive assessment.
9. Consider the distribution of scores. If most items are scoring close to the threshold, it might indicate a need for revision. A wide range of scores can provide insights into which items are strong and which might need reworking.
10. Reflect on the feedback for items that score below the threshold. It’s not just about hitting the numbers; qualitative feedback can be crucial for refining your questions.
11. No Weighting:
12. Since you mentioned that no item in the checklist is weighted, each item contributes equally to the overall validation score. This simplification makes the process straightforward but do consider if all items truly have equal importance in your study.
13. Documentation:
14. Document your methodology and rationale for choosing this passing score. This transparency is essential for the academic rigor of your thesis.
15. Alternative Approaches:
16. If desired, you could also consider setting different thresholds for different levels of validation (e.g., "strong validation" for scores above 3.5, "acceptable validation" for scores between 3.0 and 3.5). However, this adds complexity and may not be necessary depending on your objectives.

Hope this helps!