Select one of the instruments

Question

1. What levels of measurements are the variables included in the instrument or test?
2. How are the scores interpreted?
3. Are the scores standardized or normed based on age or education levels?
4. For the instrument, describe the reliability testing that has been performed on the instrument.
5. For example, Test-Retest method, Internal Consistency method
6. What factors might affect the test’s reliability? (Refer to the six factors we discussed in lecture.)
7. Did the instrument/test developer’s use a classical theory or generalizability approach in developing the instrument and testing its psychometric properties?

Part Two:

1. Explain the concepts of inter-scorer agreement and intra-scorer agreement. In what types of tests are these concepts most important?
2. What are the implications of a professor using the same test as a pre- and posttest? What, if anything, might the professor do to increase reliability of the test scores?

Answer 1

Part 1: BDI-II Instrument Analysis

1. Level of Measurement:
2. Individual Items: Ordinal (0–3 severity scale per question).
3. Total Composite Score: Interval (Summed score from 0 to 63).
4. Score Interpretation: Criterion-based severity cutoffs:
5. 0–13: Minimal
6. 14–19: Mild
7. 20–28: Moderate
8. 29–63: Severe
9. Standardization & Norms: Criterion-referenced using clinical and student norm groups. Scores are not adjusted for age or education level.
10. Reliability Testing:
11. Internal Consistency: High ().
12. Test-Retest Reliability: High stability ( over 1 week).
13. 6 Factors Affecting Reliability:
14. Test Length: 21 items provide balance between coverage and fatigue.
15. Sample Homogeneity: Broader score ranges increase reliability metrics.
16. Test-Taker State: Fatigue, anxiety, or illness introduces error.
17. Environment: Distractions or non-standard instructions lower consistency.
18. Item Clarity: Ambiguous items cause unstable responses.
19. Subjectivity/Bias: Self-reporting is vulnerable to social desirability bias.
20. Theoretical Approach: Classical Test Theory (CTT) ().

Part 2: Psychometric Concepts & Assessment

Inter-Scorer vs. Intra-Scorer Agreement

1. Inter-Scorer Agreement: Consistency between two or more different graders scoring the same item.
2. Intra-Scorer Agreement: Consistency of one grader scoring the same item at different times.
3. Where Most Important: Subjective, open-ended assessments (essays, oral presentations, portfolio evaluations).

Pre- and Post-Test (Using the Same Test)

1. Implications & Risks:
2. Practice Effects: Memory of pre-test items artificially inflates post-test scores.
3. Test Reactivity: Pre-test primes students to study only tested concepts.
4. Regression to the Mean: Extreme high/low pre-test scores shift toward average automatically.
5. Solutions to Increase Reliability:
6. Parallel/Alternate Forms: Use Form A for the pre-test and Form B for the post-test (matching difficulty and coverage).
7. Clear Rubrics: Standardize scoring criteria to eliminate grader bias.
8. Identical Testing Conditions: Keep time limits, directions, and environment uniform.