On the bias and stability of the results of comparative judgment

Comparative judgment is a method that allows measurement of a competence by comparison of items with other items. In educational measurement, where comparative judgment is becoming an increasingly popular assessment method, items are mostly students’ responses to an assignment or an examination. For assessments using comparative judgment, the Scale Separation Reliability (SSR) is used to estimate the reliability of the measurement. Previous research has shown that the SSR may overestimate reliability when the pairs to be compared are selected with certain adaptive algorithms, when raters use different underlying models/truths, or when the true variance of the item parameters is below one. This research investigated bias and stability of the components of the SSR in relation to the number of comparisons per item to increase understanding of the SSR. We showed that many comparisons are required to obtain an accurate estimate of the item variance, but that the SSR can be useful even when the variance of the items is overestimated. Lastly, we recommend adjusting the general guideline for the required number of comparisons per item to 41 comparisons per item. This recommendation partly depends on the number of items and the true variance in our simulation study and needs further investigation.

Enhancing Language Assessment Skills among Faculty Members in Saudi EFL Context

In Saudi higher education, assessment has shifted to incorporate intended learning outcomes rather than merely textbook content. Subsequently, faculty members unwillingly participate in high-stakes competitive and harmonized assessment in English as a Foreign Language (EFL) courses during the preparatory year (PY). These challenges emphasize the importance of need-based training for faculty members. Accordingly, this context-specific study scrutinized faculty members’ needs as well as the impact of a training program on engaging the participants and on fostering language assessment skills (LASs) among them. In so doing, an action research design used pre- and post-questionnaires and included a training portfolio to collect data from 31 faculty members. The study first identified those needs as instructional skills, design skills, and educational measurement skills. In the context of professional development, the researchers designed a training program based on those reported needs. During training, the participants expressed their satisfaction with the language assessment. After the training, the participants greatly improved their LASs. The paired tests indicated that the faculty members increased their instructional skills, design skills, and skills of educational measurement. Further research is recommended for enhancing LASs among EFL students.