scholarly journals Quantifying the Distorting Effect of Rapid Guessing on Estimates of Coefficient Αlpha

2021 ◽  
pp. 014662162110517
Author(s):  
Joseph A. Rios ◽  
Jiayi Deng
Keyword(s):  
Simulation Study ◽  
Average Degree ◽  
Coefficient Alpha ◽  
Test Length ◽  
Reliability Measures ◽  
Cognitive Fatigue ◽  
Probability Of Success ◽  
Item Content ◽  
Full Solution ◽  
Perceived Probability

An underlying threat to the validity of reliability measures is the introduction of systematic variance in examinee scores from unintended constructs that differ from those assessed. One construct-irrelevant behavior that has gained increased attention in the literature is rapid guessing (RG), which occurs when examinees answer quickly with intentional disregard for item content. To examine the degree of distortion in coefficient alpha due to RG, this study compared alpha estimates between conditions in which simulees engaged in full solution (i.e., do not engage in RG) versus partial RG behavior. This was done by conducting a simulation study in which the percentage and ability characteristics of rapid responders as well as the percentage and pattern of RG were manipulated. After controlling for test length and difficulty, the average degree of distortion in estimates of coefficient alpha due to RG ranged from −.04 to .02 across 144 conditions. Although slight differences were noted between conditions differing in RG pattern and RG responder ability, the findings from this study suggest that estimates of coefficient alpha are largely robust to the presence of RG due to cognitive fatigue and a low perceived probability of success.

scholarly journals Quantifying the Distorting Effect of Rapid Guessing on Estimates of Coefficient Αlpha

2021 ◽  
Author(s):  
Joseph Rios ◽  
Jiayi Deng
Keyword(s):  
Simulation Study ◽  
Average Degree ◽  
Coefficient Alpha ◽  
Test Length ◽  
Reliability Measures ◽  
Cognitive Fatigue ◽  
Probability Of Success ◽  
Item Content ◽  
Full Solution ◽  
Perceived Probability

An underlying threat to the validity of reliability measures is the introduction of systematic variance in examinee scores from unintended constructs that differ from those assessed. One construct-irrelevant behavior that has gained increased attention in the literature is rapid guessing (RG), which occurs when examinees answer quickly with intentional disregard for item content. To examine the degree of distortion in coefficient alpha due to RG, this study compared alpha estimates between conditions in which simulees engaged in full solution (i.e., do not engage in RG) versus partial RG behavior. This was done by conducting a simulation study in which the percentage and ability characteristics of rapid responders as well as the percentage and pattern of RG were manipulated. After controlling for test length and difficulty, the average degree of distortion in estimates of coefficient alpha due to RG ranged from -.04 to .02 across 144 conditions. Although slight differences were noted between conditions differing in RG pattern and RG responder ability, the findings from this study suggest that estimates of coefficient alpha are largely robust to the presence of RG due to cognitive fatigue and a low perceived probability of success.

scholarly journals Reliability for Tests With Items Having Different Numbers of Ordered Categories

2019 ◽  
Vol 44 (2) ◽  
pp. 137-149
Author(s):  
Seohyun Kim ◽  
Zhenqiu Lu ◽  
Allan S. Cohen
Keyword(s):  
Structural Equation Modeling ◽  
Simulation Study ◽  
Test Scores ◽  
Structural Equation ◽  
Reliability Coefficient ◽  
Coefficient Alpha ◽  
Equation Modeling ◽  
Ordered Categories

This study describes a structural equation modeling (SEM) approach to reliability for tests with items having different numbers of ordered categories. A simulation study is provided to compare the performance of this reliability coefficient, coefficient alpha and population reliability for tests having items with different numbers of ordered categories, a one-factor and a bifactor structures, and different skewness distributions of test scores. Results indicated that the proposed reliability coefficient was close to the population reliability in most conditions. An empirical example was used to illustrate the performance of the different coefficients for a test of items with two or three ordered categories.

Estimation of the Common Probability of Success for Several Binomials —A Simulation Study

1998 ◽  
Vol 48 (1-2) ◽  
pp. 61-72
Author(s):  
Joydeep Bhanja
Keyword(s):  
Lower Bound ◽  
Simulation Study ◽  
Regularity Conditions ◽  
Moment Estimate ◽  
Probability Of Success ◽  
Finite Set ◽  
The Common ◽  
Asymptotically Efficient ◽  
Common Probability

In this paper we consider an example where for each i, i = 1,2, ... , n, the observations Xij , j = 1, 2, ... , k are i.i.d . Binomial ( ni, θ). Based on a theory developed by us earlier, we propose estimates of θ which are asymptotically efficient under the assumption that k ≥ 2, the ni 's come from a finite set { 1, 2, ... , q} and some mild regularity conditions on the sequence { ni} and θ hold. We present the results of a simulation whlch indicate, among other thlngs, the asymptotic lower bound to variance is lower than or approximately equal to simulated Variances and a simple moment estimate of θ does as well as the asymptotically efficient estimates.

scholarly journals An evaluation of the efficiency of plant protection products via nonlinear statistical methods – a simulation study

2014 ◽  
Vol 51 (2) ◽  
pp. 171-179
Author(s):  
Ewa Skotarczak ◽  
Ewa Bakinowska ◽  
Kamila Tomaszyk
Keyword(s):  
Sample Size ◽  
Simulation Study ◽  
Statistical Approach ◽  
Plant Protection ◽  
Simulated Data ◽  
Plant Protection Products ◽  
Data Sets ◽  
Threshold Models ◽  
Probability Of Success ◽  
Simulated Data Sets

Abstract A nonlinear statistical approach was used to evaluate the efficiency of plant protection products. The methodology presented can be implemented when the observations in an experiment are recorded as success or failure. This occurs, for example, when following the application of a herbicide or pesticide, a single weed or insect is classified as alive (failure) or dead (success). Then a higher probability of success means a higher efficiency of the tested product. Using simulated data sets, a comparison was made of three methods based on the logit, probit and threshold models, with special attention to the effect of sample size and number of replications on the accuracy of the estimation of probabilities.

scholarly journals What are the Conditions Associated with Subscore Added Value Noninvariance? Implications for Improving Subscore Interpretation Fairness

2020 ◽  
Author(s):  
Joseph Rios ◽  
Alejandra Miranda
Keyword(s):  
Simulation Study ◽  
Added Value ◽  
Test Length ◽  
Test Taking ◽  
Subgroup Differences ◽  
Operational Testing ◽  
Ethnic Subgroups

Subscore added value analyses assume invariance across test taking populations; however, this assumption may be untenable in practice as differential subdomain relationships may be present among subgroups. The purpose of this simulation study was to understand the conditions associated with subscore added value noninvariance when manipulating: (a) subdomain test length, (b) differences in subgroup mean ability, and (c) subgroup differences in intersubdomain correlations. Results demonstrated that subscore added value was noninvariant for 24% to 100% of replications (depending on subdomain test length) when the subgroup difference in intersubdomain correlation was equal to .30. To examine if this condition was met in practice, applied invariance analyses of three operational testing programs were conducted. Across these datasets, noninvariant subscore added value was present for some subdomains across sex and ethnic subgroups. Overall, these results indicate that subscore added value noninvariance is largely driven by differential intersubdomain correlations among subgroups, which may be present in some operational testing programs.

Language Test Content

1983 ◽  
Vol 14 (1) ◽  
pp. 7-21 ◽  
Author(s):  
Robert E. Owens ◽  
Martha J. Haney ◽  
Virginia E. Giesow ◽  
Lisa F. Dooley ◽  
Richard J. Kelly
Keyword(s):  
Test Item ◽  
Language Assessment ◽  
Assessment Tools ◽  
Language Test ◽  
Speech Language Pathologists ◽  
Diagnostic Aid ◽  
Item Content

This paper examines the test item content of several language assessment tools. A comparison of test breadth and depth is presented. The resultant information provides a diagnostic aid for school speech-language pathologists.

Sign in / Sign up

Export Citation Format

Share Document