Meta-Analysis of Correlation Coefficients: A Cautionary Tale on Treating Measurement Error
Abstract: A scale to measure a psychological construct is subject to measurement error. When meta-analyzing correlations obtained from scale scores, many researchers recommend correcting measurement error. We considered three caveats when conducting meta-analysis of correlations: (1) the distribution of true scores can be non-normal, resulting in a violation of the normality assumption for raw correlations and Fisher's z transformed correlations; (2) coefficient alpha is often used as the reliability, but correlations corrected for measurement error using alpha can be inaccurate when some assumptions (e.g., tau-equivalence) of alpha are violated; and (3) item scores are often ordinal, making the disattenuation formula potentially problematic. Via three simulation studies, we examined the performance of two meta-analysis approaches with raw correlations and z scores. In terms of estimation accuracy and coverage probability of the mean correlation, results showed that (1) the true score distribution alone had slight influence; (2) when the tau-equivalence assumption was violated and coefficient alpha was used for correcting measurement error, the mean correlation estimate can be biased and coverage probability can be low; and (3) discretization of continuous items can result in under-coverage of the mean correlation even when tau-equivalence was satisfied. With more categories and/or items on a scale, results can improve when tau-equivalence was met or not. Based on these findings, we then gave recommendations when conducting meta-analysis of correlations.