Using rWG(J) and the Intraclass Correlation Coefficient (ICC) as Measures of Rater
Abstract
When measuring social constructs such as organizational culture and climate, researchers may aggregate perceptual measures without first establishing perceptual agreement. This results in an average of the ratings without first determining the extent to which the raters are experiencing, and thereby rating, the same social construct. Interrater agreement (rWG(J)) and reliability (ICC) are common methods used to determine perceptual agreement. Students and researchers should be proficient in using and interpreting these measures. In this article I review one measure of interrater agreement and three measures of interrater reliability. For each I provide a brief history of their development, a justification for their usage, and selected research which serves as a benchmark for interpreting their results.