The delta binomial test for interjudge reliability

This paper suggests an improved method of reporting the degree of consistency between judges' coding of marketing and advertising content. The method is based on the binomial sampling distribution theory where scores are test scores indicating statistical significance of measures of agreement. The proposed method is an improvement on the standard percent indices, which are simple measures of agreement that do not control for possible agreement by chance alone.