Categorization errors and differences in the quality of questions across countries

Oberski, Daniel L.; Saris, Willem E.; Hagenaars, Jacques A.

Categorization errors and differences in the quality of questions across countries

License

Citation

Oberski, D, Saris, W, Hagenaars, J. Categorization errors and differences in the quality of questions across countries. 2009. handle: http://hdl.handle.net/10230/28315

This citation was generated automatically.

Abstract

The European Social Survey (ESS) has the unique characteristic that in more than 20 countries the same questions are asked and that within each round of the ESS Multitrait-Multimethod (MTMM) experiments are built in to evaluate the quality of a limited number of questions. This gives us an exceptional opportunity to observe the differences in quality of questions over a large number of countries. The MTMM experiments make it possible to estimate the reliability, validity, and method effects of single questions (Andrews, 1984; Saris et al., 2004; Saris and Andrews, 1991). The product of the reliability and the validity can be interpreted as the explained variance in the observed variable by the variable one would like to measure. It is a measure of the total quality of a question. These MTMM experiments showed that there are considerable differences in measurement quality across countries. Because these differences in quality can cause wrong conclusions with respect to differences in relationships across countries, this paper studies the quality of the measures from the viewpoint of categorization. We assume that each category represents a range of scores on a latent continuous variable that have been grouped together, causing grouping errors. It depends on the distribution of values of the latent response variable in each category whether the intervals between the categories are equally far apart. If they are not, there is also transformation error. Both grouping and transformation are sources of measurement error due to categorization and therefore possible explanations for differences in the quality of questions. The results show that this effect is quite strong.