Measurement quality of 67 common social sciences questions across countries and languages based on 28 Multitrait-Multimethod experiments implemented in the European Social Survey

Abstract

Survey data is used in many social science studies. The measurement quality of these data is crucial as it determines the accuracy of the information on which these studies are based. Besides, since these studies are used to provide insights to key political and social actors, it also determines the accuracy of the information on which crucial decisions are based. In this paper, we estimated the measurement quality (proportion of the variance of the observed survey responses explained by the latent trait of interest) of 67 common social sciences questions that were part of Multitrait-Multimethod experiments in the seven first rounds of the European Social Survey. These questions were asked using response scales with different characteristics and in up to 41 country-language groups. Our results show that measurement errors are omnipresent: the average measurement quality across all questions is 0.65. Thus, overall, on average 35% of the variance in the observed survey answers can be attributed to measurement errors. Furthermore, the size of errors varies across questions as well as across country-language groups. The questions’ average measurement quality across all country-language groups ranges from 0.25 to 0.88, depending on the response scale and topic, and the country-language groups’ average measurement quality across questions ranges from 0.52 to 0.76. Thus, the impact of measurement errors on applied research can be different depending on the exact question formulation and response scale used as well as on the country and language of interest. Consequently, in each study, researchers should consider assessing the size of the measurement errors of their variables and how this affects their results.