Two approaches to evaluate measurement quality in online surveys : an application using the Norwegian Citizen Panel

Abstract

Previous research has shown that measurement errors are large in data collected through surveys. However, the size of these errors varies depending on the language, the mode of data collection, the question‟s characteristics, etc. Information about the size of these errors, or their complement, the measurement quality, can be used for designing better questionnaires, and to correct for measurement errors in substantive analyses. Getting further information about the measurement quality is therefore crucial. However, little is known about the measurement quality of questions in web surveys. Thus, in this paper, we evaluate the measurement quality of a set of survey questions from the Norwegian Citizen Panel; one of the few probability-based online panels existing at this day. To do so, we use two different approaches: (1) we implement two Multitrait-Multimethod experiments in this panel‟s 5th wave, and (2) we predict the quality of the questions using the software Survey Quality Predictor. Overall, measurements‟ quality is quite high (between 0.60 and 0.89), even if comparing the results of the two approaches, we observe some minor differences in quality. In addition, using 11-point scales with two fixed reference points, usually, provides the highest quality.