Evaluation procedures for survey questions

Abstract

In this paper different criteria for the choice of an evaluation procedure for survey questions are discussed. Firstly, we mention a practical criterion: the amount of data collection the procedures require. Secondly, we suggest the distinction between subjective judgments and evaluations based on empirical data. Thirdly , we suggest that it would be attractive if the procedure could evaluate the following aspects of the questions: 1. The relationship between the concept to be measured and the question specified, 2. The effects of the form of the question on the quality of the question with respect to: a. the complexity of the formulation, b. the precision, c. possible method effects, d. many other characteristics, 3. The social desirability of some of the response categories. Besides that, it would be attractive if the procedure could indicate the effect of the lack of the knowledge of the respondents about the topic on the respondents answers. We compare 13 procedures for the evaluation of questions with respect to these criteria and will derive some conclusions from this overview.