Impact of formulating a careful introduction and ask respondents to commit themselves on the quality in a web panel survey

Abstract

Web surveys are becoming every day more present in survey research. However, there are still problems with these surveys that have not been solved yet, in particular in the case of online panels using economic incentives. Some people can participate in order to get the reward and without any intention of answering properly and sincerely to the surveys. Speeding and low quality data may be expected in that case. One way to try to face these undesirable behaviors is to sensibilize the respondents to the importance of their answers for research and the necessity for the conclusions to be valuable that they are answering carefully. The sensibilization can be done through motivational messages. This paper reports the results of an experimental design in which respondents from the online panel Netquest in Spain were randomly assigned to three groups: one control group, one group getting a carefully formulated introduction that aimed to motivate respondents in making an effort to answer properly, and one group getting the same introduction but with an additional commitment stating that the respondents could accept or reject by clicking on a button. The results show no effect when showing only the introduction. When the introduction is combined with the commitment, a small effect is found on some behaviors but not all. It seems to come only from some specific respondents, who were answering to the survey by doing some effort but not much and can then be moved to make a higher level of effort. But respondents that behave badly continue behaving badly even when they commit to do their best. It suggests that more radical solutions than a simple sensibilization may be necessary to reduce the undesirable behaviors of these online panelists.