Do Millennials differ in terms of survey participation?

Millennials have been the focus of quite some research because of their differences with older cohorts. Besides, young respondents have been considered as a hard target population for surveys. However, to our knowledge, no research has compared the levels and types of survey participation of the Millennials versus the older generations. Using a dataset of 1,570,301 panelists of an opt-in online panel in eight countries from Europe, Latin America, and North America, we show that Millennials differ from older cohorts in terms of survey participation. Millennials show lower participation rates than older cohorts. Moreover, they present significantly higher proportions of surveys answered using smartphones. However, differences across cohorts in terms of break-offs and survey evaluation are mainly nonsignificant and/or nonmeaningful.


Introduction
define Millennials as the cohort of individuals born between 1982 and 2003. Millennials differ from other cohorts at several levels (Bowen & Chen McCain, 2015): in particular, Millennials are the first to have had, during their formative years, access to the Internet (Pew Research Center, 2014). Consequently, Millennials are the generation with the highest technology exposure (Hartman & McCambridge, 2011).
One of the main levels where Millennials seem to differ is their way to communicate. Indeed, Millennials' communicative skills have been found of lower quality than the ones of previous generations (Hartman & McCambridge, 2011). Do these differences between Millennials and the other generations also apply to their survey participation? Do these different ways of communicating affect also how they share information in surveys? Different studies found an impact of age on survey participation and the device used to complete the survey (e.g., Revilla, Toninelli, Ochoa, & Loewe, 2016). However, to the best of our knowledge, there is no study really comparing the levels and types of survey participation of the Millennials versus the older generations: Generation X (1961-1981) and Boomers (1943-60).
We try to start filling this gap, focusing on the participation within the frame of non-probability online panels, which represent the majority of online research nowadays (Baker et al., 2010). In such panels, once a person has accepted to participate, he or she regularly receives survey invitations. Each time the panelist gets an invitation, different scenarios are possible: (a) The panelist does not start the survey at all; (b) The panelist starts the survey but he or she is excluded because his or her profile does not fit the population of interest ("screened-out") or some quotas are already full; (c) The panelist starts the survey but decides at some point (it can be any moment after the start) to abandon it ("break-off"); (d) The panelist starts the survey and reaches the survey's final question. Furthermore, even when the panelist completes the survey until the end, he or she might not be satisfied with the experience he or she got while participating to this survey.
Therefore, the goal of this article is to examine within an opt-in online panel the differences between Millennials, Generation X, and Boomers, in terms of (a) participation rate, (b) break-off rate, (c) devices used to participate in the surveys, and (d) survey evaluation.

Hypotheses
Young adults have been considered a hard target population for surveys. Proposing web as a mode of data collection has been seen as a way of increasing their participation (Lugtig and Toepoel, 2016), as the Internet penetration is higher for this cohort (Pew Research Center, 2014). However, we expect that Millennials participation rate will be overall lower than those of older cohorts.
In addition, respondents may abandon the survey for several reasons, especially because they start doing something else (and then do not think/want to come back to the survey to finish it) or because they do not like it. According to the Microsoft Consumer Insights report "Attention Spans" (Microsoft Canada, 2015), Millennials are less effective at filtering out distractions. Thus, we expect more break-offs for Millennials.
Moreover, because a higher proportion of Millennials use smartphones (Pew Research Center, 2014), we expect a higher proportion of Millennials to participate into the surveys through smartphones. As previous research found higher break-off rates for mobile respondents (see, for example, Buskirk & Andrus, 2014), this, in turn, could contribute to higher break-off rates for Millennials. Besides, visibility might not be as good on smartphones due to the smaller screens and because completion times have been found to be longer for mobile respondents (see, for example, Mavletova & Couper, 2013). Thus, overall, we expect that they will evaluate the surveys more negatively.
To sum up, we want to test the following hypotheses:

H1:
The participation rate is lower for Millennials than for Generation X and Boomers.

H2:
The break-off rate is higher for Millennials than for Generation X and Boomers.
H3: The proportion of smartphone respondents is higher for Millennials than for Generation X and Boomers.
H4: Millennials evaluate overall the surveys more negatively than Generation X and Boomers.

Data
We test these four hypotheses in eight countries from different regions: Europe (Portugal, Spain), Latin/central America (Argentina, Brazil, Chile, Colombia, Mexico), and North America (United States). We use data from the Netquest online fieldwork company (www.netquest.com). In this study, we focus on the 1,570,301 panelists born between 1943 and 2003 who were considered as active at May 1, 2018, in the eight countries just mentioned (largest panels). Panelists are considered active if they have completed at least one of the last 12 surveys received. If they do not answer any of the last 12 surveys, the Netquest team sends them an email asking if they want to continue being panelists. If they answer affirmatively, they become active panelists again. For each panelist, we consider all the surveys he or she has been invited to during 1 year (May 1, 2017-May 1, 2018) to compute the participation and break-off rates and the proportions of smartphone respondents. For the survey evaluation, only the surveys programmed by Netquest can be used, as this information is not available when the surveys are programmed directly by the clients. This reduces the total N from 1,570,301 to 502,410 for the survey evaluation analyses.
Netquest sends panelists survey invitations via email, with no information about the survey itself. Overall, most surveys use quotas (Revilla, 2017) and typically respondents are not allowed to skip questions or to go back. Most surveys are optimized for mobile devices and a significant proportion of respondents use mobile devices to complete the surveys (Revilla et al., 2016).

Method/analyses
Using the data just mentioned, we compare Millennials with Generation X and Boomers within each country with data collected from May 1, 2017, to May 1, 2018, for each of these four aspects: • • The (annual) participation rate for each panelist, defined as the number of surveys he or she started during this period, divided by the number of surveys to which he or she was invited during the same time frame. Then, we compute and report the average of these individual participation rates. • • The (annual) break-off rate for each panelist, defined as the number of times he or she abandoned a survey divided by the number of times he or she started a survey. Then, we compute and report the average of these individual break-off rates. • • The (annual) rate of surveys answered with smartphones for each panelist, defined as the number of surveys he or she participated using a smartphone divided by the number of times he or she started a survey. The device used to start the survey, measured using paradata about the user-agent-strings, is the one considered (i.e., if a respondent switches to another device later on, this is not taken into account). Then, we compute and report the average of these individual rates of surveys completed through smartphones. • • The evaluation of the surveys, measured using the question that Netquest proposes at the end of each survey programmed by the company to its panelists, asking them, "Finally, what do you think about the survey? Select one to five stars to indicate if you think that the survey was (1) very badly done to (5) very well done." We compute and report the proportion of panelists with an average across all surveys of four stars or more.
Within each country, we use Z tests to compare proportions, always testing Millennials versus another cohort. Because we use a large sample, even small differences can be statistically significant. Hence, to analyze if differences are meaningful, we also compute and report the effect sizes (Cohen's h).

Participation rate
The first indicator used to compare Millennials with other cohorts was the annual participation rate. Table 1 presents these average participation rates per country and cohort.
First, there are differences across countries: European countries have higher average participation rates across all generations. The low participation rate shown by the United States may be related with the fact that it is the younger panel of all presented, so the probability of having loyal panelists is lower.
Furthermore, the average participation rate of Millennials is significantly lower than the one of other cohorts in all countries except Argentina. The differences go up to 21.7 percentage points in Mexico (between Boomers and Millennials). The effects sizes when comparing Millennials and Generation X are mostly small or very small (<.2). Compared with Boomers, effects sizes are small to medium (between .23 and .46). Hence, we find support for our first hypothesis: Millennials present significantly lower participation rates than older generations.

Break-off rate
Next, we compare the annual break-off rate. Table 2 presents the averages per cohort and country.
Latin American countries present higher average break-off rates than Spain, Portugal, and the United States. Besides, cross-country differences are similar across generations.
Moreover, Millennials have a significantly higher average break-off rate than Boomers for three countries. However, they have a lower average break-off rate than Generation X for all countries (significant at the 1% level, except for Spain). 1 In addition, all effect sizes are very small (<.2). Hence, our second hypothesis receives very little support: Millennials have higher break-off rates than Boomers in only half of the countries, and Generation X performs worse than Millennials. The asterisks in the columns "Generation X" and "Boomers" indicate a significant difference between Millennials and the respective cohort. *p < .01.

Device of participation
Moving to the device of participation, Table 3 presents the average proportion of surveys answered using smartphones, per country and cohort. Spain, Portugal, and Colombia have, in general, lower average proportions of surveys answered using smartphones. However, cross-country differences are more pronounced for Millennials than for Boomers. Furthermore, Millennials present significantly higher average proportions of surveys answered using smartphones than both Generation X and Boomers. Concretely, the proportion of surveys answered using smartphones for Millennials is between 9.9 and 25.0 percentage points higher than for Generation X and between 20.2 and 42.9 percentage points higher than for Boomers. Differences between Millennials and Generation X present small to medium effects sizes, whereas the effect sizes for the differences with Boomers range from medium (.61) in Spain to large (.94) in Portugal. Hence, differences are much higher between Millennials and Boomers, but our third hypothesis is supported for both cohorts. The asterisks in the columns "Generation X" and "Boomers" indicate a significant difference between Millennials and the respective cohort. *p < .01. The asterisks in the columns "Generation X" and "Boomers" indicate a significant difference between Millennials and the respective cohort. *p < .01.

Evaluation of the surveys
Finally, Table 4 presents the proportions of panelists with an average evaluation of four stars or more, per country and cohort. Table 4 shows mixed results. A significantly higher proportion of Millennials in Portugal, Argentina, Brazil, and the United States evaluated surveys with higher punctuations than Generation X and/or Boomers, whereas in Spain and Chile results go in the opposite direction. 2 Besides, Colombia and Mexico do not present any significant differences across cohorts. In addition, all differences present very small effect sizes (<.2). Therefore, there is no clear pattern across countries nor across cohorts. Thus, our last hypothesis is not supported.

Discussion and conclusion
From our results, we can conclude that Millennials do differ from older cohorts in terms of survey participation, even if there are fewer differences than expected. Overall, we found that on average, Millennials have a significantly lower annual participation rate than older cohorts, confirming our Hypothesis 1. Thus, new strategies to engage and incentivize this generation are needed. However, Millennials have higher break-offs rates than Boomers only in half of the countries, while Generation X tends to abandon surveys in a higher extent than Millennials. Moreover, the differences in terms of break-off are very small, providing little support for our second hypothesis. Next, Millennials on average answered a significantly higher proportion of surveys using smartphones than older cohorts in all countries, which confirms our Hypothesis 3. As expected, the sizes of the differences are higher when compared with Boomers than with Generation X. Hence, practitioners must consider that if Millennials are part of their population of interest, they will answer most of the surveys using smartphones instead of computers. Thus, if surveys are not adapted to mobile devices, data quality and survey experience for this cohort might be negatively affected. However, the high proportion of surveys that Millennials answer with smartphones open new opportunities for research. Smartphones have sensors as GPS or cameras that can be used to collect new data (e.g., Bosch, Revilla, & Paura, 2018). Furthermore, no clear pattern has been found regarding survey evaluation. Considering these mixed results and the small size of the effects, we conclude that our Hypothesis 4 cannot be supported. The asterisks in the columns "Generation X" and "Boomers" indicate a significant difference between Millennials and the respective cohort. *p < .01.
Finally, it should be considered that younger (16-24) and older (25-33) Millennials are not homogeneous. Appendix A (see supplemental material online) shows that, for all countries and indicators, younger and older millennials are significantly different. However, the sizes of the differences are in general small or very small. Besides, future research should study the variables determining these differences across cohorts (e.g., device used or past experiences).

Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental material
Supplemental material for this article is available online. Notes 1. After controlling for the rate of surveys answered by smartphone (analysis of covariance [ANCOVA]), the cohort effect is also significant when comparing Millennials and Generation X in Spain and Millennials and Boomers in Colombia. 2. After controlling for the rate of surveys answered by smartphone (ANCOVA), the cohort effect is nonsignificant when comparing Millennials and Generation X in Spain and the United States and Millennials and Boomers in Portugal. Moreover, the cohort effect is significant when comparing Millennials and Boomers in Mexico.