1Interest in online surveys is currently on the rise, which is not surprising: Their use promises considerable advantages, at least at first glance. Conducting surveys online, i.e. using Internet technology, saves time and money. Given the wide-spread shortage of resources, conducting surveys online is an intriguing idea. However, that might be deceiving, if online surveys do not yield satisfactory results that meet the established standards of empirical social research.
- 1 For a detailled discussion of advantages and disadvantages of different kinds of online survey see(...)
2Obviously, several problems come along with online surveys to date: First and foremost, online surveys suffer from a severe coverage bias. As the technique ultimately rests on Internet access, offliners are systematically excluded from online surveys. However – coverage is by no means the only problems. Sampling and the recruiting of participants is another challenge inherent in online surveys. Several approaches are used these days in online surveying to recruit participants, each linked to specific advantages and disadvantages.1
- 2 Alvarez, Sherman und van Beselaere (2003) provide an analysis of different methods of recruiting p(...)
3Here, two kinds of online surveys will be looked at in more detail. The first approach is based on self-selection (also euphemistically called "passive sampling") and represents probably the most common, but also most problematic approach to online surveying. Such open unsolicited online surveys rely on the voluntary participation of self-selected respondents: A survey is simply put on the Web, possibly promoted extensively2 – and awaits participants (see, e.g., Bandilla and Bosnjak 2000, Hauptmanns 1999). From a sampling point of view, this approach resembles that of the "Literary Digest" (leading to a fatal wrong forecast of the winner of the 1936 US presidential election) quite closely, which is ever since used as the prime example of the Don'ts in surveying. It is not surprising then that people call for active (instead of passive) sampling procedures when conducting web surveys to avoid a fatal déjà vu.
4Hence, the second one is based on pre-recruited access panels of Internet users and represents one of the more rigorous approaches to online surveying. Still, sampling is quite a challenge. Who should the participants of such actively recruited Web surveys be? As neither complete lists of Internet users nor procedures comparable to random-routes (used in offline surveys) exist in the online case, how can a random sample be drawn given such premises? One proposed solution is the use of pre-recruited panels. "Pre-recruited" means that survey institutes ask the (randomly selected) participants of their regularly conducted offline surveys whether they have Internet access at home and whether they would be willing to participate in future online surveys. If both is the case, these participants join the so-called access panel. If an online survey is commissioned, respondents are then randomly selected from this "pool of the willing" and invited (first by email, but later on also by postal mail) to take part. The idea underlying this laborious procedure is that it yields a random sample, at least of current Internet users, as it represents a multi-staged sampling process: At first, respondents are selected for offline surveys "as usual"; a random sample of Internet users should arguably be among them, which in turn are invited to join the access panel. In the final stage, respondents are randomly drawn from this pool for a given online survey. Whether this ideal type works empirically, though, is another – so far hardly tested – question. Doubts are indicated: Each stage of the multi-staged sampling process is potentially affected by systematic distortions, e.g. due to low response rates in offline representative surveys or systematic differences concerning the willingness to join an access panel or take part in online surveys. The panel-related problem of "conditioning" is another distracting factor: As pool members are interviewed on a regular basis, specific answering or learning patterns might emerge.
5However, little is still known about the effects and results stemming from these different types of surveys compared to one another, but also compared to traditional offline surveys. But such systematic comparisons of different kinds of data (offline vs. online, active vs. passive sampling) are the only way to answer the question whether these techniques yield valid results.
- 3 Given the persisting differences between the online community and the general population, this imm(...)
- 4 Similar comparisons were conducted by Bandilla, Bosnjak and Altdorfer (2001) or Berrens et al. (20(...)
6In addition, systematic comparisons provide a useful opportunity to test another claim that is often made when discussing online surveys. Advocates of online surveys claim that appropriate weighting mechanisms are capable of levelling out existing differences between online and offline surveys. To test this, offline and online surveys should be weighted in the same way (i.e. adjusted to the marginals of the same target population)3 to see whether differences in other substantial variables vanish (or at least decrease in size). The present article is supposed to provide such a systematic comparison based on two online and one offline survey.4
7The three surveys, on which the present analysis rests, were all conducted in the context of the 2002 German election.5 While similar, in large parts even identical in terms of content, their respective methodologies differ considerably, thus providing the opportunity to conduct a rigorous methodological comparison (see table 1). The first one is a traditional offline survey designed to yield a representative sample of all Germans aged 16 or older. It was fielded from August 12th to September 21st, a total of 1.665 respondents were selected by means of sample points, random routes and last birthdays and interviewed face-to-face. Its response rate was 63.8 per cent. The second survey is an Internet-based online survey of 1.165 German Internet users. As they were selected from the previously in offline surveys recruited access panel of a survey institute, it should render a representative sample of German Internet users. Field time in this case lasted from September 13th to October 1st,6 the respective response rate was 74.2 per cent. Finally, the third survey is also an Internet-based online survey that could be accessed by anyone without restrictions at www.wahlumfrage2002.de (which stands for electionsurvey2002). In other words, respondents recruited themselves; they could do so from August 20th to September 22nd and a total of 34.098 made us of it. Restricting respondents to only those that gave at least five valid answers leaves 29.583 participants that comprise the third element for the following comparison of unweighted samples.7
- 5 Other countries have also seen the advent of virtual electoral research, at least as supplements t(...)
- 6 In other words, this survey includes pre- and post-election interviews. However, this does not aff(...)
- 7 For further details concerning this survey see Faas (2003).
Table 1: Details concerning the three surveys
PAPI = Paper and Pencil Interview, CASI = Computer Assisted Self-Administered Interview
- 8 The weighting was based on the following age groups: 16/17, 18 thru 24, 25 thru 34, 35 thru 44, 45(...)
8Since we are also interested in the effects of weighting, the three surveys are also weighted. After weighting, the three survey resemble the (common) distributions of age and sex in the German population aged 16 and over, as it is known from official statistics.8 It can thus be tested whether emerging (unweighted) differences can be (mainly) attributed to socio-demographic biases or whether they go beyond that. In the former case, differences should disappear after weighting; in the latter, they should prevail. From a substantial point of view, the following comparison will comprise socio-demographic (age, sex, education), but also substantial variables (voting intentions, interest in politics).
9Comparing the three surveys with respect to their socio-demographic composition yields considerable differences concerning the marginal distributions of sex, age and education between them (see table 2). Looked at in more detail, one can first of all see that the well-known distortions of online users in relation to the population as a whole are once again confirmed: Online users are younger, better educated and more often male. This pattern is especially pronounced in the case of the open unsolicited survey: Among its self-selected respondents, 77.9 per cent are male! The comparable figures are 58.8 per cent (for the access panel, i.e. Internet users) and only 51.5 per cent for the general survey. The same tendencies apply to the age pyramid: An average respondent of the open online survey is only 32.8 years old. This figure rises to (still only) 36.7 years in case of the access panel, but 50.0 years for the general survey. The respective shares of the youngest and oldest respondents are especially telling: 30 per cent of the respondents of "wahlumfrage2002" were between 16 and 24 years old, but only 4.2 per cent were over 60. For the general survey, the findings are almost a mirror-image of that: 9.3 per cent are juniors, 35.8 per cent are seniors. The most remarkable differences emerge from a comparison of the respective educational levels. More than two thirds of the respondents of the open online survey have the "Abitur" (i.e. are entitled to go to university); this applies to only 44 per cent of the access panellists and less than a third of the general population. The prototype of respondents of the open online poll is thus male, young and highly educated – and that is true in a much more pronounced way than it is already true for the average Internet user when compared to the average German.
Table 2: Socio-demographic composition of the three surveys (unweighted data)
10East German respondents were purposely over-represented in the face-to-face survey. This was corrected using a simple weight reflecting the ratio of East to West German inhabitants.
11Given the size of these differences, it is not surprising that they are all highly significant (see table 3). Taking a closer look at the measures of associations, one can furthermore see that the two online surveys are – at least in terms of their respective socio-demographic composition – most similar, while open online survey and general survey are (with the exception of age) furthest apart from each other.
- 9 About half of access panelists were invited to participate in the survey after election day. For t(...)
- 10 According to the official result of the election, Social Democrats and Christian Democrats each re(...)
12Leaving socio-demographics behind, substantial variables yield a very similar picture (see table 4). Remarkable differences still prevail. Looking at voting intentions,9 it should first of all be pointed out that the distribution of voting intentions in the representative general population survey matches the official result of the 2002 German election quite closely. This provides some confidence that this really is a representative population survey.10
Table 3: Size and significance of the observed socio-demographic differences between the three surveys (2 and Cramers V, unweighted data)
Table 4: Voting Intentions and interest in politics of the respondents of the three surveys (unweighted data)
Table 4 (suite)
East German respondents were purposely over-represented in the face-to-face survey. This was corrected using a simple weight reflecting the ratio of East to West German inhabitants.
13But how do the results obtained from the two online surveys differ from that? Both online surveys have in common that the Christian Democrats' vote share crashes: Less than a quarter of the respondents expresses a voting intention for them. In addition, both online surveys contain biases that favour the small parties (Liberals and Greens), although the extent varies between the two. The bonus is much more pronounced in the open online survey: More than 20 per cent of the respondents of the open online survey intend to vote for the Greens (compared to "only" 14.1. per cent of the access panellists); for the liberals, the pattern is similar, the respective figures are 18.1 and 11.4 per cent. Finally looking at the Social Democrats, the results are puzzling. Their best result stems from the access panel; here, they fare even better than they finally did in the general election. However, in the open online poll, they come off almost as bad as the Christian Democrats, receiving only 29.3 per cent. Looking at the picture in terms of coalitions (i.e. Social Democrats and Greens on the one hand, Christian Democrats and Liberals on the other), both online surveys contain an obvious pro-government bias, mostly at the cost of the Christian Democrats.
14Apart from voting intentions, interest in politics is another substantial variable of interest here. Once again, the emerging deviations, especially concerning open online survey on the one hand, general survey on the other, are striking. The respondents of the former are highly involved and engaged in politics and clearly distinctive to the rest of the population in this respect. Three out of four indicate that their interest in politics is strong or very strong – the respective figures for the other surveys are 40 per cent (access panel) and 33 per cent (general survey). Thus, it is once again the case that biases that occur when moving from the offline to the online world are further exaggerated in the case of the open online poll.
15The observed substantial differences are significant throughout (see table 5). Concerning the voting intentions, it is also again true that the two online surveys are most similar (as Cramers V indicates), while with respect to interest in politics, the open online survey with its remarkably high level of interest is clearly the outlier.
Table 5: Size and significance of the observed substantial differences between the three surveys (2 and Cramers V, unweighted data)
16So far we have seen two things: First, self-selection contains a tremendous potential for bias. Mainly politically interested and involved people took part in the open online poll "wahlumfrage2002". The extent of the resulting distortions is striking. Secondly: Even though around 50 per cent of the German population have Internet access nowadays, considerable differences are also present when the general survey is compared to the survey based on an access panel (which is designed to yield representative results for Internet users). Obviously, great caution is indicated when looking at results of online surveys, at least if it is claimed that they yield representative results for the population as a whole. Advocates of online surveys could object that weighting mechanisms are a solution to these problems. We will take a look at that now.
17When comparing weighted results, it is, of course, redundant to look at socio-demographics as they (except for education) are used to weight the data. We can thus restrict the analysis to substantial variables, i.e. voting intentions and interest in politics. Before we do so, however, it is worth taking a look at the individual weights that are necessary to adjust the data. For the general survey, the minimum (individual) weight is 0.053, the maximum one is 7.061; the lower quartile of the distribution of weights is 0.531; the upper one is 1.299. These scores seem quite "normal". However, things change as soon as we move to the online world. For the access panel, the minimum weight is 0.256, the maximum one amounts to a remarkable 20.321! The lower quartile is 0.568, the upper one 0.894. The respective scores for the open online poll are 0.160 and 44.670 (!), 0.317 and 0.994 for the quartiles. Obviously, more adjusting is necessary for both online surveys. In these cases, specific individuals are assigned huge weights (as the upper values for the weights show); certain groups are obviously heavily under-represented – identifying them shows that it is especially the group of elderly women.
18Looking at the weighted results (table 6), one can easily see that the differences have not vanished after the weighting; quite contrary, they are hardly affected and still highly significant (see table 7). Concerning voting intentions, the most pronounced change caused by the weighting of the general survey is that the vote shares of the two major parties move closer together: Christian Democrats gain 0.8 percentage points, Social Democrats lose 0.6 percentage points. There are hardly any changes for other parties. Interest in politics is also only slightly affected by the weighting: It drops slightly by 0.1 percentage points.
Table 6: Voting Intentions and interest in politics of the respondents of the three surveys, (weighted data, changes compared to unweighted results in parentheses)
Table 6 (suite)
Table 7: Size and significance of the observed substantial differences between the three surveys (2 and Cramers V, weighted data)
19Looking at the access panel, the weighting-induced changes are more pronounced. The vote shares of Social Democrats and other parties drop by 2.6 and 1.0 percentage points respectively, while the vote shares of the Liberals and Socialists rise by 1.7 and 1.9 points. However, there is no uniform improvement in terms of predicting vote shares. While the Social Democrats' weighted vote share, in fact, perfectly matches their real one, the changes caused for Liberals and Socialists lead to an even worse forecast. In addition, the Christian Democrat's vote share – already way off the mark without weighting – is only slightly affected by the weighting and 23.8 per cent are an even worse predictor for their real vote share.
20In the case of the access panel, interest in politics is also more strongly affected by the weighting. However, the effect is again rather dubious. As we have seen from the unweighted results, interest in politics was already very high in that survey compared to the general survey. Now, the effect of weighting is that interest rises even further, while the weighting of the general survey leads to a slightly lower average interest in politics there. In other words, weighting does not decrease, but rather increases differences between the surveys!
21Weighting also affects the results obtained from the open online poll, but the effects are again not in a uniform (and improving) direction. The vote shares of Social Democrats (+2.2 points), Liberals (-2.7 points) and Socialists (+1.1 points) are most strongly affects, while the vote share for the Christian Democrats is again hardly affected. That said, one has to conclude that electoral forecasting based on unweighted or socio-demographically weighted online samples is – at least so far – hardly of any value. Finally, interest in politics is only marginally affected. Even after harmonizing socio-demographic variables, it is still true that 75 per cent of the respondents of the open online survey have a strong or very strong interest in politics. In other words, the differences between the three surveys are not rooted in their differing socio-demographic composition. The weighting has not lead to the result aimed at; the results stemming from the three surveys have not moved closer together. As table 7 indicates, some differences have become smaller, others larger; overall, they still prevail and are still highly significant.
22Expectations concerning online surveys are huge – but can they be met? The results of the present analysis are rather disenchanting. The comparison of two online and one offline survey has revealed that they yield very different marginal distributions of socio-demographic, but also substantial variables. It was clearly shown that especially the open online survey based on self-selected respondents produces strongly biased results. Due to the self-selection this survey is systematically biased towards young, educated people with a high interest in politics and a significantly different voting behaviour. The same tendencies also apply to the comparison of the general survey and the representative survey of Internet users. Moreover, all these differences in substantial terms prevail after the samples were socio-demographically weighted (by age and sex).