- * Revised version of a paper originally presented at the 57th Annual Conference of the American Asso (...)
1In a famous split-ballot experiment originally published in 1941 Daniel Rugg tested for differences in responses when either the words "allow" or "forbid" were used. In one version, the wording was "Do you think that the United States should allow public speeches against democracy?"; in the other version, it was "Do you think that the United States should forbid public speeches against democracy?". In the first version, 21% of respondents opted for the permission of speeches against democracy and 62% denied it. In the second version, 39% were in favor of permission and 46% for prohibition. 17%, respectively 15%, were undecided or without opinion (Rugg 1941).
2The contrast between a rate of 62% and 46% for logically the same response of denial has made this example one of the most often-cited in the international literature. It has been cited as a example of the importance of question wording, and is routinely referred to in standard textbooks in methodology. Later replications in the USA (Schuman and Presser 1981), Germany (Hippler and Schwarz 1986), Belgium (Waterplas et al. 1988, Loosveldt 1997), the Netherlands (Holleman 2000) and New Zealand (Glendall and Hoek 1990) have proven the stability of the effect across time and country. Moreover, similar effects were not only obtained in direct replications of the original wordings, but also in replications using other examples (see Schuman and Presser 1981, Holleman 1999).
3However popular the examples are in demonstrating that logically identical wordings may produce different responses when being worded in a slightly different way, a basic question remains: whether the effect exists because of the difference in wording or because response alternatives are not explicitly stated. According to long-established knowledge in survey literature, it is reasonable to make the response alternatives explicit (e.g., Cantril 1947: 35, Sudman and Bradburn 1982: 139). Otherwise, the danger of response bias exists: Respondents, especially those with poorly-crystallised attitudes, may agree to the explicitly-specified option and neglect the implicit one. They might, in other words, show an acquiescence effect. Could it be, as has been argued elsewhere (Reuband 2001), that the main reason for the large response effect in the allow-forbid question-wording example is due to the negligence of the standard rule? In fact, Stanley Payne, in his classical treatise on "The art of asking questions" (1951), was suspicious and remarked that it would be useful to know the results with the response alternatives of the allow-forbid question specified (1951: 57). However, no one has taken up this suggestion.
4It was only recently that the issue of whether or not the lack of specified alternatives has an impact on the responses was taken up in research. In a split-ballot experiment in a West German metropolitan town, with 616 survey respondents, the effect was tested by using a question (on a different topic than Rugg's original research) that explicitly specified the alternatives and varied their verbal labels. "Should legislators allow or not allow organisations that are hostile to foreigners" vs. "Should the legislators allow or forbid organisations that are hostile to foreigners"? The effect, that was originally described by Rugg, was replicated, but the differences were much smaller than the original ones: Whereas 8% opted for allowance in the "allow-not allow" version, 14% opted for allowance in the "allow-forbid" version. 82% and 78%, respectively, decided for not allowance/forbiddance, 8-10% gave other responses, including "Don't know, undecided" (Reuband 2001). One reason for the small differences in percentages – in contrast to the Rugg example - could have been the skewed distribution of the responses, creating upper limits to variations and ceiling effects. But the main reason may have been that the alternatives were explicitly specified in the question wording.
- 1 The interviewers who were social science students had to administer five questionnaires, each one (...)
- 2 The quota criteria were sex, age (18-29, 30-44, 45-59, 60+) and education (we preferred not having (...)
5In the following, the question of wording effects is taken up by replicating the original Rugg experiment in a German setting and by extending it through modifications as in the aforementioned experiment. Similar to the Rugg case, the wording was framed as to whether "public speeches against democracy should be allowed" vs. "should be forbidden". In two additional versions, the alternatives were explicitly specified, but the labels were varied ("allow or not allow" vs. "allow or forbid"). The third response option that was available to the interviewers (as in the original version) was "Don't know, no answer". It was usually not read to the respondents, apart from a few interviews where this was done due to a mistake in the questionnaire design (later corrected). No major substantial effect concerning our findings resulted from this, as further analysis proved. The four different splits were distributed randomly across the respondents.1 To include people who usually do not participate in surveys, but might do so for social obligations, the interviewers were allowed to include among their interviewees people with whom they were also closely or loosely acquainted (such as neighbours). This happened in 43% of the interviews. In the rest of the cases, actually the majority, the respondent was unknown to the interviewer. Based on a quota sample, all in all 620 people were interviewed in the city of Düsseldorf and its surrounding region in 2001.2
6We first test whether the lack of alternatives in the original question wording has an impact on responses and whether the use of the words "allow" vs. "not allow" or "forbid" makes a difference. Then we test to what extent the level of education has an impact on the response effect.
7The responses to the four versions of question wordings are listed in table 1. The original responses to the wording by Rugg, transformed to the German context, are found in columns A and B. When respondents are asked whether one should "allow" public speech against democracy (version A) 41% of the respondents agree. When they are asked whether one should "forbid" public speech against democracy (version B), the percentage of respondents who prefer allowance increases to 72%. The respective rate of people who perceive public speech as non-acceptable drops from 47% to 12%.
Table 1: Allow or forbid public speech against democracy according to question wording (in %)
8(under preparation/en préparation)
Question wording: A= "Do you think that in Germany one should allow public speech against democracy?"; B= "should forbid ..."; C= "should allow or not allow ..."; D= "should allow or forbid ..." (A= "Sollte man Ihrer Ansicht nach in der Bundesrepublik öffentliche Reden gegen Demokratie ... erlauben?", B = "... verbieten?", C = "... erlauben oder nicht erlauben?", D = "... erlauben oder verbieten?"). The four versions of question wording were distributed randomly among the interviewers. Since some, by sheer accident, tended to prefer one set of questionnaires over the others, the number of interviewers per wording version varies slightly (between N=142 and N=164). This does not effect the results in any way and is of no importance here.
9The original Rugg experiment is thus confirmed by the data. In fact, the confirmation is even stronger than in the original US study. Whereas in Rugg's example, the percentage differences for "not allow/forbid" is 16 points, it is 35 points in our case. The difference in our case is statistically significant (Cramérs V=.39).
10In contrast to what is usually assumed in the literature about one-sided question wording (e.g., Cantril 1946: 46, Noelle-Neumann and Petersen 2000: 195), in both the Rugg study and our own, it is not the specific, one-sided-worded response category that gets the higher percentage. If this were the case, one should have expected the one sided "forbid" version – compared to the one sided worded "allow" version – to produce the higher rate of respondents who deny the right of public speech against democracy. But this turned out not be the case. The result is a first sign that the meaning of the specific wording might have an impact. But the evidence is not sufficient since each of the wordings leave open the respective response alternatives and thus violate the standard rules of question wording.
11In the modified version, where the alternatives are specified (C vs. D), we find an effect that is basically similar in direction – with less people opting for "not allow/forbid" in the "forbid" version (D) in comparison to the "not allow" version (C). But the difference in percentages is much smaller than in the original version; it amounts to 11 points only for the "allow" response (Cramérs V=.12) and does not reach statistical significance. Nonetheless, it is in agreement, also in effect size, with the above-mentioned German study in which similar options were used on a different topic. Taken together, the results make it clear that a major reason for the wording effect in the original version is the lack of specified response alternatives. The use of the specific wording itself – "(not) allow or forbid" – has less an impact on the overall results.
12Furthermore, it becomes evident that the difference between the one-sided version of "allow" and the two-sided version of "allow or not allow" (Columns A vs. C in table 1) is relatively small, seen in percentages. But the majority position changes. Whereas in the one-sided "allow" version (A), respondents who reject this option constitute a majority, they shrink to a minority in the other case, as they do in the two other versions. This tendency becomes even greater once the term "forbid" rather than "not allow" enters into the wording (A vs. D). In contrast to original assumptions about "yes"-saying tendencies when responses are stated in a one-sided manner, it is not the one-sided question wording that receives the greatest support for the "allow" option, but the version where the alternatives are explicitly mentioned. This can be seen not only in the one-sided-worded version A compared to the two-sided C ("allow" vs. "allow or not allow"; allow: 41% vs. 45%), but also in the comparison of the one-sided-worded version B to the two-sided version D ("forbid" vs. "allow or forbid"; forbid 12% vs. 28%). It seems as if people tend to reject the notion of permission, yet also feel uncomfortable in expressing explicitly any version of denial. They seem to be inclined toward some intermediate position which might also be the reason why the proportion of undecided responses increases, once the term "not allow" or "forbid" enters the formulation.
13It is not known to what extent these perplexing findings are due to the substantial complexity of the question and should be seen as a sign of bewilderment on part of the respondent. Without doubt, the question on free speech against democracy represents a major cognitive problem for respondents since free speech is generally considered as basic element of democracy (see Noelle-Neumann and Köcher 1993:558) and not allowing speeches against democracy can therefore be interpreted as a violation of democratic ideals. On the other hand, allowing groups to argue and work against democracy might finally result in the abolishment of democracy (as Nazi Germany has shown). Seen from this perspective, the effort to safeguard democracy should include prohibiting speeches against democracy. This logic has in fact been the reason why in Germany the constitutional court can forbid (and has actually forbidden) parties whose aim it is to abolish the democratic state. It might be these specific historical circumstances which account for the much greater response effects in the German study compared to the US case.
14Given the described perplexities in question formulation, some of the findings of earlier research must be seen in a different light: When people with poorly-crystallized attitudes and little education show the strongest effects in the "allow-forbid" question, as some studies have demonstrated (e.g., Hippler and Schwarz 1986, Waterplas et al. 1988, Narayan and Krosnick 1996, Loosveldt 1997), this effect may not necessarily be due to the specific words used but could equally well be due to the one-sided wording of the question: Since people with poorly-crystallized attitudes and little education have proven to be especially susceptible to wording where the alternatives are not explicitly specified (e.g., Esser 1974: 128, Martin 1983: 173f., DeMaio 1984: 273, Schräpler 1996: 56). But the complexity of the problem does not end here: if understanding of each wording ("not allow" vs. "forbid") works the other way round – the more sophisticated making finer distinctions between these verbal labels, as the earlier German study suggests (Reuband 2001) – complex interaction patterns and mixtures of responses are conceivable. Therefore, we need a fresh look at what "allow" or "forbid" means for groups with different levels of sophistication in the context of questions that follow the rules and specify the alternatives.
15In order to test the effect of comprehension due to education, we have divided respondents according to level of education. As can be seen in table 2 (columns A vs. B), there is virtually no difference between people with high or lower education when it comes to the original version of the question. In both groups, more people prefer the implicit "not allow" to the explicit "forbid" version. The differences in percentages are similar and Cramérs V turns out to be identical (.39). If we exclude the undecided, this would not alter the conclusions (Cramérs V will be .42, respectively .41). In this respect, the findings differ from earlier US studies where somewhat stronger response effects were found among the lower educated (Schuman and Presser 1981). Whether or not this is due to a difference in history or in the understanding of what forbidding free speech means in each of the countries remains an open question.
Table 2: Allow or forbid public speech against democracy according to education and question wording (in %)
16(under preparation/en préparation)
- 3 The statistical significance is of the .06 level. If we exclude the undecided, Cramérs V reaches s (...)
17When we turn next to the version where the alternatives are specified (columns C vs. D), a different picture emerges. Whereas the different versions ("allow or not allow" vs. "allow or forbid") hardly produce different responses among the lower educated, they do so among the higher educated, totalling up to 16 points for the response category "allow". Cramérs V is .19 and nearly reaches significance at the 0.5 level.3 Apparently the better educated make a finer distinction in the use of the words "not allow" or "forbid" than the lesser educated. In this respect, they reproduce the findings of the earlier German experiment that resembled version C and D and was based on a somewhat different topic, namely whether anti-foreigner organisations should be allowed or not (Reuband 2001).
18When the findings in table 2 on the different sets of question wordings are compared to each other for each level of education, it can be inferred that the differences among the lower educated in the original question-wording case (A vs. B) is heavily influenced by the lack of specified alternatives. If the alternatives were specified in the "milder" way ("allow or not allow", as in column C), the percentage of people who prefer permission would be 47% rather than 38%, and if they were specified in a "harsher" way ("allow or forbid"), it would be 50% (i.e., a 12 point difference with A). The stand-alone version of pure prohibition (B) reaches the highest percentage: 67% (i.e., a 29 point difference with A). If the original "allow-forbid" versions (A vs. B) and its percentage difference is taken as a reference the figures suggest that the major reason for the difference lies in the one-sided nature of the wording: since the use of the word "forbid" (in conjunction with "allow", as in C) makes for only a 12 point difference out of a total difference of 23 points, whereas the one-sided wording makes for a 17 point difference.
19Among the better educated, the situation differs. Whether the alternative is specified in the "milder" way ("allow or not allow", as in C), or is not mentioned at all, does not make a difference to the original one-sided allow version (see A vs. C). However, if the "forbid" alternative is introduced instead of "not allow" (as in D), the divergence increases. And in contrast to the lower educated, it also does so when the two versions with the alternatives stated (C and D) are compared. Compared to version A, version D produces a difference of 18 points concerning permission, B compared to A of 33 points. So among both the lower and better educated, the wording chosen makes a difference. But whereas the lower educated are primarily influenced by a yes-saying tendency, among the higher educated it is the verbal labels which account most for the results.
20The most famous example of question-wording effects – the allow-forbid question – deserves a new perspective. Rather than reflecting logical contradictions due to choice of wording, our findings demonstrate the importance of alternatives being specified explicitly. Furthermore, data indicate that the relationship between cognitive sophistication and response effects needs a revision in interpretation. In the past, it was found that the less sophisticated – the less educated and those with little interest – were most influenced by the difference in question wording. It has now become evident that it is not the wording itself but the lack of correctly-specified response alternatives that plays a predominant role in this effect.
21Nonetheless, it can also be shown that the wording itself exerts an influence on responses, though not as strong as formerly assumed in the literature. Especially the better educated – and not as formerly assumed, the less educated – respondents are affected by the kind of verbal labels used. Higher-educated respondents seem to make finer verbal distinctions: They are more inclined to opt for "not allow" rather than for "forbid". Above all, "not allowing" speech against democracy probably entails, in the public's mind, informal social-control mechanisms or other measures of extra-legal control that are less strict and rigid than legal ones. The "forbidding" of speech on the other hand might be seen by them as some kind of legal intervention.