- * In alphabetic order. The authors sincerely thank the organizers and participants of the 2002 inter (...)
1In the past decades, nonresponse, that is, nonparticipation in surveys appears to be increasing in Europe and in the United States (De Heer, 1999; Bradburn, 1992). This trend has been sufficiently alarming to alert survey researchers all over the world, which led to a gathering of dedicated survey methodologists and statisticians in 1990 when the first workshop on household survey nonresponse was held in Stockholm. Since 1990, the workshop has met yearly, each year in a different country (De Leeuw, 1999).
2In the early years of the workshop, the emphasis was on description: how high is nonresponse in different countries, what methods do research institutes and statistical agencies use to reduce nonresponse and to adjust for nonresponse? This resulted among others in an international questionnaire on nonresponse (de Heer, 1999). Due to these efforts we now know that nonresponse overall has been growing over the years and that in some countries nonresponse is a far greater problem than in other countries (De Leeuw & De Heer, 2002).
3In later years the focus of the workshop shifted to the understanding of nonresponse, including the development of theories on survey participation, and the search for relationships and causal mechanisms of nonresponse (e.g., Groves & Couper, 1998; Groves, Cialdini & Couper, 1992; Hox, de Leeuw & Vorst, 1996). This has led to a network of related research projects on interviewer attitudes and behavior (Hox et al., 2002), interviewer-respondent interactions (Campanelli, Sturgis & Purdon, 1997), the use of incentives (Singer et al, 1999, 2002), and other factors in survey design and fieldwork procedures. Several specialized books have been published that reflect some of the work discussed at the workshop (e.g., Morton-Williams, 1993; Laaksonen, 1996; Japec et al, 1997; Campanelli, et al, 1997; Koch & Porst, 1998, Groves & Couper, 1998).
4The 10th anniversary of the workshop took place in Portland, Oregon, 1999, where based on this decade of nonresponse research a large international conference was organized on survey nonresponse. The invited papers of this conference were published in a monograph edited by Groves, Dillman, Eltinge & Little (2002), which describes the sate of the art in nonresponse reduction, nonresponse adjustment, and nonresponse error measurements. How to proceed from there? To answer this question we conducted a structured focus group with experts in the field. In this paper we describe the future research agenda for nonresponse as seen by a group of experienced survey methodologists and statisticians. Our main goal was to identify and prioritize potential interesting research topics in the field of nonresponse and suggest a research agenda for the next years.
5We used a highly structured form of qualitative research, called concept mapping, to extract information. This technique, which resembles very structured focus groups, was developed by Trochim (1989) and has been successfully applied in diverse fields like policy making, planning, evaluation research, and interviewer debriefing (e.g., Dunn, 1981; Hox, de Leeuw & Snijkers, 1998. Snijkers, Hox & de Leeuw, 1999). The major advantage of concept mapping is that it quickly leads from fuzzy and dispersed knowledge to one interpretable conceptual framework, in this case a future research agenda. Furthermore, the framework can be comprehensibly expressed in a graphical representation, which shows all major ideas and their interrelationships.
6Trochim (1989) describes concept mapping as a structured conceptualization process in order to represent ideas in the form of a picture or map. To construct the map, first ideas have to be generated in a focus group, then ideas have to be prioritized (e.g., by rating methods), and the interrelationships between the ideas have to be made explicit (e.g., by sorting procedures). In the next step multidimensional scaling and cluster analysis are used to depict the information, that was generated by the group, in a map format. In the last phase, the group interprets the map and discusses the results of the analysis, and decide how this information may be utilized.
7In the next section we first give a short description of the group of experienced survey methodologists and statisticians who acted as informants and we outline the procedures used in concept mapping. We continue with the major results and end with a research agenda for the future.
8At the 13th international Workshop on Household Surveys Non-response in Copenhagen, 29-31 August 2002, 46 survey methodologists and statisticians gathered. There were representatives from 12 countries (Belgium, Denmark, Finland, Germany, Hungary, Israel, The Netherlands, Norway, Slovenia, Sweden, UK, USA). Their affiliations varied; there were survey specialist from universities, official statistical agencies, and public opinion research firms. Also, the specialization varied and represented the whole survey process.
9The concept mapping took place during two session: the first to collect the data, and the second to discuss the results and interpret the map. Not all workshop participants could attend these two sessions; in total 34 survey specialist (74 %) provided us with data.
10Concept mapping is not a standard focus group; it is a highly structured set of procedures for data collection, analysis, and interpretation. In the data collection phase a mixture of the following techniques are used: structured brainstorm, rating, and sorting. For the data-analysis both multidimensional scaling and cluster analysis techniques are used. In the final interpretation stage the results of the analyses are fed back to the original participants and discussed with them.
11Concept mapping in focus groups consists of five steps: (1) preparation and developing the focus, (2) statement generation by the group, (3) statement structuring and rating by the group, (4) statistical analysis and statement representation as a cluster tree and concept map, and (5) interpretation of the results by the group.
12Step 1, the preparation phase, should result in two separate products: the primary focus or domain of interest for the brainstorming session with the focus group, and the rating scale needed for the structuring of statements in step 3. We decided on the following focus for the brainstorming session: "What should our research agenda be over the next four years? What are necessary theoretical developments, empirical studies, and/or collaboration. Include both nonresponse reducing (data collection) and (statistical) adjusting". The rating focus concerned the immediate importance of the research topics, and was stated as follows: "For each topic mentioned, give a rating of its importance. Use the following response categories:
13Step 2 consists of statement generation. During a 1½ hour brainstorming session statements were generated by the members of the focus group. A brief introduction first described the concept mapping method and outlined the procedures. The focus statement described above constituted the prompt for the brainstorming. In introducing the focus we stated "you are all experienced researchers, what would you like to hear reported at the next workshop, when you come back to your office what studies would you like to initiate".
14The usual rules for brainstorming applied, such as, encourage lots of statements, and emphasize the importance of no criticism or discussion during the generation of statements. The statements were recorded on a whiteboard by the moderator. The wording was checked with the group members, and if necessary the text was adjusted. The final text was written on small index cards by all participants and entered into a laptop computer by one of the team members.
15In step 3, structuring, the individual participants were instructed to rate the statements as to effectiveness, using the five‑point rating scale described above. Members of the focus group were asked to imagine having a limited budget, asking themselves what research they would subsidize. After this rating the individual participants were asked to sort the cards which the rated statements into different piles ' the way it makes sense to you'. Restrictions were: each statement can only be placed in one pile, all statements may not be put in one large single pile, and all statements may not be put into a pile of one, although a small number of piles of one statement are allowed.
16Step 4 is the analysis or "statement representation" phase. The individual sorts were combined into a group similarity matrix. This similarity matrix is the input for a multidimensional scaling (MDS) procedure and cluster analysis. The two‑dimensional plot of points created by the MDS may be viewed as a representation of the 'emerging concepts' of group knowledge, hence the name concept mapping. The cluster solution is superimposed on the map of points to facilitate interpretation by the group members. Furthermore, the mean group ratings for each statement are computed. It is possible to overlay the ratings onto the concept map.
17Step 5 is again a group activity. The participants discussed possible meanings and acceptable names for each cluster of statements. This last step attempts to identify relations between tactics in the form of a group‑approved map.
18The brainstorming resulted in 20 different statements. Each statement was thought to be an important research topic by at least one group member. Each statement was individually rated on immediate importance. Table I lists the statements in order of perceived importance; given is the average group rating (original scale: l, 2, 3, 4,5), the text of generated statement, and the number of order in which statements were generated.
Table 1: Most important research topics: statements in order of average (mean) rating
19When we look at Table 1, we should keep in mind that the researchers were asked to generate important research topics for future research in the field of non-response. Every statement in the table is therefore important in the opinion of at least one experienced non-response expert. This does not mean that everybody completely agrees on every statement, there is some variance among the participants, as is shown by the standard deviations (presented in Appendix A). The average ratings indicate the relative importance of each statement.
20Looking at list of statements we notice that research topics concerning both adjustment and reduction of non-response are mentioned. This reflects the composition of the group of researchers. Both issues are considered important. And, furthermore, a number of statements focus on cross-cultural, cross-national and non-western world issues. With respect to their average ratings, however, these statements are rated less important. A more detailed interpretation of the data is given in the next subsection, discussing the results of the sorting.
21Cluster analyses based on the similarity matrix of the sortings, resulted in six clusters. In the second session clusters were discussed and named by the group. Cluster names and average importance ratings based on researchers opinion. The resulting concept map is depicted in Appendix B. Table 2 lists the statements grouped by named cluster; for each cluster the average cluster rating on effectiveness is given in parentheses. The clusters are ordered in descending effectiveness. Most effective clusters are named first.
Table 2: Statements grouped by cluster: Clusters in order of mean rating
Table 2 (suite)
22If we concentrate on the clusters that are rated as most important, a clear picture emerges. Research on bias control is considered most important, followed by research on non-response reduction. The lowest ratings are given to research concerning response climate and international and cross-cultural issues.
23It should be noted that the low rating for international and cross-cultural issues is only relative to the other issues. We know that the response climate in the various countries from which representatives were present, differs to a great extent, as for instance is shown by De Leeuw and De Heer (2002). In addition, differences in response rates between countries can seriously affect the validity of international surveys (Couper & de Leeuw, 2003). But research into the causes of this phenomenon and international standardization is apparently considered less important, as compared to research into non-response adjustment and reduction.
24Non-response adjustment and reduction are considered almost equally important, with "bias control" being the most important. As to non-response reduction, optimization of the data collection process is considered most important, followed by monitoring this process. Research into new methods for data collection is rated a little less important.
25When we take the variance in ratings into consideration, we see that there is considerable variance in ratings for these four clusters, with largest variance for the cluster on "bias control". This is clearly illustrated in Figure 1. This means that the views on the importance of these issues differ within the group of experts.
Figure 1: Box plots of cluster scores
26(under preparation/en préparation)
27At the 13th international Workshop on Household Surveys Non-response in Copenhagen, 29-31 August 2002, a discussion was started on topics for future research. To structure the discussion concept mapping was used. This discussion resulted into the following:
Research into statistical bias control was rated as the most urgent.
Rated a little less urgent was research into non-response reduction. As to
this research, optimization and monitoring of the data collection process is considered more important than the development of new methods.
Research into international comparison of response climates and cross-
cultural differences is rated relatively less important.
28To conclude, the group of experts gathered at the 13th nonresponse workshop would prefer to subsidize research on non-response adjustment and measures to reduce non-response including monitoring the effects of these measures.
29This makes sense. Since non-response affects data quality, both non-response correction and non-response reduction is important. The first step is to develop measures that improve response rates. To accommodate the resulting non-response, methods to obtain valid estimates have to be developed. Thus, what is needed is research that demonstrates the linkage between design features and errors. What is needed are experiments that systematically investigate the effectiveness of design measures, in relation to costs. Only then we will be able to make good design decisions, controlling variation by design. Only then survey methodology will be a science in stead of an art.