Navigation – Plan du site
Ongoing Research/Recherche en cours

Report on the Third International Workshop on Comparative Survey Design and Implementation

Janet Harkness
p. 83-89

Résumés

Compte-rendu du Troisième Atelier International sur la Construction et l'Implémentation d'Enquêtes Comparatives : les principaux objectifs du programme "Comparative Survey Design and Implementation" (CSDI) sont de fournir un forum et une assise pour des chercheurs concernés par les aspects méthodologiques des enquêtes trans-nationales et multi-culturelles ; d'encourages la coopération sur des projets d'amélioration de la recherche comparative ; et de disséminer le savoir et promouvoir les meilleures applications. Le Troisième Atelier International de CSDI a eu lieu du 10 au 12 mars à Madrid, avec les sessions plénières où des participants ont fait des présentations et avec des sessions parallèles où des groupes de travail se sont réunis pour discuter des travaux en cours et un ensemble commun de questions présentées ci-dessous.

Haut de page

Texte intégral

1The Third International Workshop on Comparative Survey Design and Implementation (CSDI) was held in Madrid from the 10th to 12th March, 2005.

2The main goals of the CSDI are to provide a forum and platform for researchers concerned with methodological issues in cross-national and cross-cultural surveys; to foster co-operation on projects aimed at improving comparative research; and to disseminate knowledge and promote best and good practices.

3CSDI meetings are annual. The first two workshops took place in Brussels (2003) and Paris (2004). CSDI 2006 will meet in Tilburg, the Netherlands, and in 2007 CSDI will meet in Chicago, USA.

4Attendance at CSDI meetings is to date free of charge. At each meeting, a local host has generously organized a meeting place and provided lunches for the group. Details of past and upcoming meetings, participants, programmes and some presentations and papers are available on the CSDI web site (www.csdi-workshop.org).

5At the Madrid meeting, plenary sessions, at which participants presented papers, were combined with break-out sessions in which work groups met to discuss work in progress, as well as a common set of questions described below. Proceedings from the Madrid workshop will appear in autumn 2005. Their publication will be announced on the CSDI web site.

Work Groups in the CSDI

6Directly after the first meeting in 2003, core members of the CSDI organized work groups on areas of key importance for comparative survey research. Each group began to set out short and mid-term goals. Details of the goals and activities of the groups are available at the CSDI web site, and the contact people are identified there for each work group. At the 2004 meeting in Paris, the following work groups reported on their progress to the plenary: Questionnaire Design, Data Collection, Survey Process and Documentation Tools; Measurement and Testing and three groups subsumed under Multilingual Issues:Interpreting, Translation Guidelines, andBasic Research onTranslation.

7A Research Agenda work group was also formed in 2003. In collaboration with the other work groups, this group is elaborating a general research framework for comparative survey research. A draft paper on this topic from the 2005 meeting can be downloaded from the CSDI web site. At CSDI 2005, a new work group was formed on harmonization issues in survey research.

International Conference in 2008

8By the 2004 meeting, CSDI had decided to target an international conference in 2008 on methodological issues in cross-national and cross-cultural survey research. The conference is intended as a landmark event, presenting state-of-the-art methodological research across cultures, nations, regions and markets. It will bring together research and researchers from official statistics, educational research, health research and psychological testing, market research, establishment surveys and the political and social sciences, including cross-sectional and longitudinal designs. An edited volume will accompany the conference. Members of the CSDI steering committee are currently working on administrative and substantive aspects of this conference. A number of activities for the 2006 and 2007 CSDI workshop meetings will be linked to goals for the 2008 conference. For example, the break-out sessions introduced at the 2005 workshop to allow participants with shared interests more time together will be extended in 2006. The number of attendees can thus also grow to better include such important sectors as business surveys, official statistics, and educational research. The plenary discussions for 2006 and 2007 will focus on subject areas targeted as section topics for the 2008 conference.

CSDI 2005 Presentations at Plenary Sessions and in Work Group Sessions

9The majority of the plenary presentations at the 2005 meeting were given on the first day to provide blocks of time for break-out sessions on the second. New attendees were given plenary slots wherever possible. The papers covered a wide range of topics. Presentations and reports from colleagues already active in CSDI were thus sometimes presented in work groups rather than during the plenary session. Abstracts, overheads, and papers (as available) will be posted on the CSDI web site.

10The papers, in order of presentation, were :

  • (Plenary) What’s so special about cross-national surveys?, Peter Lynn (University of Essex), Lilli Japec, Lars Lyberg (Statistics Sweden)

  • ISO standards for market, opinion, and social research: A review, Tom W. Smith (NORC, Chicago)

  • An examination of pretesting methods for multicultural, multilingual surveys, Patricia Goerman (US Bureau of the Census)

  • Issues ineliciting education level from Latin American immigrants in the US, Alisù Schoua-Glusberg (Research Support Services, Evanston IL, USA)

  • Non response and response quality: A Swiss perspective, Dominique Joye (SIDOS, Neuchatel, Switzerland)

  • A method for valid statistical tests after response conversion, Stef van Buuren, Gert Jacobusse (TNO Prevention and Health, Leiden, Netherlands)

  • Use of measurement models across many countries using EVS/WVS data, Jacques Hagenaars, Ruud Luijkx ( University of Tilburg, Netherlands)

  • Measuring political participation of young people in Europe,Vlasta Zucha (Institute for Social Research and Analysis, Vienna)

  • Cultural values and survey response styles in the European Social Survey, Timothy Johnson (University of Illinois at Chicago), Peter Mohler, Janet Harkness (ZUMA, Mannheim), Fons van de Vijver (University of Tilburg, Netherlands)

  • Comparisons of poverty through household surveys, Patrick Festy (INED, Paris)

  • Harmonizing education codes in existing data files, Harry Ganzeboom, (Free University, Amsterdam)

  • Measuring income in comparative social survey research, Jürgen Hoffmeyer-Zlotnik (ZUMA, Mannheim), Uwe Warner (CEPS/INSTEAD, Luxembourg)

  • The Data Documentation Initiative (DDI) and comparative survey data, Peter Granda (ICPSR, University of Michigan), Meinhard Moschner (Zentralarchiv, Cologne)

  • Survey management and documentation system update, Beth-Ellen Pennell, Karl Dinkelmann, Sue Ellen Hansen, Peter Granda, Grant Benson, James Wagner, Gina-Qian Cheung, Deborah Mike, Peter Sparks, Ashley Bowers (Survey Research Center, ISR, University of Michigan), Peter Ph. Mohler, Janet Harkness, Evi Scholz (ZUMA, Mannheim)

  • Web site documentation: Current practice; best practice,

  • Deborah Mike, Sue Ellen Hansen, Beth-Ellen Pennell (Survey Research Center, ISR, University of Michigan)

  • ASEP-JDS Data Archive, Juan Diez Nicolás (ASEP, Madrid)

  • Category and comparison across what kind of frontier?, John MacInnes (University of Edinburgh, Universitat Autònoma de Barcelona)

  • Anchoring Vignettes: A tool to assess the cultural comparability of survey items, Patricia Gallagher (Center for Survey Research, University of Massachusetts, Boston)

  • (Work Groups), Fieldwork details in the European Social Survey 2002/2003, Achim Koch, Michael Blohm (ZUMA, Mannheim)

  • Interpreting in surveys, Brad Edwards (Westat, USA)

  • The use of interpreters in the conduct of households surveys, Yuling Pan (US Bureau of the Census)

  • Communicative norms and survey translation, Brian Kleiner (Westat, USA), Yuling Pan (US Bureau of the Census)

  • Oral translation in telephone surveys, Janet Harkness (ZUMA, Mannheim), Dominique Joye(SIDOS, Neuchatel, Switzerland)l

  • Analysing value items across ESS countries, Peter Mohler, Kathrin Wohn (ZUMA, Mannheim)

CSDI 2005 Work Group Session Reports

11In the two break-out sessions scheduled during the meeting, work groups had the opportunity to discuss projects already under way, present papers prepared for the smaller groups, and consider their response to three questions posed by the Research Agenda work group. It had asked CSDI participants to discuss the following three questions and report back on these to the plenary session:

  • What are the aspects of design and implementation that (potentially) affect equivalence/comparability of survey data?

  • How would your rank these aspects is terms of priority?

  • Why would you rank them this way?

12Full reports from the work groups will be posted on the CSDI web site. Only general points from work group sessions are presented here.

Process and Documentation Tools Work Group

13The goal of the Process and Documentation Tools group is to provide minimum guidelines for process and for study documentation in cross-national survey research projects. Such documentation must be sufficiently detailed to allow replicability. Preparing this level of documentation can be a labor-intensive endeavor. Therefore, a secondary goal of the group is to provide templates and programs that facilitate the documentation process throughout the survey design, implementation and data processing steps.

14During the 2004 meeting, the group presented work on tool prototypes that help co-ordinate questionnaire programming, that monitor and document key aspects of study design and implementation, and that ultimately will produce full-study documentations for process monitoring and study reporting. At the 2005 meeting, beta-version of the tools was presented, and a new module on paradata was introduced. Future development of these tools will be closely linked to the Data Documentation Initiative (DDI) and documentation strategies being pursued by major survey data archives.

Multilingual Work Groups

15Three work groups joined forces in Madrid: the work group on Interpreting, the work group on Translation Guidelines and the work group on Basic Research on survey translation issues. The groups reported back to the plenum with a 14-point agenda for the coming year(s) which can be downloaded from the CSDI web site. The Translation Guidelines group is currently working on an overview of existing guidelines and protocols. This work will feed into what will ultimately be the CSDI guidelines on production of questionnaires, pre-testing, translation, and translation assessment. The Interpreting group plans to conduct experiments to collect evidence on the effects of using interpreters versus using translations or bilingual interviewers. The Basic Research group is currently working on answer scale research and on oral translations. An international CSDI survey is planned to investigate how survey agencies handle multilingual issues (for example, by making written or oral translations, or interpreting). The questionnaire is currently being finalised. For further information on this, contact Janet Harkness: harkness@zuma-mannheim.de

Questionnaire Design and Pretesting Work Group

16Research on questionnaire design in an assumed mono-cultural context is extensive and many faceted. Research on questionnaire design for cross-cultural purposes is considerably more limited. While the literature describes several general models of procedure, detailed examinations of implementations following specific models or evaluating one approach against others are rare. A first priority identified was to provide clear descriptions of existing procedural models, outlining their potential and requirements and providing examples at the study and the item level. In terms of basic research, the work groups proposed first to identify what the distinctive components of cross-national and cross-lingual questionnaire design may be and what additional or adapted strategies may be needed for appropriate pretesting.

17The overlap and interdependence of translation and adaptation issues, on the one hand, and questionnaire design, on the other, were a recurring feature of discussion in the group. In some design models, translation is a part of questionnaire design. It remains to be seen where lines will be drawn between design and language-version production.

Data Collection Work Group

18The group raised a number of wide-ranging issues that will be posted on the CSDI web site. A general problem to be faced is that little concrete information is available in concise form about how data is collected across countries. Data collection modes and their potential in different contexts are only one set of issues, if admittedly an important one. While there is general agreement, for example, that key variables need to be monitored during the data collection phase, we are not yet in a position to say what the key variables are in cross-national projects. As far as more concrete proposals for the research agenda were concerned, two ideas were raised. First, the issue of mixed-/multi-mode designs was seen as especially relevant for cross-national surveys. Secondly, the group found it important to learn more about the effects of measures and procedures to enhance response rates in a cross-national perspective. As far as setting priorities is concerned, the group suggested a start be made with information that is easier to collect. Information on data collection mode, interviewer training efforts, or incentives used are relatively easy to collect, since they pertain to an entire study in a country. In contrast, it is much harder to collect information at micro level, on, for example, all the calls made at all sampled addresses. In thinking about recommended practice, the group felt that different standards of providing information could be applied, such as minimum standards, good practice and best practice. The issue of standardisation (what issues should be kept the same, and what needs country-specific adaptation) was found particularly relevant for data collection in cross-national surveys.

Measurement and Testing Work Group

19The work group formulated a number of basic questions in response to the Research Agenda group requests. These will be posted on the CSDI web site as part of the ongoing debate. They discussed, for example, the degree to which methodological findings were diffusing into the substantive literature and whether the applied literature actually reflects the results of methodological findings. With new players entering the burgeoning field, the need to disseminate methodological knowledge is pressing.

20In terms of practical planning, the group decided to start a project on answer scales that will complement ongoing work in the multilingual groups. For the longer term, the stated aim of the group is to develop and promote guidelines for testing and assessment issues in cross-cultural surveys. The guidelines will aim to be straightforward, user-friendly and practically-oriented, of use to both researchers carrying out cross-cultural research and to reviewers when evaluating cross-cultural research projects. 

21Colleagues who would like to learn more about CSDI, to join in CSDI activities or simply to inform CSDI about their activities and upcoming events are invited to visit the web site mentioned earlier: http://www.csdi-workshop.org

Haut de page

Pour citer cet article

Référence papier

Janet Harkness, « Report on the Third International Workshop on Comparative Survey Design and Implementation », Bulletin de méthodologie sociologique, 86 | 2005, 83-89.

Référence électronique

Janet Harkness, « Report on the Third International Workshop on Comparative Survey Design and Implementation », Bulletin de méthodologie sociologique [En ligne], 86 | 2005, mis en ligne le 28 mars 2008, consulté le 25 mars 2017. URL : http://bms.revues.org/968

Haut de page

Auteur

Janet Harkness

ZUMA, Mannheim, harkness@zuma-mannheim.de

Haut de page

Droits d’auteur

© BMS

Haut de page
  • Revues.org