Navigation – Plan du site
Ongoing Research/Recherche en cours

Interviewer/Data Quality

p. 65-72

Résumés

Plusieurs membres de RC33 de l’Association Internationale de Sociologie, le Comité de recherche “Logique et Méthodologie”, ont fourni des informations et contacté d’autres chercheurs pour contribuer à ce numéro thématique « intervieweur/qualité des données ». Des contributions de types différents sont venues de France, de Suisse, de Suède, d’Allemagne et des Pays-Bas, et sont inclues ci-dessous.

Haut de page

Texte intégral

Lilli Japec – Quality Issues in Interview Surveys – Some Contributions

1Lilli Japec at Statistics Sweden (lilli.japec@scb.se) recently defended her Ph.D. thesis, Quality Issues in Interview Surveys: Some Contributions, which includes an introductory section including the following extract (with references) and sections on Interviewers and Survey Quality, Characteristics of the Survey Interview, Two Schools of Interviewing, Interviewer Effects, Quality Control, Some Recurring Issues in Interview Surveys, and Summary of Papers (the first paragraph of which is also included at the end of the extract).

Introduction

2According to Eurostat, survey quality can be seen to consist of six dimensions: the relevance of the statistical concepts, the accuracy of estimates, the timeliness and the punctuality in disseminating results, the accessibility and the clarity of the information, the comparability, and the coherence. In interview surveys all these dimensions apply and interviewers affect mainly the accuracy component but to some extent also timeliness. The effect on the latter, however, is more of an indirect one since it is the responsibility of management to estimate the time needed to complete a survey in the prescribed manner and monitor the procedure to ensure that this estimate is correct.

3The accuracy of statistics can be viewed in terms of total survey error, which is the total effect of various specific sources of error associated with the survey process. The sources can be viewed as operations performed in a certain sequence, such as specification of a research problem, defining a target population, designing a sampling scheme, developing a measurement instrument, choosing a data collection mode, and choosing a set of data processing operations such as data capture, editing, coding, weighting, and disclosure avoidance. When the sources are viewed like this, in terms of survey operations and the execution of them, each operation adds to an estimate’s mean squared error (MSE). According to Biemer and Lyberg (2003) a simple decomposition of the MSE, for an estimate of, say, an average could be:

MSE= Bias2 + Variance =
(BSPECIFICATION + BFRAME + BNONRESPONSE + BMEASUREMENT + BDATAPROCESSING)2
+ VarSAMPLING + VarMEASUREMENT + VarDATAPROCESSING

4That is, each survey operation contributes to the bias or to the variance or sometimes both. Ideally we would like to design a survey and our processes so that we get an MSE that is as small as possible given a certain budget. To estimate the MSE components, however, special evaluation studies are needed and these are often associated with large costs. Therefore data quality indicators are often used as proxies for estimates of MSE components. For example, nonresponse rates are often used as proxies for nonresponse bias. As a consequence in many interview surveys a large proportion of the survey budget is allocated to reduce nonresponse rates through the use of elaborate strategies with many call attempts.

5There are many actors involved in the design and execution of survey operations that can contribute to the total survey error. Actors include managers, interviewers, respondents, and survey methodologists. The actor effect cannot be illustrated in the same straightforward way, as was the case with the sequence of survey operations. Still, viewing various actors’ influence on total survey error is very important to gain guidance on issues related to tradeoffs in survey planning. In interview surveys interviewers are very important actors. They are involved in the process of gaining respondent cooperation, being in charge of the question-answer process, “training” the respondent (Groves et al 2004; Fowler 1991), and sometimes even in the frame construction through being in charge of listing units that should be included in the frame. Some sampling designs demand that the interviewer to select a respondent from a sampling unit comprising more than one potential respondents. Such schemes include the Kish method and “birthday” methods (Oldendick et al. 1988). Thus, there is an interviewer influence on the total survey error that is best handled by installing work processes that come close to the ideal.

6When trying to improve survey quality different perspectives must be kept in mind. First, an assessment of the relative importance of error sources must be made. Second, such an assessment calls for studies of error rates and of error structures, and knowledge about quality indicators and actual quality. Third, developing processes that are stable, predictable and successively approach the ideal seems to be a reasonable complement to measuring error sizes reflecting MSE decomposition.

7General descriptions of survey quality along these varying lines are to be found in Hansen et al. (1964); Groves (1989); Biemer and Trewin (1997); Biemer and Stokes (1991); Biemer and Lyberg (2003); Groves et al. (2004); and Morganstein and Marker (1997).

8This thesis deals with some of the issues associated with the quality in interview surveys. The interviewer has many tasks, and some of them are extremely error-prone. I describe some general themes and how these are usually handled. I will also suggest some new approaches and ideas for further work, both when it comes to gaining cooperation and when it comes to doing a good job in administering the question-answer process. We also evaluate a specific procedure to reduce nonresponse errors in interview surveys in terms of costs and error reduction.

9The subsequent sections and sub-sections of Japec’s thesis/book are:
Interviewers and Survey Quality
Characteristics of the survey interview
Two schools of interviewing
Interviewer effects
Quality control
Some recurring issues in interview surveys
Summary of Papers
Paper I: Research Findings in Interview Surveys – Some Implications and Research Needs
Paper II: The Interview Process and the Concept of Interviewer Burden
Paper III: Interviewer Burden and Its Effects on Data Quality in the Swedish Part of the European Social Survey (ESS)
Paper IV: Effects of Field Efforts on Nonresponse Bias and Costs in the Swedish Labor Force Survey

References (extracts – only those mentioned above)

10Biemer, P.P. and Lyberg, L. (2003). Introduction to Survey Quality. New York: Wiley.

11Biemer, P. P. and Stokes, L. (1991). “Approaches to the Modeling of Measurement Errors”. In Measurement Errors in Surveys, P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, and S. Sudman. New York: Wiley and Sons. 487-517.

12Biemer, P. and Trewin, D. (1997). “A Review of Measurement Error Effects on the Analysis of Survey Data”. In L. Lyberg, P. Biemer, M. Collins, E. De Leeuw, C. Dippo, N. Schwarz, and D. Trewin (eds), Survey Measurement and Process Quality, New York: Wiley and Sons, 603-632.

13Fowler, F.J. Jr (1991). “Reducing Interviewer-Related Error Through Interviewer Training, Supervision, and Other Means”. In P. Biemer, R. Groves, L. Lyberg, N. Mathiowetz, and S. Sudman (eds), Measurement Errors in Surveys, New York: Wiley, 259-278.

14Groves, R.M. (1989). Survey Errors and Survey Costs. New York: Wiley.

15Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., and Tourangeau, R. (2004). Survey Methodology. New York: Wiley.

16Hansen, M., Hurwitz, W.N., and Pritzker, L. (1964). “The Estimation and Interpretation of Gross Differences and the Simple Response Variance”. In Contributions to Statistics, presented to professor P.C. Mahalanobis on the occasion of his 70th birthday, 111-136.

17Morganstein, D. and Marker, D. (1997). “Continuous Quality Improvement in Statistical Agencies”. In L.E. Lyberg, P. Biemer, M. Collins, E. de Leeuw, C. Dippo, N. Schwarz, and D. Trewin (eds), Survey Measurement and Process Quality, New York: Wiley, 475-500.

18Oldendick, R.W., Bishop, G. F., Sorenson, S.B., and Tuchfarber, A.J. (1988). “A Comparison of the Kish and Last Birthday Methods of Respondent Selection in Telephone Surveys”. Journal of Official Statistics, 4, 4, 307-318.

Geraldine Vivier & INED - Projet de recherche exploratoire sur le métier d’enquêteur

19At the official French Institut National des Etudes Démographiques (INED), Géraldine Vivier (vivier@ined.fr), is responsible for an exploratory research project on the profession of interviewer. In the extracts below (in French), she presents is newly developed project.

20Concernant la présentation de notre recherche sur les enquêteurs dans le prochain BMS, nous en sommes à un stade encore très préliminaire et exploratoire ; plus qu'une recherche en cours, il s'agit d'un projet en construction : beaucoup de points importants restent encore à définir,différentes options sont possibles... Voici donc les grandes lignes et objectifs de cette première étape de défrichage.

21Ce projet exploratoire porte sur le métier d'enquêteur et vise à saisir les parcours et les pratiques professionnels des enquêteurs. Rejoignant une réflexion méthodologique plus large sur la situation d'entretien, cette étude a pour objectif de mieux comprendre les conditions de production des données, qualitatives ou quantitatives, en intégrant le point de vue des enquêteurs, rarement interrogé. En effet, si beaucoup de recherches ont questionné la situation, complexe, de l'entretien, souligné les biais induits par la relation enquêteur-enquêté ou discuté de la « bonne façon » de conduire un entretien, peu de travaux ont abordé ces difficultés méthodologiques et plus largement, le travail de terrain, en y associant le point de vue, l'expérience et les savoir-faire des enquêteurs. A travers des entretiens semi-directifs exploratoires, il s'agit donc d'interroger le métier d'enquêteur :
en replaçant cette expérience professionnelle dans l'ensemble de la trajectoire de l'individu : dans quelles circonstances, familiales et/ou professionnelles, est-on devenu enquêteur ? Quel est le statut de cette activité dans l'ensemble du parcours ? Quels autres emplois ont éventuellement été exercés...? Etc.
en cernant les pratiques professionnelles et de leurs possibles évolutions : types d'enquêtes et d'univers professionnels ? Évolutions techniques, évolutions personnelles? Contextes des entretiens et dimensions non verbales des interactions, etc.

22Il s'agit ainsi d'éclairer les pratiques de terrain, les conditions d'enquêtes et d'approfondir ce qui se joue et ce qui est investi dans la rencontre enquêteur-enquêté à l'aune du parcours et de l'expérience personnelle de l’enquêteur.

23Bessière Céline et Houseaux Frédérique, 1997 – « Suivre les enquêteurs »;, in Genèses, 29, pp. 100-114.

24Dussert Françoise, 1996 – « Le réseau d’enquêteurs de l'Insee: 50 ans d’histoire », in Courrier des statistiques, n. 78, pp. 47-52.

25Insee, 1981 - Eléments pour un dossier sur les enquêteurs de l’Insee, Tomes 2, fasc. 1 et 2, DR de Paris.

26Memmi Dominique, 1999 – « L’enquêteur enquêté. De la « connaissance par corps » dans l’entretien sociologique », Genèses (35).

Interviewer Effects in the Swiss Household Panel Survey

27Annette Scherpenzeel (Adstat, Bureau for Advanced statistics and Analysis, a.scherpenzeel@planet.nl, corresponding author) and Olivier Lipps (Swiss Household Panel, Neuchâtel, oliver.lipps@unine.ch) are currently working on a report entitled “Interviewer Effects in the Swiss Household Panel Survey” from which we publish below the detailed abstract.

28In order to carry out an evaluation of interviewer effects, the Swiss Household Panel (SHP) has sent a paper-and-pencil questionnaire to the interviewers after the yearly panel interviews in up to now three years. The characteristics of the interviewers as measured by this interviewer questionnaire were then used as predictors for the interviewer variance in response rates, item nonresponse and data quality. The interviewer variance in these dependent variables was considerable, sometimes as much as 12% of the total variance in a variable. However, in the first two interviewer studies, carried out after the 2000 and the 2003 panel waves, a large part of this interviewer variance remained of unknown origin. The interviewer characteristics that were included in these studies explained only part of it and a large part remained unexplained. Therefore, the interviewer questionnaire was adapted and sent again to the interviewers after the 2004 panel wave. Based on a review of recent literature on interviewer effects, the new questionnaire included questions about:
Interviewer's attitude towards ease of questionnaire
Interviewer's attitude towards ease of parts of questionnaire
Interviewer's attitude towards ease of persuading respondents
Overall life satisfaction, positive and negative affects.
Interviewer's attitudes and respondent-interviewer attitudinal distance.

29The goal of the third interviewer study is to determine how much of the interviewer variance can be attributed to these newly included interviewer characteristics, measuring rather “soft” than “hard” facts. A special feature of this study is that a number of interviewers have participated in all three interviewer studies and also in all three data collection waves that preceded the interviewer studies. They are what we will call the “longitudinal interviewers”, for whom we have three-year data on (changes in) their own situation, experience and attitudes as well as on (changes in) the response behaviour and attitudes of the people they interview. Unlike most CAPI panel surveys, the respondents in the SHP, which uses CATI technique, are basically randomly assigned to interviewers, allowing to study interviewer effects with a random design. This is a rather unique situation and the secondary goal of the third interviewer study is hence to construct a three-wave longitudinal model, relating changes in the situation, experience and attitudes of the longitudinal interviewers to (changes in) the response behaviour and attitudes of the people they interview.

30To summarise, the present paper aims to deal with the following main research questions:

311.) We will build a cross-sectional multilevel model, in order to estimate the additional interviewer-respondent interactive effects by a stepwise inclusion of the new variables asked in the 2004 interviewer questionnaire. Contrary to most up-to-date interviewer questionnaires, we try to explain a hitherto large portion of unexplained interviewer variance from interviewer survey data, which mostly contain “hard” facts.

322.) We will build longitudinal multilevel growth models, in which we analyse interviewer specific change of attitudes, controlled for the available factual variables. These models will contain different dependent variables, which measure data quality in a broader sense.

333.) Structural equation modelling techniques will be used to study the longitudinal interviewers in more detail: A panel model on the interviewer level will show whether training effects exist in the performance of these interviewers, whether their attitude towards this type of surveys and the questions in it change across the panel waves, and whether these changes affect the response rates obtained and the data quality.

344.) An existing structural equation panel model of well-being will be tested across different interviewer profiles. Tests of the equivalence of the model structure and the strength of the effects across interviewers will show whether causal relations at the respondent level can be modified by interviewer characteristics.

Other Resources and References

35The preceding issue of the BMS included in the « Ongoing Research » section a report-call for contributions concerning this theme of Interviewer/Data Quality (Karl M. van Meter, “Studying Survey Interviewers: A Call for Research and an Overview”, BMS, n. 88, October 2005, pp. 61-71). That article included all the previously published BMS articles related to this theme and is a good starting point for further research.

36Tom Smith of the National Opinion Research Center at the University of Chicago also brought to our attention a book by Jean Converse, Conversations at Random, published in the 1970s, which deals with conversations with survey interviewers and contains an extensive bibliography.

37In Bonn, Germany, the Informationszentrum Sozialwissenschaften, under Jürgen Krause, kindly searched their Foris and Solis data bases for research on interviewers, surveys and data quality. Most of the entries had to do with sensitive questions (income, drugs, personal problems), but we found the following entries of particular interest:

38Jörg-Peter Schräpler, Gert G. Wagner, “Identification, characteristics and impact of faked interviews in surveys: an analysis by means of genuine fakes in the raw data of SOEP”, DIW Diskussionspapiere 392, Berlin 2003, 34 pp.

39Jost Reinecke, “Regelmässigkeiten des Interviewverhaltens“, 1998, in Statistisches Bundesamt, Interviewereinsatz und –qualifikation, Wiesbaden.

40Rainer Schnell, “Besuchs- und Berichtsverhalten der Interviewer“, ibid.

41Joop Hox, Edith de Leeuw and Ger Snijkers, “Fighting nonresponse in Telephone interviews: Successful interviewers tactics“, in Achim Koch and Rolf Porst (editors), Proceedings of the Eight International Workshop on Household Survey Nonresponse, 24-16, September 1997, ZUMA, Mannheim, pp. 173-185.

42Patrick Sturges and Pamela Campanelli, “The effect of interviewer persuasion strategies on refusal rates in household surveys“, ibid., pp. 187-200.

43Edith de Leeuw, Joop Hox, Ger Snijkers and Wim de Heer, “Interviewer opinions, attitudes and strategies regarding survey participation and their effect on response“, ibid., pp. 239-248.

44Geert Loosveldt, Ann Carton and Jan Pickery, “The effect of interviewer and respondent characteristics on refusals in a panel survey“, ibid., pp. 249-262.

45This work remains very much « ongoing research »; the BMS would be interested in receiving further information, contacts and references concerning the theme interviewer/data quality.

Haut de page

Pour citer cet article

Référence papier

« Interviewer/Data Quality », Bulletin de méthodologie sociologique, 89 | 2006, 65-72.

Référence électronique

« Interviewer/Data Quality », Bulletin de méthodologie sociologique [En ligne], 89 | 2006, mis en ligne le 01 juillet 2009, consulté le 25 mars 2017. URL : http://bms.revues.org/742

Haut de page

Droits d’auteur

© BMS

Haut de page
  • Revues.org