- 1 The term subjective concepts that I use here is about synonymous with the term subjective phenomena (...)
1When looking for explanations of social behavior, one soon meets prejudices, opinions, preferences, and attitudes. Characteristic of these subjective concepts1 is that they are not directly observable. About the only way to find out what is someone’s opinion or attitude, is to ask for it. The same holds for past events and past behavior, not recorded in documents. And when one is asking these questions, one might, ‘in one go’, ask questions about issues that are observable or recorded in documents. For that reason, asking people questions is by far the most often used method of data collection- in the social sciences (Presser, 1984; Saris & Van der Zouwen, 1999).
- 2 A more elaborate argumentation for the standardization of questionnaires is presented elsewhere (Va (...)
2Because subjective phenomena, like opinions and attitudes, are only indirectly observable, it is of great importance that the respondents all answer exactly the same questions. And if they are questioned at more points in time, that they then answer questions that are similarly worded. Otherwise the researcher will never know whether a difference in the answers between two respondents is caused by a difference in opinion, or by a difference in question wording. Or that a change in the response of a particular respondent results from a changed opinion, or from an altered question wording. The researcher who will measure variables concerning subjective concepts has to solve one equation with two unknowns. This is only possible by holding one of the unknowns constant. So, researchers measuring subjective variables are forced to use standardized questionnaires.2
3In the past questionnaires were usually administered by interviewers who read out the questions and recorded the answers of the respondents (Platt, 2002). However, this mode of administration has become very expensive, especially so if the interview is not administered by telephone but during a visit to the respondent’s home, that is, in a personal interview. For most survey researchers the costs of personal interviews have become prohibitive. In a recent German study it appeared that the costs of a standardized personal interview have been tripled over the last twenty years, up till over 110 euro per interview (Koch, 2002).
4Yet there are situations in which it is desirable, or even necessary, to administer the questionnaire in a personal interview. For example, if the respondents have insufficient command of the language in which the questionnaire is worded, are cognitively handicapped, or not sufficiently motivated to cooperate. And also if the questionnaire is long and contains difficult questions, or consists of complex sequences of follow-up questions, aimed at the reconstruction of someone’s life history or social network.
5Personal interviews are not only expensive, they also are at odds with the ideal of standardized observation. The mere presence of a second person asking questions and listening to the answers may affect the choice of the response categories. Extreme, and socially less desirable opinions are in personal interviews less often expressed than in self-administered questionnaires (De Leeuw, 1992; Van der Zouwen & De Leeuw, 1990, 1991). And this kind of social desirability bias is not equally strong for all respondents (Gove, 1982; Stocké, 2004). So it is hardly possible to correct for this bias.
6Moreover, if the researcher calls in interviewers, it have to be dozens. Even the interviewing of a small sample of 400 persons requires at least ten interviewers. These interviewers differ from each other in their appearance, and in their treatment of the respondents. And these differences between interviewers form an important source of error (Groves, 1989). Finally, part of the tasks of interviewers is dependent on what the respondent does: the interviewer only has to give an additional explanation to the question if the respondent asks for that. And the interviewer has to probe further only if the answers of the respondent make this necessary. The execution of these respondent-dependenttasks threatens the standardization of the process of observation.
7Survey researchers are not inclined to scrutinize the influence of the interviewer on the answers collected. They rather assume, for the sake of convenience, that the interviewer is only a kind of neutral extension of the questionnaire. Using this assumption, the researcher can simply put together filled out questionnaires, stemming from different interviewers, into one database. And information about the interviewers does not play a role in the further analysis. That makes work easy.
8However, it is found time and again that even if interviewers are randomly assigned to respondents, some kind of systematic interviewer effect appears, threatening the comparability of survey data (Van Meter, 2005).
- 3 Examples of ‘interviewer cheating’ can be found in Schreiner, et al. (1988), Biemer & Stokes (1989) (...)
- 4 A remarkable example of interviewer fraud is recently reported by Matschinger et. al (forthcoming). (...)
9Sometimes the negative influence of the interviewer on data quality is much stronger, namely in the case of interviewer fraud. Interviewer fraud usually means that some interviewers skip parts of the questionnaire or fill them out by themselves.3 By doing so these interviewers can shorten the interviewing time considerably, and even put the financial incentives, meant for the respondents, in their own pockets. The answers invented by these interviewers, will of course very negatively affect data quality.4
10The behavior of the interviewers seems to be more problematic than survey researchers assume or hope for. But we deal with conjectures only. The precise causes of interviewer effects are not as yet clear and the suspicion of interviewer fraud is often difficult to prove.
11To obtain a detailed description and explanation of interviewer effects, one has to investigate the actual behavior of interviewers during interviews:
In how far do interviewers behave according to the instructions given to them?
In how far is the interviewer attentive to problems the respondent might have with understanding the question and finding an adequate answer?
In how far is the interviewer able to probe in such a way that the information obtained becomes more relevant and adequate, without biasing the answers?
And finally, how precise is the interviewer when recording and coding the answers given by the respondent?
- 5 An elegant procedure for the processing and analysis of interview protocols (SEQUENCE) is developed (...)
12The only usable method to answer these research questions is to open up the ‘black box’ of the interview and to record what interviewer and respondent actually do and say during the interview. This is normally done by audio taping the interviews, and transcribing these recordings into verbatim protocols. The resulting protocols are then analyzed, using a dedicated coding scheme.5
13A basic requirement of a standardized interview is that the questions from the questionnaires are posed exactly as they are worded. This sounds so trivial that one would expect that this indeed happens. However, it often occurs that when one listens to the audiotape of an interview, one hears the interviewer reading out text that differs from the one in the questionnaire. Mostly this only concerns minor deviations that leave the content of the question intact. But it also happens that the interviewer reformulates the question such that also the meaning of the question alters (Brenner, 1982; Van der Zouwen & Smit, 2004a; Van der Zouwen et al. 2005).
- 6 For the sake of readability, the interviewer is indicated in this article with ‘she’ and ‘her’ and (...)
14If the respondent answers this altered question by using one of the presented response categories, irrelevant information is obtained. The survey researcher does not know that the respondent gave an answer to another question than the intended one, so he6 may come to false conclusions. Here the interviewer certainly is no help to the survey researcher, but a hindrance.
15How does it come that interviewers sometimes alter the text of the questions?
16Firstly, one may think of inattentiveness, insufficient motivation, or even incompetence of the interviewer. But part of the ‘guilt’ can also be sought with the survey researcher who constructed the questionnaire. Some questions that look clear when printed on paper, are difficult to understand when they are read to the respondent. For example, if the explanation concerning a term used in the question is only given when the question itself is already asked, and some respondents start to answer that question without having heard the explanation (Houtkoop-Steenstra, 2000). Or if a question “whether one is satisfied with X” has to be answered by making a choice from a set of response categories, each representing a certain degree of (dis)satisfaction (Van der Zouwen & Smit, 2004a).
17Some interviewers are ‘improving’ the text of the questionnaire during their fieldwork. For example, by including the explanation of a term into the text of the question itself. Or by altering the satisfaction question from: ”Are you satisfied with X?” into “How satisfied are you with X?”. But other interviewers do not alter the question wording, so different respondents are actually asked different questions, therewith destroying the comparability of their answers.
- 7 In Houtkoop-Steenstra (1995) this process of 'meeting both ends' by the interviewer, is described.
18It often occurs that respondents have told so much about themselves that the question to be asked is in fact already answered; or is not applicable to this respondent. In these cases the interviewer has to improvise and adjust the question such that it shows that the interviewer has indeed listened to what the respondent said earlier.7 If the interviewer succeeds with that she clearly has been a help to the researcher.
19Ideally, the question by the interviewer is followed by the respondent’s choice of one of the response categories. In practice, such an ideal question-answer sequence does not occur as often as survey researchers are hoping:
For example, because the respondent does not give an answer, but asks for an explanation to parts of the question; or
Because not the respondent, but a third person, starts answering the question; or
Because the respondent refuses to answer the question, or says that he does not know the answer; or
Because the respondent answers a question that is partly different from the question asked by the interviewer.
- 8 The term ‘repair’ stems from Conversation Theory. See Houtkoop-Steenstra (2000, pp 29-33), Moore & (...)
20By these reactions of the respondent, the question-answer process leaves the track, resulting in an answer that is incomplete, ambiguous or irrelevant. It is the task of the interviewer to put the sequence back on the track, that is, to intervene and make an attempt to repair the sequence.8 Below, some examples are given of these attempts at repair.
21Survey researchers can ask questions that are hard to answer. The respondent might react with “I don’t know”, but such a reaction is not beneficial to his self-esteem.
22It is more attractive to answer a related question that he indeed can answer (Belson, 1981). One respondent, asked whether he expected that his income would increase in the coming two years answered with “Yes, I hope so”. An attentive interviewer will not immediately enter the response category ‘Yes’, but will tell the respondent that the question is not about what one hopes for but about what one expects that will occur.
23This sounds more trivial than it is. Because during the training, interviewers are told to build up good ‘rapport’ with the respondent. In this example the interviewer actually tells the respondent that he only has answered the easy question about his hopes, but that he still has to answer the more difficult question about his expectations about the development of his income in the coming two years. This repair action requires a lot of tact of the interviewer.
24With closed questions the respondent only has to mention the response alternative that best fits with his situation, or best represents his opinion. That task seems quite easy. Nevertheless, it often occurs that the respondent gives an answer that is clearly related to the question, but is not equal to one of the response alternatives presented (Ongena, 2005). Then the interviewer has to change, in cooperation with the respondent, the initial ‘free’ response (or ‘mismatch answer’) into one of the response alternatives mentioned in the questionnaire.
- 9 This inadequate interviewer behavior is indicated with the term ‘choosing’ (Dijkstra & Van der Zou (...)
25This repair tasks again requires tact of the interviewer, for the interviewer has to point out to the respondent that his initial answer is indeed relevant, but not adequate – at least not from the perspective of the survey researcher; and that, consequently, the respondent has to redo his answer. This request does not improve the ‘rapport’ between interviewer and respondent. For that reason some interviewers themselves ‘translate’ the ‘free answer’ of the respondent into one of the response categories from the questionnaire.9 The interviewer thus corrects the respondent without him being aware of that. This saves time and avoids irritation. But this interviewer behavior conflicts with the basic principle of the interview, namely that it is the respondent, and not the interviewer, who has to provide the answers
26Authors of handbooks about the standardized interview all agree that the just mentioned ‘choosing’ of the response category on behalf of the respondent, is not correct interviewer behavior. But they have different opinions about how an interviewer has to react to a mismatch answer of the respondent:
27Should the interviewer repeat all response categories and ask the respondent to make a choice from this set (Brenner, 1982; Fowler & Mangione, 1990)?
28Or is it enough that the interviewer mentions only those few categories that are clearly related to the initial answer of the respondent (Viterna & Maynard, 2002)?
29Or is it sufficient that the interviewer chooses from the set of response categories the one that best reflects the initial answer and then asks the respondent for a verification of this selection; i.e., the ‘verification’ probe: “Do you mean X?” (Moore, 2000).
30For each of these three probing techniques good arguments can be mentioned. The repetition of all response categories makes the probability of bias by the interviewer smallest, but may sound unnatural and even rude. After all, the respondent has indeed already answered the question. Only asking for a verification of the response category that, according to the interviewer, best fits with the initial ‘free answer’ sounds much more natural and does hardly threaten the ‘rapport’ between interviewer and respondent. But the probability of interviewer bias is much larger, for the question “Do you mean X?” is a slightly suggestive question. Whether this suggestion has a negative impact on the quality of the data strongly depends on the skill of the interviewer to select the best fitting category.
- 10 These four questions are chosen because this kind of income related questions is often asked in sur (...)
31In order to see whether attempts at repair of deviating question-answer sequences are successful, Van der Zouwen and Smit, (2004a en 2004b) made an analysis of parts of 201 interviews about income related issues with elderly respondents. Four closed questions concerned, respectively: (1) the amount of the yearly income, (2) whether a decrease of income had taken place (and if so, how much) during the last three years, (3) how satisfied the respondent is with his current income, and (4) whether he expects that his income will increase, decrease, or remain the same in the coming two years.10
32The transcripts of the parts of the interview concerning these four questions were analyzed with a coding scheme. For example, it was coded whether the initial answer of the respondent to the question was adequate or not, and if not, in how far the interviewer made a correct attempt at repair. The results of the analysis of the 804 (4*201) sequences are presented in Table 1.
Table 1: Outcome of attempts at repair of inadequate initial responses
33Fifty nine per cent of the sequences directly resulted in an adequate answer, for the respondent chose one of the response alternatives presented. However, in 49 % of the sequences the initial answer was not adequate, so in 332 sequences the interviewer had to make an attempt at repair. In three quarter of these 332 cases the interviewer indeed asked the respondent to provide as yet an adequate answer. If such an attempt at repair was made, it was successful in 86% of the cases. So the repair activities of the interviewers increased the number of adequate answers from 472 to 684, an improvement of 45 per cent. In these cases the interviewer clearly functioned as a help to the survey researcher.
34However, in one quarter of the cases in which an attempt at repair was needed, it was not made. The interviewer accepted the inadequate answer, chose a response code and asked the next question. These interviewers are no help to the researcher - on the contrary. Moreover, in 246 cases where an attempt at repair was made, this happened in an incorrect way, usually in the form of a leading question (“So it is X?”), a suggestion mostly endorsed by the respondent. This kind of ‘helping’ behavior of the interviewer is an obstacle on the road to valid, unbiased information.
35The coding of sequences makes it possible to see where, in which interview, the interviewer makes a correct and successful attempt at repair. So there are good reasons to analyze these fragments, and see what techniques of repair are most successful. However, the outcome of this analysis is quite disappointing, because there appears to be no one best repair technique, but there are dozens (Van der Zouwen & Smit, 2004b). Whether a certain repair technique will be successful depends on characteristics of the question, the type of inadequacy of the initial answer, but also on the creativity of the interviewer. So we have to give up the aim of making a short list of ‘best practices’.
36Seen in retrospect it is not so strange that especially with repairing deviations from the ideal question-answer sequence, standardization does not help. To elucidate this proposition, an excursion in the area of cybernetics, the science about control and communication (Wiener, 1965) has to be made.
37The interview may be described as a controlled communication process in which three actors are involved, each with there own set of tasks (Van der Zouwen, Dijkstra & Van de Bovenkamp, 1986; Van der Zouwen, 2001; Van der Zouwen & Smit, 2005); see Figure 1 below.
Figure 1: The interview as controlled communication process
38Firstly, there is the survey researcher who constructs the questionnaire, instructs the interviewers and analyzes the distributions of the response codes. Secondly, there is the interviewer who reads out the questions from the questionnaire, and evaluates the initial answers of the respondent. Adequate answers receive the corresponding response code; inadequate answers require further probing. In the middle of Figure 1 the third actor is located, the respondent, who by means of an internal cognitive process, interprets the question, searches for relevant information and reports his answer.
39The activities of each of these three actors are aimed at a specific goal. The goal of the survey researcher is to get comparable, relevant and unbiased information about the respondents. To reach this goal he has to control the interview process. In cybernetics, two types of control are distinguished: closed loop control and open loop control. Or, in other words, control by using feedback loops and control without the help of feedback loops.
40Ideally, in the standardized interview only open control takes place. Then the interview process directly proceeds from phase 1 till phase 5: The question as worded by the researcher (phase 1) is posed about exactly so by the interviewer (phase 2), and leads to the cognitive process within the head of the respondent (phase 3), resulting in an adequate answer (phase 4), correctly coded by the interviewer (phase 5) and entered into the data matrix of the survey researcher. But sometimes things go wrong, for example in the cognitive process within the respondent, leading to an inadequate answer, forcing the interviewer to make an attempt at repair. In that case there emerges a feedback loop from phase 4 back to phase 2.
41The first Law of Cybernetics states that closed loop control is only effective if there is enough variety in the set of control actions (Ashby, 1956/1964). That variety must be at least as large as the variety in the deviations that have to be controlled.
42So if the interviewer has to undertake a control action, her repertoire of action alternatives must be as extensive as the variety in deviations in the task related behavior of the respondent.
43There are survey researchers who, because of the necessary standardization, strongly restrict the repair behavior of their interviewers. In doing so they ignore the fundamental difference between open loop control and closed loop control of the interview process. One example of such an unnecessary curtailment of the action repertoire of the interviewer is that an inadequate answer of the respondent may only be repaired by again summing up all response categories. Another example is that the interviewer, when asked by the respondent what is meant with a certain term in the question, is only may allowed to react with “Whatever you think of it”. In this way these survey researchers hamper their interviewers to act as an effective help.
44These examples show that the relationship between researcher and interviewer is a reciprocal one. The interviewer can act as a help, but also as an obstacle for reaching the goal of the researcher, that is, obtaining valid and comparable answers of the respondent. And, reversely, the researcher can help or hinder the interviewer with reaching her goal, namely building a good ‘rapport’, and having an efficiently proceeding interview with the respondent.
45For that reason, researchers, when constructing the questionnaire, have to imagine the difficulties interviewers and respondents will have with certain questions. The ‘user friendliness’ of the questionnaire has to be thoroughly tested in test interviews with respondents belonging to the population to be researched; not only with understanding colleagues. During the actual fieldwork it has to be checked, preferably by listening to audio recordings of the interviews, whether the questionnaire functions as intended. These testing activities are time consuming and expensive. But in view of the high costs of the personal interview, skimping on these activities can be characterized as "penny wise and pound foolish". The expensive personal interview is meant for the administration of complex questionnaires under difficult circumstances. Therefore, it has to be made fit for these difficult circumstances
46Finally, in the literature about the standardized interview the interviewer is usually described as a source of error who, with her deviations from the question text, her choosing or leading probing, or even fraud, forms a considerable obstacle to reaching data of good quality. It would only be fair if the literature could be supplemented by papers describing how tactful interviewers can act when they repair inadequate answers, and how creative when they have to set off the shortcomings of the questionnaire designed by the survey researcher.