Sampling Fundamentals - Nonresponse Problems
Sampling Fundamentals - Nonresponse Problems
The object of sampling is to obtain a body of data that is representative of the population. Unfortunately, some sample members become nonrespondents because they (1) refuse to respond, (2) lack the ability to respond, (3) are not at home, or (4) are inaccessible.
Nonresponse can be a serious problem. It means, of course, that the sample size has to be large enough to allow for nonresponse. If a sample size of 1000 is needed and only a 50-percent response rate is expected, then 2000 people will need to be identified as possible sample members. Second, and more serious, is the possibility that those who respond differ from nonrespondents in a meaningful way, thereby creating biases.
The seriousness of nonresponse bias depends on the extent of the nonresponse. If the percentage involved is small, the bias is small. Unfortunately, however, as the discussion in Chapter 7 made clear, the percentage can be significant. For example, a review of 182 telephone studies found a refusal rate of 28 percent. Further, this level is likely to increase with the increase in the use of telemarketing and the problem is generally more severe in home personal interviews and worse in mail surveys where non-response of 90 percent is not uncommon.
The nonresponse problem depends on how the nonrespondents differ from the respondents particularly upon the key questions of interest. The problem is that the very act of being a nonrespondent often implies a meaningful difference. Nonrespondents to in-home interviews tend to be urban dwellers, single or divorced, employed, and from the higher social classes. A comparison of 100 nonrespondents to a telephone interview with 100 respondents revealed that refusals were associated with older age, lower income, nonparticipation in the workforce, interest in the question, and concern with invasion of privacy. Clearly, the nonresponse problem can be substantial and significant in many studies although its impact will be dependent on the context.
What can be done about the nonresponse problem? A natural tendency is to replace each nonrespondent with a "matched" member of the sample. For example, if a home is included in the sample but the resident is not at home, a neighbor maybe substituted. The difficulty is that the replacement cannot be matched easily on the characteristic that prompted the non-response, such as being employed or being a frequent traveler. Three more defensible approaches are (1) to improve the research design to reduce the number of nonresponses, (2) to repeat the contact one or more times (callbacks) to try to reduce nonresponses, and (3) to attempt to estimate the nonresponse bias.
Improving the Research Design
In Chapter 7, the refusal problem was discussed in some detail, along with suggestions on how to improve the research design to reduce the incidence of refusals. The challenge in personal and telephone interviewing is tp gain initial interest and to generate rapport through interviewer skill and the design and placement of questions. In mail surveys, the task is to motivate the respondent, through incentives and other devices, to respond, The number of not-at-homes can~l^feduced by scheduling calls with some knowledge of the respondents' likely activity patterns. For example, midday is obviously a bad time to reach employed homemakers. Sometimes it is useful to make a telephone appointment for an in-home interview, although one study found that this tactic tended to increase the refusal rate.10
Call-backs
Call-backs refer to overt new attempts to obtain responses. The use of callbacks is predicated on the assumption that they will generate a useful number of additional responses and that the additional responses will reduce meaningfully a nonresponse bias. If the nonresponse is due to refusals or the inability to respond, call-backs may not reduce significantly the number of nonrespondents. It is most effective for the not-at-home nonrespondent. For some surveys, it may be worthwhile to use as many as six callbacks to reduce the number of nonrespondents to acceptable levels, although the first and second call-backs are usually the most productive.11 The efficiency of the call-backs will be improved by scheduling them at different times of the day and week.
In a mail survey, the call-back is particularly important, because the nonresponse level can be so high. As was noted in Chapter 7, it is common practice to send the questionnaire to prompt nonrespondents at regular intervals.
Estimating the Effects of nonresponse
One approach is to make an extra effort to interview a subsample of the nonrespondents. In the case of a mail survey, the subsample might be interviewed by telephone. In a telephone or personal survey, an attractive incentive, such as a worthwhile gift, might be employed to entice a sample of the nonrespondents to cooperate. Often, only some of the critical questions thought to be sensitive to a nonresponse bias are employed in this stage.
The Politz approach is based on the fact that not-at-homes can be predicted from a knowledge of respondents' frequency of being away from home.12 The respondents are asked how many evenings they are usually at home (if the interviewing is to be done in the evening). This information serves to categorize them into groups that can serve to represent the not-at-home respondents. For instance, if a respondent is usually at home only one night a week, it might be assumed that there are six more similar ones among the nonrespondents. On any given night, there would be only one chance in seven of finding one home. Thus, on the average, six homes with people with this tendency to be away would have to be contacted to find one person at home. This respondent is therefore assumed to represent six of the nonrespondents. There are uncertainties introduced by this approach, but it does provide a way to proceed, especially when call-backs are costly.
The object of sampling is to obtain a body of data that is representative of the population. Unfortunately, some sample members become nonrespondents because they (1) refuse to respond, (2) lack the ability to respond, (3) are not at home, or (4) are inaccessible.
Nonresponse can be a serious problem. It means, of course, that the sample size has to be large enough to allow for nonresponse. If a sample size of 1000 is needed and only a 50-percent response rate is expected, then 2000 people will need to be identified as possible sample members. Second, and more serious, is the possibility that those who respond differ from nonrespondents in a meaningful way, thereby creating biases.
The seriousness of nonresponse bias depends on the extent of the nonresponse. If the percentage involved is small, the bias is small. Unfortunately, however, as the discussion in Chapter 7 made clear, the percentage can be significant. For example, a review of 182 telephone studies found a refusal rate of 28 percent. Further, this level is likely to increase with the increase in the use of telemarketing and the problem is generally more severe in home personal interviews and worse in mail surveys where non-response of 90 percent is not uncommon.
The nonresponse problem depends on how the nonrespondents differ from the respondents particularly upon the key questions of interest. The problem is that the very act of being a nonrespondent often implies a meaningful difference. Nonrespondents to in-home interviews tend to be urban dwellers, single or divorced, employed, and from the higher social classes. A comparison of 100 nonrespondents to a telephone interview with 100 respondents revealed that refusals were associated with older age, lower income, nonparticipation in the workforce, interest in the question, and concern with invasion of privacy. Clearly, the nonresponse problem can be substantial and significant in many studies although its impact will be dependent on the context.
What can be done about the nonresponse problem? A natural tendency is to replace each nonrespondent with a "matched" member of the sample. For example, if a home is included in the sample but the resident is not at home, a neighbor maybe substituted. The difficulty is that the replacement cannot be matched easily on the characteristic that prompted the non-response, such as being employed or being a frequent traveler. Three more defensible approaches are (1) to improve the research design to reduce the number of nonresponses, (2) to repeat the contact one or more times (callbacks) to try to reduce nonresponses, and (3) to attempt to estimate the nonresponse bias.
Improving the Research Design
In Chapter 7, the refusal problem was discussed in some detail, along with suggestions on how to improve the research design to reduce the incidence of refusals. The challenge in personal and telephone interviewing is tp gain initial interest and to generate rapport through interviewer skill and the design and placement of questions. In mail surveys, the task is to motivate the respondent, through incentives and other devices, to respond, The number of not-at-homes can~l^feduced by scheduling calls with some knowledge of the respondents' likely activity patterns. For example, midday is obviously a bad time to reach employed homemakers. Sometimes it is useful to make a telephone appointment for an in-home interview, although one study found that this tactic tended to increase the refusal rate.10
Call-backs
Call-backs refer to overt new attempts to obtain responses. The use of callbacks is predicated on the assumption that they will generate a useful number of additional responses and that the additional responses will reduce meaningfully a nonresponse bias. If the nonresponse is due to refusals or the inability to respond, call-backs may not reduce significantly the number of nonrespondents. It is most effective for the not-at-home nonrespondent. For some surveys, it may be worthwhile to use as many as six callbacks to reduce the number of nonrespondents to acceptable levels, although the first and second call-backs are usually the most productive.11 The efficiency of the call-backs will be improved by scheduling them at different times of the day and week.
In a mail survey, the call-back is particularly important, because the nonresponse level can be so high. As was noted in Chapter 7, it is common practice to send the questionnaire to prompt nonrespondents at regular intervals.
Estimating the Effects of nonresponse
One approach is to make an extra effort to interview a subsample of the nonrespondents. In the case of a mail survey, the subsample might be interviewed by telephone. In a telephone or personal survey, an attractive incentive, such as a worthwhile gift, might be employed to entice a sample of the nonrespondents to cooperate. Often, only some of the critical questions thought to be sensitive to a nonresponse bias are employed in this stage.
The Politz approach is based on the fact that not-at-homes can be predicted from a knowledge of respondents' frequency of being away from home.12 The respondents are asked how many evenings they are usually at home (if the interviewing is to be done in the evening). This information serves to categorize them into groups that can serve to represent the not-at-home respondents. For instance, if a respondent is usually at home only one night a week, it might be assumed that there are six more similar ones among the nonrespondents. On any given night, there would be only one chance in seven of finding one home. Thus, on the average, six homes with people with this tendency to be away would have to be contacted to find one person at home. This respondent is therefore assumed to represent six of the nonrespondents. There are uncertainties introduced by this approach, but it does provide a way to proceed, especially when call-backs are costly.
Comments
Post a Comment