Chapter 4 - Judging the Quality of a Survey
One of the most famous examples of a poorly conceived survey is the 1948 poll that predicted Harry Truman would lose the presidential election to Thomas Dewey. The survey's main flaw was its sample, which failed to fairly represent all segments of the American electorate-particularly those who eventually voted for Truman.
Survey Non-response and Measurement
Problems with the sample are not the only source of uncertainty in survey findings. Non-response occurs when members of the sample cannot-or will not- participate in the survey. Measurement difficulties are linked to problems in gathering the data used to generate survey results. Although some problems with inferior surveys can be attributed to negligence or mistakes, many problems are unavoidable and can only be minimized rather than eliminated altogether. For example, non-response is nearly inevitable for most surveys because some members of the sample will refuse to participate- despite every reasonable effort made by the survey taker. This pamphlet examines a few of the more common problems arising in surveys and how competent survey takers may handle them.
How Do Problems Affect Survey Results?
Survey problems lead to either of two effects on survey results. Bias is the tendency for findings to be off the mark in projecting from the sample to what is happening in the population as a whole. Variance, on the other hand, is a less predictable effect that may cause projections to be higher one time but lower the next.
Where Do Problems Arise in Surveys?
Difficulties may arise at any point during these basic steps of the survey process:
Strategies To Deal With Survey Problems
- Organization-The survey taker determines who is to be sampled and what is to be learned about the sample.
- Questionnaire Design-Based on the goal of the survey, questions for survey respondents are prepared and arranged in a logical order to create the survey questionnaire.
- Sampling-A repeatable plan is developed to randomly choose a sample capable of meeting the survey's goals. Then a sample is selected.
- Data Collection-A plan for contacting the sample and collecting information from participants is developed and carried out.
- Data Processing-Collected data are entered into the computer and checked for accuracy.
- Analysis-The results of the survey are compiled and disseminated.
There are many and varied strategies for dealing with survey problems, although most can be described as an effort to:
- Prevent the problem
- Adjust the survey data to compensate
- Measure any remaining effect of the problem
To the extent resources will allow, all three types of remedies are at least considered in planning the best surveys.
Three examples of real surveys will help to illustrate how the types of remedies are used to deal with some common survey problems:
- A state-wide mail survey of high school football coaches to profile the use of athletic trainers for varsity football teams
- A county-wide telephone interview survey to poll adults' views on an upcoming school bond referendum
- A national in-person Interview survey to find out how often, on average, people visited a doctor in the last year.
Sampling: Specific Problems and Remedies
Sampling problems are tied to how the sample is chosen and to how the collected survey data are used to produce findings. Sampling problems can cause either bias or variance effects in survey results.
SPECIFIC SAMPLING PROBLEMS
- Imprecise Findings-One common source of error in all three survey examples arises because the findings are extrapolated from a sample rather than obtained directly from the entire population.
Increase the sample size, particularly for the most important and heterogeneous segments of the population.
Choose a stratified sample. This might be done in the mail survey by selecting separate samples for a number of school categories defined by student enrollment. This stratified sampling of schools by size would improve findings for the state, if those in larger schools are different (e.g., more likely to hire trainers) than those in smaller schools.
- Findings that Disregard the Sample Design-The plan for selecting football coaches in the mail survey might call for those at private schools to be sampled at a relatively higher rate to assure that the number of respondents from this type of school is large enough. Failure to account for the relative oversupply of private schools in the sample during data analysis would cause a biased underestimate in the projected percentage of the state's high school football teams that have a trainer, if private schools are less likely to have them.
Give survey data from private schools relatively less influence in shaping the final results projected for the state.
- Incomplete Sample Coverage-Some lists used to select survey samples exclude parts of the population (e.g., adults without access to a telephone in the school bond survey). In most cases those excluded differ from those included, thus creating a nonrandom imbalance in the resulting sample. An undercoverage problem like this in the telephone survey example would produce a biased underestimate of the level of support for the school bond, if those without a telephone tended to favor it more strongly.
Figure out the percentage of adults in the county who have no access to a telephone.
Adjust the findings to try to account for any sample imbalance.
Nonresponse: Specific Problems and Remedies?
Survey nonresponse often biases survey results because it makes the sample less representative of the population. For example, there tends to be an overrepresentation of female respondents in surveys of the general public because women are usually more likely to participate than men.
Most preventive remedies for nonresponse are tied to the fact that its biasing effect on survey results is lowest when the percentage of the eligible members of the sample who participate (i.e., response rate) is high.
SPECIFIC NONRESPONSE PROBLEMS
- Nonresponse In Mail Surveys- if the 30 to 50 percent of football coaches who complete the mail survey questionnaire are more likely to have trainers than those who do not respond, then the findings from the survey would tend to exaggerate the use of trainers in the state's high schools.
Offer cash or some other valued reward for participating in the survey.
Adjust the findings to account for sample imbalance.
Send reminders or make follow-up telephone calls to those who do not respond after the first mailing.
- Nonresponse in Telephone Surveys-If the survey of football coaches were done by telephone, the higher 60 to 80 percent response rate ordinarily would be expected to cause the nonresponse bias to be less than in the mail survey.
- Nonresponse to In-Person Surveys-If the survey of coaches were collected through an in-person interview, the expected 80 to 95 percent response rate would cause the lowest level of nonresponse bias among the three approaches (mail, telephone, in-person) to data collection.
The following remedies, and the first two for mail surveys, can be used for both nonresponse in telephone and in-person surveys.
Develop a plan to be uniformly applied in calling each member of the sample, requiring that calls be made at various times when coaches are available.
Allow as many attempts to interview each selected football coach as resources permit.
Prepare the interviewers with effective responses to concerns about the survey that reluctant coaches might express.
- Nonresponse to Certain Questions- A selected adult in the school bond survey may agree to participate in the interview but rightfully decline to answer some of the questions. This type of nonresponse is more common for questions on sensitive or invasive topics (e.g., sexual behavior or family income).
A Potential Partial Remedy
Replace the missing answer with a substitute one that is chosen at random from other similar participants who answered the question.
Measurement: What Are Some Specific Problems and Remedies?
A measurement problem occurs when the answers provided by the respondent do not match the data actually needed. This discrepancy is usually tied to
- Questionnaire content
- How well the respondent answers the survey questions
- (In interview surveys) How appropriately the interviewer asks the survey questions.
SPECIFIC MEASUREMENT PROBLEMS
- Inability to Recall Answers-Asking a respondent to remember the number of doctor visits during the last year is likely to contribute to a biased underestimate of the average number of visits per person. This happens because people tend to underreport less prominent or more distant past events.
Encourage respondents to use personal schedules, insurance records, and other sources to help them remember.
If possible, shorten the length of the period for which doctor visits are to be counted (e.g., to the last two weeks rather than the last calendar year).
- Leading Questions- Using the following question to obtain adults' views in the telephone survey might bias the results in favor of the referendum: "Wouldn't you say it's about time for our county to pass the school bond referendum?" Phrasing an opinion question this way leads the respondent to a "yes" answer and a distorted perspective of the public's views on the issue.
Ask the question more objectively (e.g., by using: "Do you favor or oppose the school bond referendum?").
- Unclear Question Wording-The lack of a clear working definition for "doctor visit" would lead to a troublesome measurement problem in the in-person interview survey. For instance, some might consider an optometrist, chiropractor, or osteopath to be a "doctor," but others might not. To some a "visit" would happen only if the patient traveled to the doctor, but to others it would include house calls. The effect of allowing variable interpretations of key words and phrases in survey questions is to reduce the precision of survey results.
Try out the question on a small but broad cross-section of likely respondents before interviewing starts.
Find out what is confusing about the phrase, and then clarify the interviewer or respondent instructions as needed.
Check the interviewer carefully throughout the data-collection phase (especially early on), to make sure that definitions of these terms are correctly interpreted for respondents.
How Good IS a Particular Survey?
The potential for problems is a reality in all surveys today. The good news is, however, that researchers have found at least partially effective ways to deal with most problems that occur.
The main issue for the discriminating user of results from any survey is to determine whether
Problems like those described previously were recognized.
Steps were thoughtfully taken to deal with them.
Indeed, the quality of a survey is best judged not by its size, scope, or prominence, but by how much attention is given to dealing with all the many important problems that can arise.
Where Can I Get More Information
The Section for Research on Survey Methods of ASA periodically publishes best practice volumes and these should be examined since the measurement of survey quality continues to improve, both by applying the common sense methods highlighted in this Chapter and through other more technical advances of the many capable practitioners in this field. The work of Deming and Juran have been cited earlier (in Chapter 2) and their general advise can always be valuable. Closer to home and an important resource is the Kalsbeek and Lessler (1991) book entitled Nonsampling Errors.