Chapter 2 - How to Plan a Survey
A survey usually originates when an individual or institution is confronted with an information need and the existing data are insufficient.
At this point, it is important to consider if the required information can even be collected by a survey. Maybe it cannot? Is an experiment needed instead? Perhaps only an indirect way of measuring is possible.
If a survey is decided upon, the first step is to lay out the objectives of the investigation. This is generally the function of the sponsor of the inquiry.
A sponsor may be...a government agency trying to assess the impact on the primary recipients and their families of a social welfare program...a university researcher examining the relationship between actual voting behavior and expressed political beliefs...a computer maker gauging the level of customer satisfaction among existing and potential purchasers.
The objectives of a survey should be as specific, clear-cut, and unambiguous as possible. Trade-offs typically exist and sometimes this only becomes apparent as the planning process proceeds. Therefore, it is important to make the sponsor a full participant in every planning step.
How to Plan a Survey Questionnaire
First, the mode of data collection must be decided upon (e.g., mail, telephone, or in person). Once this has been determined a questionnaire can then be developed and pretested.
Planning the questionnaire is one of the most critical stages in the survey development process. Social and behavioral scientists have given a great deal of thought to the design issues involved.
Questionnaire construction has elements that often appear to be just plain commonsense, but, when they are implemented, may involve some subtlety. It is common sense to require that the concepts be clearly defined and questions unambiguously phrased; otherwise, the resulting data are apt to be seriously misleading.
Consider how we might apply this strategy in a survey to estimate the incidence of robbery victimization. One might start out by simply asking, "Were you robbed during the last six months?" Although apparently straightforward and clear-cut, the question does present an ambiguous stimulus. Many respondents are unaware of the legal distinction between robbery (involving personal confrontation of the victim by the offender) and burglary (involving breaking and entering but no confrontation).
Therefore, in the National Crime Survey conducted by the U.S. Bureau of the Census, the questions on robbery victimization do not mention "robbery." Instead, there are several questions used; when taken together, they seek to capture the desired responses by using more universally understood phrases.
See the following example from the National Crime Victim Survey Questionnaire.
I'm going to read some examples that will give you an idea of the kinds of crimes this study covers.
As I go through them, tell me if any of these happened to you in the last 6 months, that is since _________ _________, 200_.
Was something belonging to YOU stolen, such as-
- Things that you carry, like luggage, a wallet, purse, briefcase, book-
- Clothing, jewelry, or calculator-
- Bicycle or sports equipment-
- Things in your home-like a TV, stereo, or tools-
- Things from a vehicle, such as a package, groceries, camera, cassette tapes-
- Did anyone ATTEMPT to steal anything belonging to you?
Briefly describe incident(s)
Designing a suitable questionnaire entails more than well-defined concepts and distinct phraseology. Attention must also be given to its length. Long questionnaires are apt to induce respondent fatigue and errors arising from inattention, refusals, and incomplete answers. They may also contribute to higher non-response rates in subsequent surveys involving the same respondents.
There are other factors to take into account when planning a questionnaire. These include such diverse considerations as...the order in which the questions are asked...their appearance... even such things as the questionnaire's physical size and format.
How to Get Good Coverage
A critical element in any survey is to locate (or "cover") all the members of the population being studied so that they have a chance to be sampled. To achieve this, a list-termed a "sampling frame"-is usually constructed.
In a mail survey, a frame could be all of the postal addresses in Tampa, Florida....for an in-person business survey, a frame might be the names and addresses of all the retail establishments licensed in Westchester County, New York...in a telephone survey at The George Washington University in Washington, DC, the frame might simply be a list of student names and telephone numbers.
A sampling frame can also consist of geographic areas with well-defined natural or artificial boundaries, when no suitable population list exists (as might be true in some parts of rural America). In this instance, a sample of geographic areas (referred to as "area segments") is selected and interviewers canvass the sample area segments and list the appropriate units- households, retail stores or whatever-so that these units have a chance of being included in the final sample.
The quality of the sampling frame-whether it is up-to-date and complete- is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed.
Selecting a sample of households for a telephone interview is easier than that for an in-person interview. The telephone survey is generally less expensive and simpler to carry out. Its one main drawback is that only about 95 percent of all households have telephones. Therefore, some people will be missed. Persons without telephones generally have much lower incomes than those in households with telephones-so telephone surveys do not adequately re p resent the low-income population. Sampling from a frame of all possible telephone numbers, including unlisted ones, is called random digit dialing (RDD). This may seem relatively easy today but "weeding out" non-residential telephone numbers can be difficult. Nonetheless, several ingenious methods have been developed to enable RDD samples to be picked in an efficient way.
How to Choose a Random Sample
Virtually all surveys taken seriously by social scientists and policymakers use some form of random sampling.
Even the U.S. Decennial Census employs sampling techniques for gathering the bulk of the data items. Complete (100 percent) enumeration is used for just the basic population counts-only a subset receives the so-called "long form."
Methods of random sampling are well grounded in statistical theory and in the theory of probability. Reliable and efficient estimates of needed statistics can be made by surveying a carefully constructed sample of a population. This is provided, of course, that a large proportion of the sample members give the requested information.
The particular type of sample used depends upon the objectives and scope of the survey. Factors include the nature of potentially available frames, the overall survey budget, the method of data collection, the subject matter, and the kind of respondent needed.
Some types of samples are straightforward, requiring little in the way of experience or training; others are highly complex and may require many stages of selection. Consider the range of difficulty between a sample of sixth graders in a particular school on the one hand and a sample of the homeless in the same city on the other.
Whether simple or complex, the goal of a properly designed sample is that all of the units in the population have a known, positive chance of being selected. The sample plan also must be described in sufficient detail to allow a reasonably accurate calculation of sampling errors. These two features make it scientifically valid to draw inferences from the sample results about the entire population that the sample represents.
Ideally, the sample size chosen for a survey should be based on how precise the final estimates must be. In practice, usually a trade-off is made between the ideal sample and the expected cost of the survey.
How to "Plan In" Quality
An integral part of a well-designed survey is to "plan in" quality all along the way. One must devise ways to keep respondent mistakes and biases to a minimum. For example, memory is important when the respondent is expected to report on past events, such as in a consumer expenditure survey. In these "retrospective" surveys it is essential that the respondent not be forced to report events that may have happened too long ago to be remembered accurately.
Other elements to pretest during the planning phase include...whether any of the questions are too sensitive...whether they unduly invade the respondent's privacy...or whether they are too difficult even for a willing respondent to answer. Each of these concerns has an important bearing on the overall statistical validity of the survey results.
Deciding on the right respondent in a household sample is a key element in "assuring" quality. For surveys where the inquiry is basically factual in nature, any knowledgeable person may be asked to supply the needed information. This procedure is used in the Current Population Survey (CPS), where any responsible adult in a household is expected to be able to provide accurate answers to employment or unemployment questions.
In other surveys, a so-called "household" respondent may produce erroneous or even invalid information -for example, when the information is known only by a specific individual and no one else.
A different, but related, issue arises in "attitude" surveys. It is generally accepted that a randomly chosen respondent produces a more valid cross-section of opinion than does a nonrandomly selected household respondent. This is because a nonrandomly selected individual, acting as a household respondent, is more likely to be someone who is at home, so the working public and their attitudes would be underrepresented.
One final point: for a quality product, checks must be made at every step to ensure that the sample is selected according to specifications; that the interviewers do their work properly; that the information from the questionnaires is coded accurately; that computer data entry is done correctly; and that the computer programs used for data analysis work properly.
How to Schedule
How much time should be allotted for a survey? This varies with the type of survey and the particular situation. Sometimes a survey can be done in two or three weeks-if it involves a brief questionnaire and if the data are to be collected by telephone from a list already available. More commonly, a survey of 1,000 individuals or more could take anywhere from a few months to one year- from initial planning to having results ready for analysis.
The steps in a survey are not necessarily sequential; many of them can be overlapped. Some, such as listing and sampling housing units in the areas to be covered, can be carried out while a questionnaire is being put into final form. Although they are not additive, all of these steps are time consuming. Perhaps the most common planning error is to underestimate the time needed by making a global estimate, without considering these individual stages.
How to Budget
A checklist of budget factors, such as this partial one, may be useful in estimating total survey costs (whether in time or money). A "traditional" (paper and pencil) in-person interview survey will be used to illustrate the budget steps. Many of these are general; however, increasing use of survey automation is altering costs- reducing some and adding others.
- Staff time for planning the study and steering it through the various stages, including time spent with the sponsor in refining data needs
- Sample selection costs, including central office staff labor and computing costs
- For "area segments" samples, substantial field staff (interviewer) labor costs and travel expenses for listing sample units within the segments
- Labor and material costs for pretesting the questionnaire and field procedures; the pretesting step may need to be done more than once and money and time should be set aside for this (especially when studying something new)
- Supervisory costs for interviewer hiring, training, and monitoring
- Interviewer labor costs and travel expenses (including meals and lodging, if out of town)
- Labor and expense costs of redoing a certain percentage of the interviews (as a quality assurance step) and for follow-up on non-respondents
- Labor and material costs for getting the information from the questionnaire onto a computer file
- Cost of spot-checking the quality of the process of computerizing the paper questionnaires
- Cost of "cleaning" the final data-that is, checking the computer files for inconsistent or impossible answers; this may also include the costs of "filling in" or imputing any missing information
- Analyst costs for preparing tabulations and special analyses of the data; computer time for the various tabulations and analyses
- Labor time and material costs for substantive analyses of the data and report preparation
- Potentially important are incidental telephone charges, postage, reproduction and printing costs for all stages of the survey- from planning activities to the distribution of results
A good survey does not come "cheap," although some are more economical than others.
As a rule, surveys made by an in-person interviewer are more expensive than those made by mail or by telephone. Costs will increase with the complexity of the questionnaire and the amount of data analysis to be carried out.
Surveys that involve a large number of interviews tend to be cheaper on a per-interview basis than surveys with fewer interviews. This is particularly so when the sample size is less than 1,000 respondents, because "tooling up" is involved for just about any survey-except one that is to be repeated on the same group.
Where Can I Get More Information
The quality of survey efforts can suffer because there is too little time set aside to do planning. Two important general sources to examine on planning in quality are Juran (1989) Quality Planning and Deming (1986) Out of the Crisis. The publications of the American Society for Quality are also worth looking for and can be searched for on the Internet. Software tools like MS Project are also useful in making planning more systematic and in increasing the efficiency of lessons learned, as Juran calls them from one survey to the next.