Chapter 6 - Designing a Questionnaire
In survey taking it is clear that special training and expertise are required to draw the sample, or to create any necessary computer programs. But what about writing the questions for the survey questionnaire? We've all been asking questions and obtaining answers from those around us throughout our lives. Aren't we already "question-asking experts"? Maybe. Maybe not!
Where to Start
The place to start in designing a questionnaire is with your data collection goals-What information do you need and from whom? Once these objectives have been clearly identified, the next step is to decide what pieces of specific information are needed to satisfy these objectives.
Many experienced questionnaire designers actually draft an outline of the final report, detailing how they will answer their fundamental data analysis concerns. This pinpoints exactly which pieces of information will be required and leads to the construction of a "data analysis plan"-which connects every data collection objective to each of the specific questions and how they should be asked. For example, consider answering an inquiry, such as How do people differ in their eating habits?
Visualize a questionnaire that captures:
- attitudes about food preferences and likely food choices in different circumstances
- self-reports of quantities, frequencies, and type of food intake
- age, income, and gender information to distinguish different groups
It may also be good to have a question in which people use their own words to describe their eating habits. Sometimes this approach can reveal whether the other questions were really understood.
The data analysis plan may be quite informal- a table or flowchart linking everything together at a high level. Whatever the formality, each broad goal should be clearly set and linked to each of the specific questions on the questionnaire as they are constructed.
The use of an analysis plan at this early stage may seem extravagant; however, it is one of the easiest ways, if kept updated, to ensure that the questionnaire contains everything that is needed and nothing extraneous.
The larger and more complex the inquiry, the more emphasis should be placed on an analysis plan. Otherwise, it becomes virtually impossible to keep all of the details in mind through the constant revisions a questionnaire undergoes. No one wants to come to the end of a $50,000 (or $500,000) survey project and discover that a critical variable was missing or was collected in the wrong way.
As the survey team approaches the point of constructing specific questions, they must decide whether the questionnaire will be self-administered or interviewer-administered. The team also must decide how to deliver the questionnaire-by mail or email, by fax, by telephone, or in person. Because the mode of data collection determines how questions and response options are constructed, this decision must be made early in the design process.
In many cases decisions about the collection mode will be driven by financial constraints or other resource limitations. Still, considerations such as overall questionnaire length, question complexity, and question sensitivity must be weighed in determining the mode of collection. For example, long questionnaires may not work well on the telephone, complex questions may require an interviewer to be sure that they are understood, and sensitive questions may be best done in a self-administered format.
After the mode of collection is determined-but before the designer can draft the first question-the data collection team has to "operationalize" all the variables. For example, continuing our earlier illustration, we must define what we mean by an "eating habit" and which behaviors will identify it. We might choose to define an eating habit as any of the following:
- food and drink actually consumed within the past 24 hours, whether typical or not
- most frequently consumed food items during a certain time period, such as last month
- food and beverages preferred when one is given a choice
- typical patterns of consumption, even if these patterns may not currently be in place.
We also have to decide whether the information is to reflect the patterns of food and beverage consumption for the individual respondent or whether we want the respondent to report for the entire household. These decisions should take into consideration what needs to be included in the final report. The essential task is to convey the same information to all respondents about what is wanted. Questions can be formatted for open-ended or close-ended responses.
"How many cups of coffee did you drink yesterday?"
Open-ended Response (specify number)
______ (enter answer)
Close-ended Response (circle one)
None 1 2 3 4 5 6 or more
Close-ended response choices must exhaust the entire range of answers. These choices must be mutually exclusive so that a single answer cannot fall into more than one category. The differences between the response choices should also be clear, so that respondents find it easy to select the response that best represents their answer.
In summary, questions and response choices need to be constructed so that respondents can be successful in giving answers that meet the analytic needs of the inquiry.
Good Question Structure
To design a good question, it is crucial that all the concepts be clear and simply expressed. The designer must think about how the answer to the question will be processed and prepared for analysis.
If there are terms with precise or technical meanings that everyone should use, these definitions must be included in the questionnaire and respondents should be instructed in them. It is essential to provide respondents with the tools necessary to translate their varied experiences accurately into a common, relevant set of response options.
One of the first points to think about (and one that will emerge quickly if an analysis plan is used) is whether a particular question is included primarily to make comparisons over time or comparisons across groups. Question consistency becomes paramount, for example, if the new data are to be compared with previous versions of the same questionnaire or with previous studies that collected some of the same information.
It is often easier to ensure consistency by repeating word for word the earlier question. A dilemma may arise if it appears that the previous question is flawed (when retested, as it should be in the new context). Social changes since the previous survey also may have altered the meaning of terms or the frequency of behaviors. Consider the question:
"About how many times did you speak with someone on the telephone today?"
How would the range of response options have to change if the goal is to compare an office worker's answer in the 1920s to what an office worker might say now? In the 1920s, the response options might have been:
None 1 2 3 4 5+
These options seem unreasonable in the current business world. Today, we might have:
None 1-5 6-10 11-15 16-20 21+
For a comparison over time, it would be better if the response options were:
None 1-4 5-8 9-12 13-16 17-20 21+
This way, with an extra category, there would be a better contrast between today's office world and that of the 1920s. Another factor to consider is that the range of response categories affects how people think about a question. Pretesting should be conducted to tell if this is occurring. (See the How to Conduct Pretesting Chapter in this booklet for more information.)
Conveying Required Precision
When a question is being created- particularly one requesting information about the frequency of a behavior-it is important for researchers to agree in advance on the level of precision being asked from the respondent.
If respondents are asked to estimate the frequency of their behavior, the questions may be prefaced by such phrases as "roughly how often?" or "about how many?"
It may be necessary to ask the respondents to count the exact frequency of events within a set period of time or to otherwise request that they be as precise as possible. You may ask them to consult records (assuming these are handy and do not overly delay the data collection or raise the chance of the interview breaking off before completion).
If precision needs are not conveyed clearly to all respondents, one person may choose to estimate within very broad ranges and another may make an effort to closely count the episodes or behaviors. The result would be that these various respondents would be answering different questions and their data would not be comparable. Remember, respondents are not mind readers; they cannot be expected to guess what is desired by the researcher or questionnaire designer.
Many concepts we ask people to report on in surveys do not have universally agreed-upon definitions. Surprisingly, there is little social consensus about the definitions of some commonplace everyday terms. Straightforward words, such as job, work, or income, can have many nuances and different meanings for different people.
There are many things that may make a question difficult to answer and should be avoided. For instance,
- questions that tax the respondent's memory,
- questions that ask for details that may never have been committed to memory.
Pretesting such questions will quickly reveal the problem. Likewise, questions that ask for sensitive or self-incriminating information (e.g., on illegal drug use or cheating on taxes) are ones respondents may not want to answer. For the most sensitive types of information, questionnaires may need to be self administered with an unbreakable guarantee of respondent anonymity.
Use extra caution when developing new questions-a great deal of preliminary effort is needed. Questionnaire designers budget a good deal of time for this.
The Questionnaire as a Whole
Respondents are more likely to cooperate if the questions are simple, clear, easy to answer, and personally relevant to them. It is recommended that questionnaires be written at the 5th-grade reading level.
When you think you've finished the individual questions, step back and look at the questionnaire as a whole. Remember, the questionnaire is a total package and needs to be considered as such.
- It needs a strong introduction conveying to the respondent what the survey is about.
- It should indicate why the questions are being asked.
- It needs interesting and readily answerable questions at the beginning to gain respondent attention and build rapport.
- The conclusion should be gentle and friendly, expressing gratitude for the respondent's time and effort.
The questions need to flow well from one to the next, and designers should be aware that earlier questions provide information and context to the respondents that they may use in later answers. Often the answer to one question may influence the answer to a later question. For instance, suppose respondents are asked first How do you feel about your job? and later on How do you feel about life in general?
Answers to the second question may be tempered by the first question. Because respondents have already reported their feelings about their job, including those feelings in the second answer may be redundant. On the other hand, if their job is very important to them (or salient for some other reason), then the answer to the first question may be used when constructing the second answer. These so-called "order effects" are difficult to predict and often become apparent only through field tests of the questionnaire, in which different orderings of the questions are compared.
KISS Principle- Keep It Simple, Statistician
The three most important things for any questionnaire designer to remember are simplicity, simplicity, and simplicity. Ideas need to be conveyed clearly and questions should be easy to comprehend. There must be no guesswork for the respondent when it comes to understanding exactly what information is being requested.
Most questionnaires are not about trivial matters. It is the questionnaire designer's greatest challenge to take important topics and translate them into simple concepts, simple behaviors, and simple words. The style of the questionnaire must not get in the way of respondents' providing their information; otherwise the result could be incomplete or misleading data, item refusals, respondent fatigue effects-even the respondent's refusal to complete the questionnaire.
It is a good idea to try out the questions on many different people-even as the questions are evolving. At different stages of development, the entire questionnaire should be tested to identify weaknesses and potential difficulties.
Think about which respondents might have the most problems answering the questions, and deliberately seek out those respondents for pretests. Another good method for identifying difficulties is for the questionnaire designers to actually serve as respondents and answer the questions themselves. It is amazing what insight may be gained by turning the tables in this way. The questionnaire designer must understand the need to pretest, pretest, and then pretest some more.
Where Can I Get More Information
Cognitive psychology is making major inroads into survey practice, not only in questionnaire design but also improving methods for eliciting informed consent on questions of privacy and the effective use of incentives. For more information on these changes as they affect questionnaire design, see Cognition and Survey Research, Sirken, et al (1999), Wiley.