Survey Research

Overview Survey research is the method of gathering data from respondents thought to be representative of some population, using an instrument composed of closed structure or open-ended items (questions). This is perhaps the dominant form of data collection in the social sciences, providing for efficient collection of data over broad populations, amenable to administration in person, by telephone, and over the Internet. Some forms of survey research by telephone or the Internet may be completely automated. Critics of survey research methodology hold it to be a method which artificially forces respondents to formulate opinions, masking the complexity of conflicting views and unconscious biases within each respondent, and critics note that in many arenas (e.g., race relations) survey items poorly predict actual behavior. See also focus group research and also the separate sections on sampling and on standard measures and scales. Also, analysis of repeated surveys is discussed in the separate section on time series analysis.

Contents Key concepts and terms Data collection Types of computer support Survey design issues Types of items Item bias Assumptions Frequently asked questions Survey software Bibliography

Key Terms and Concepts

• Survey Instrumentation. The survey instrument is name given to the schedule of questions or response items to be posed to respondents. It is equivalent to the terms questionnaire or interview schedule, depending on the type of survey research being conducted.

• Interviews and questionnaires. Ordinarily, "interviews" refers to the face-to-face administration of a survey instrument, whereas "questionnaires" refers to mail or other indirect methods of administration.

• Items are the individual survey questions or statements for which a response is solicited from the respondent.

• Survey model. A social science model of any phenomenon or outcome can be depicted graphically as a set of variables (represented by circles) connected by a set of causal effects (represented by single-headed arrows) and correlations (represented by double-headed arrows). The survey should, at a minimum, include items pertaining to each circle in the model. Multiple items to measure each variable and relationship are preferred because that is necessary in order to model error (not just variables) using techniques such as structural equation modeling. Also, multiple items for each construct (circle in the model) provide stability of measurement and thus increase the reliability of the model.

• Response rate is discussed in the section on sampling.

• Response structure refers to the format of items, whcih may be open-ended or structured. Structured items may be multiple choice, multiple response, dichotomies, rank lists, or a variety of other formats.

• Survey error, over and above item bias, survey error includes such factors as faults in sampling, coding, tabulating, data processing, interviewer bias, researcher bias, and data misinterpretation.

• Pretesting is considered an all-but-essential step in survey research. No matter how experienced the survey researcher, pretests almost invariably bring to light item ambiguities and other sources of bias and error. In fact, Converse and Presser (1986: 65) argue cogently that a minimum of two pretests are necessary, with pretest sizes of 25 - 75 (75 is best) administered to respondents similar to those who will be in the final sample. The first pretest may have up to twice the number of items as the final, as one purpose is to identify weaker items and drop them from the survey. Items may also be dropped if the first pretest shows they exhibit too little variance to be modeled. The first pretest will also have many more probe items that the final, and respondents may even be told they are being given a pretest and their help solicited in refining the instrument.
  • Other purposes of the first pretest are to see if respondent interest is aroused, if respondent attention can be maintained, if interviewers and respondents feel the survey has a natural flow, and if interviewers experience problems such as need to repeat questions, need to correct misinterpretations of questions, need to handle volunteered information, sections where the respondents wanted to say more, or questions which seemed sensitive. Pretests also check for such practical things as whether there is enough white space provided for open-ended items and if instructions for returning the questionnaire are clear in the case of mail surveys. In all surveys, pretests provide statistical information (hence the desirability of 75, not 25, pretesters) on whether there is sufficient variance in the dependent and other critical variables, and if there are too many responses in residual categories. The first pretest results, hopefully, in a much-changed, finished instrument. The second pretest is used simply for polishing, trimming, rearranging, and other refinements, but not adding new dimensions or making major substantive changes in the survey. If such changes prove necessary, a third pretest will also be needed.

• Split sample comparisons. After pretesting, if there is any remaining doubt about the reliability of one or more items, the researcher should consider split sample comparisons, where two versions of the same item appear on two different survey forms administered randomly. If the mean responses of the comparison subsamples differ significantly, the inference is made that the two versions represent two different items, not two versions of the same item.

• Analysis of non-response is almost always a required aspect of survey research and is discussed in the section on sampling. Planning ahead for analysis of non-response may cause the researcher to add items to the survey instrument.

  
  

• Data collection modes
  • Face-to-face interviewing is usually the most expensive but it can obtain the highest response rate because it maximizes engagement with the respondent and imposes the lowest burden on the respondent. For the same reasons, face-to-face interviews can support longer instruments. It is also the best data collection type for open-ended responses. In situations where the respondent is initially unknown, this may be the only feasible data collection type. Liabilities of this method include possibility of interviewer bias, respondent bias on sensitive items, slow data collection time, and cost. Face-to-face interviewing is often used when the research purpose requires in-depth exploration of opinions.

  • Mail surveys are usually among the least expensive modes, are often best for sensitive items, and there is no interviewer bias. However, mail surveys are a poor choice for open-ended items or complex survey designs as they place a high burden on the respondent. Other liabilities include lack of control over respondent interaction with unknown others while taking the survey and while being faster than face-to-face administration, mail surveys are slower than phone or web surveys. Mail surveys are often used for very short instruments.

  • Telephone interviewing has the advantage of speed of data collection while supporting longer instruments than mail surveys and supporting open-ended responses, though not as well as face-to-face interviewing. There is also moderate control over interviewer bias since interviewers follow predetermined protocols, and there is high control over respondent interaction with others during the interview. Telephone interviewing can be a good moderate-cost compromise between mail and face-to-face survey research.

  • Web surveys are often the least expensive to administer yet can be fast in terms of data collection, particularly since implementation time may be low. Web surveys can be administered to very large populations, even internationally. Also, they support complex survey designs with elaborate branching and skip patterns depending on the response. The central drawback to web surveys is the near-impossibility of drawing random samples since web access is known to be biased by income, race, gender, and age. Also, web surveys impose almost as high a burden on the respondent as do mail surveys, and thus are not optimal for long instruments or open-ended questions. Web surveys are frequently a good choice when an unscientific but quick “straw poll” is needed on some subject.

  
  

• Types of computer support
  • CAI is computer-assisted interviewing, in contrast to PAPI, which is paper-and-pencil interviewing. CAI is a generic term which may include the types below.

  • CATI is computer-assisted telephone interviewing, usually employed in conjunction with random-digit dialing (RDD) as a means of approximating random sampling.

  • CAPI is computer-assisted personal interviewing, allowing the face-to-face interviewer to speed the survey process by entering responses directly into an uploadable database using a hand-held computer. CAPI also supports more complex branching and skipping in the interview.

  • CASI is computer-assisted self-interviewing, where the respondent is directed through the interview protocol by survey software delivered locally on the respondent’s computer or via the web. CASI is particularly desirable for responses to sensitive items.

  • Audio-CASI is a variant where voice technology is used to prompt the respondent with items and receive responses by voice. Audio-CASI is particularly desirable for non-English-speaking populations or where literacy is low.

  
  

• Survey design considerations
  • Survey layout. For mail and web surveys, the survey layout itself needs to be attractive, easy-to-use, and non-intimidating. Some rules of thumb are:
    • Include plenty of white space (as opposed to dense single-spaced print)
    • Have response areas be consistent and easy to check off (checkboxes are clearer than blank lines)
    • Have response areas on the right since most respondents are right-handed
    • Use one or at most two colors to set of sections and items, avoiding distracting color schemes.
    • Avoid multiple fonts, italics (harder to read), , animations, blinking font,
    • Have clear separation of the question part of an item and the response part, by line spacing and/or by putting response categories in bold
    • In general, follow good print or web design principles

  • Survey order. Typically, after an introduction which discloses the sponsorship of a survey, a survey begins with non-threatening items which arouse interest. The first question should be clearly related to the announced purposes of the survey (not a background item, for instance). Some recommend the second question be open-ended, to allow the respondent to "get into" the subject. If the survey is mixed open-ended and structured items, some researchers prefer to put the open-ended questions first so as to avoid priming the respondent with the structured items.

    Non-threatening background information questions (ex., demographic information) should be posed early so these controls will be available if the respondent fatigues and does not answer all the later items. The survey then proceeds to attitude questions, often sequencing from general and less threatening items toward more specific and more sensitive items.

    Sensitive background items, particularly the income item, are usually put at the end. I However, the more toward the end of the survey an item is, the lower its response rate is apt to be.tems which measure the dependent variable(s) or other research-critical items should not be left to the end for this reason. Balancing the foregoing considerations, it is desirable to administer the survey with different forms with different orders so as to lessen the order/response bias.

  • Survey length. There is no correct length to a survey: they should be as long as needed within the constraint of respondent attention span. Surveys need to have a minimum of one item per variable referenced in the set of hypotheses the researcher is investigating and at least three items per variable are recommended, along with demographic items.

    Dividing instruments into sections enables a sense of progress and reduces survey fatigue, especially for mail and even web surveys. In general, face-to-face interviews can sustain attention for the longest period, such as an hour. Telephone surveys tend to be at the opposite end, sustaining attention for perhaps ten minutes or ten questions as a rule of thumb. In between are self-administered questionnaires by mail or on the web.

    There is, of course, a trade-off between survey length and item response rate, with the rate of missing values increasing as respondents tire in long instruments. Testing survey length for pilot respondents similar to the eventual real respondents is one of the important reasons to pretest, discussed below. Testing also gives the researcher empirical information about time-to-complete, and telling respondents the time required is thought to reduce subject frustration and increase response rate.

  • Item length. The longer the item, the more the chance respondents will misunderstand. As a rule of thumb, items should target a length of 25 words or less. Items with such terms as and, or, except, with, and but may be a sign of overcomplexity and excessive length.

    
    

• Types of items
  • Model items are those that measure variables in the survey model.

  • Control items measure variables which, while not in the model, are to be used by the researcher in multivariate analysis to determine if model relationships are controlled, modified, or rendered spurious by virtue of other variables (control variables). Often control items are demographic items such as age, race, or gender.

  • Filter items are ones which allow the elimination of unqualified respondents during post-processing (ex., respondents who lack minimum information to respond intelligently; respondents who state they felt a lack of information or competence to complete the survey).

  • Cross-check items are internal check items which test consistency with other responses. Asking age at one point and date of birth at another would be a pair of cross-checked items. An extension of this is the split-form interview, which might be administered to a both partners in a marriage with a view toward cross-checking for consistency.

  • Random probe items of an open-ended nature are randomly interspersed by item and by survey form such that any one respondent is only probed on a few items. Probe items are "Could you tell me a little more about what you mean?" or "Could you give me an example of what you mean?" Probe responses are used to better understand the meaning of frequency distributions of responses to the corresponding structured items.
    • Instrumentation probe items are probe items concerning the respondent's processing of the survey instrument. For instance, in follow-up to a mail survey, the researcher may ask if the respondent remembers receiving the survey package, and responses to this item indicate the proportion of surveys discarded as junk mail without reading. The respondent may also be asked who opened the mail, or asked an open-ended question about what they thought about the survey package when they saw it in the mail, and what they did with it first. Other questions of this type are whether the respondent was away when the survey arrived, if the survey was misplaced, if the respondent forgot to mail the survey back, if there were language problems, if the respondent has had bad experiences with surveys in the past, if the respondent felt information would be kept confidential, and so on. Evaluative items may also be asked, such as wehterh the survey seemed too difficult, too long, too uninteresting.

  • Priming items. One type of survey research investigates differences between unprepared responses to items and responses where the respondent has been "primed" by prior information or items embedded in the instrument to provide a certain context. For instance, in a study on perceptions of crime in neighborhoods, responses may be studied with and without priming information presenting factual data on actual community crime patterns.

  
  

• Item bias. Bias may creep into survey items in a variety of ways.
  • Ambiguity. Questions should be specific, avoiding generalities. Ex.: "On a scale from 1 to 10, how popular is President Clinton at this time?" This example begs the question, "popular with whom?" Some respondents will answer the item in terms of popularity with themselves, others will answer in terms of popularity with their peers, and yet others will respond in terms of their recollection of past opinion polls. Items often need to be anchored in one or more of the familiar journalistic dimensions: who, what, when, where, why, and how? If it is possible for respondents to interpret questions in dissimilar terms, they will. "Have you participated in a political rally?" is also ambiguous, for instance, because participation is undefined and some will answer affirmatively due to passing within earshot of a political speaker, while others will answer negatively because, while they attended rallies, they were not the organizers. As a third example, "Do you favor government policy toward crime control?" leaves ambiguous the level of government (local, federal, state) as well as just what dimension of law enforcement is at issue. As a fourth example, when value preferences are requested by ranking "above average," "average," "below average," etc., ambiguity is introduced unless the question is answered, average in comparison with what? In general all pronouns, nouns, and noun-phrases should have a clear referent.

  • Mutual exclusivity. When multiple response items are allowed, bias is introduced if the items are not mutually exclusive yet only a single item may be selected.

  • Non-exhaustive response set. Bias is introduced when the response alternatives available to the respondent leave out valid choices they would otherwise make. The most common example is leaving out such responses as "neutral" or "don't know" when, in fact, respondents may well be neutral or may actually not know, rather than be hiding their "true" responses which the researcher is trying to force out by omitting these categories. Ambiguous terms, such as "parents," can also lead to non-exhaustive response sets, as by omitting the distinction between birth parents and adoptive parents. Likewise, "single" may omit the distinction between "never married," "widowed," and "divorced." This is discussed further in the FAQ section.

  • Residual categories. "Don't know," "Don't care," "Not applicable," and "Did not respond" are separate types of responses. It is better for the instrument to keep them separate so they can be analyzed for patterns, even if they are combined for other purposes in statistical processing. The question of omitting such responses so as to force substantive choices is discussed further in the FAQ section.

  • Rank lists. Ranking can be a challenging task. Many survey researchers recommend that respondents not be asked to rank more than four or five items. Beyond that, respondents may give arbitrary rankings just to get past the item.

  • Social favorability. Certain topics deal with actions or beliefs for which there is a socially favored position (people are supposed to enjoy parenting, to vote, not to approve of marijuana use, etc.). Such items should be buffered with phrasing such as "Some people favor x, some people favor y, which do you favor" or "There are many reasons why someone might do x instead of y, or do y instead of x. Did you happen to do x or y the last time the opportunity for either came up?"

  • Loaded terms . Ex. "Do you lean more toward the pro-life or toward the pro-abortion position on issue of termination of late-term pregnancies where the health of the mother is threatened?" This example is biased because one position is labeled with its most favorable label (pro-life, rather than anti-abortion), while the other position is labeled with its less favorable label (pro-abortion, rather than pro-choice). As another example, more people will "not allow" something than will "forbid" the same thing. Implicit endorsements is another form of loading to be avoided. "Doctors recommend drinking eight glasses of water a day. What do you think is the correct amount for you?" is an example with an endorsement by a prestigious class.

  • Leading questions. Ex., "Do you favor an increase in the federal minimum wage to $8.00?" This example is not as leading as an item of the type, "You favor X, don't you?" but it is still leading because it does not legitimize both affirmative and negative responses. A negative response may appear stingy or mean-spirited toward the poor, and this may bias the survey toward too many affirmative responses. A better method is to phrase such policy options in the form, "Some people favor X, while other people are opposed to X. What is your opinion?"

  • Unfamiliar terms and jargon. Ex., "Do you consider environmental regulation of wetlands to be an invasion of the sovereignty of the states?" Terms such as "sovereignty" are apt to not be well understood by typical survey populations. When a term not in popular usage must be used in an item, such as an item asking opinions about the Civil Rights Act of 1964, the interviewer must precede the item with a brief explanation. Wherever possible, familiar terms should be substituted for unfamiliar terms.

  • Requiring inaccessible information. An item may use familiar terms but require information most respondents would not know. For instance, a question about Stalin's treatment of the Cossacks might have been acceptable long ago, but today's population of respondents is apt to know little about this subject and perhaps not even recognize "Stalin." As a second example, the item "Is your family income above, about equal to, or below the official poverty rate for a family of four?" is flawed, among other reasons, because people are not apt to know what dollar amount is the referent today, making the item unacceptable for a determination of fact (although possibly acceptable for a determination of self-perception).

  • Multidimensionality. A form of ambiguity arises when items are multidimensional. Ex.: "On a scale of 1 to 10, please rank the performance of the president?" The respondent may be torn between multiple dimensions: personal vs. official performance, or domestic vs. foreign policy performance, for instance. Example two: "Do you believe President Clinton lied under oath in the Monica Lewinsky scandal, impairing his effectiveness as president?" This item calls for two opinions, not one: one on lying under oath and one on subsequent effectiveness. It should be broken into two separate items.

  • Compound items. Items with compound clauses may not be multidimensional but may involve undue complexity (see below). For instance, the item, "Do you have or have you every had a physical, mental, or other health condition which has lasted over six months and which has limited the kind of work you could do on your job?" is better broken into two items: "Do you have or have you every had a physical, mental, or other health condition which has lasted over six months?" and the follow-up item, "If you answered yes to the previous question, did this condition limit the kind of work you could do on your job?"

  • Recall items. People's ability to recall the past is limited. The more current and specific the question reference, the better. If recall is necessary, the time frame should be as recent as possible and not over six months unless the reference is to major events (ex., marriage, changing jobs, buying a car).

  • Complexity and memory overload. It is possible to overtax the respondent by requiring an excessive memory burden due to complexity. The more complex the item, the easier it is to overload memory. The explanation should not demand the respondent learn complex new material. Simply using double negatives in an item may introduce undue complexity. If there are over five alternatives, a show card should be used to allow the respondent to view the alternatives, not simply hear them orally in an interview situation. Also, do not start questions with the response categories (Not "Do you strongly disagree, disagree, ....") as this is a more complex phrasing which undermines respondents' ability to grasp the central concept of the item. In some cases it may be appropriate to encourage respondents to check personal records on recall items. In other cases it may be appropriate to provide examples of how to fill in a complex item.

  • Poor grammatical format. Weak grammatical format can introduce bias. For instance, the item, "Would you say that you approve very strongly, strongly, ..." presents "dangling alternatives" which the respondent must memorize before even knowing what the question is. This format is frustrating and may bias responses toward the first-presented response, or toward negativity. Putting the subject first is the preferred order.

  • Hypothetical items. Hypothetical items (ex., "What would you do if ...") creates a difficult challenge for respondents. Seriously considering such items requires time for imagination and consideration. People tend to base responses to such items on their most-related actual experiences, and it may be better to ask about such experiences directly.

  • Inappropriate assumptions. Items should not contain false or arguable premises. The respondent should not be required to make a false or arguable assumption in order to respond to an item on its face. For instance, "How much improvement in your life would passage of the Equal Rights Amendment make? A great deal, some, very little, or none?" is an item which assumes that the effect of ERA passage could not be negative on a respondent, forcing respondents who believe the effect would be negative to either skip the item or give a response which does not represent their views.

  • Gender bias. Avoid gender-specific wording in items. Not only may some respondents take offense, but also some respondents will interpret a reference to "man,"" for example, to refer to humans while others will interpret the reference to refer to males, thereby creating an unwanted response ambiguity.

  • Mismatched item and response set. The response categories should be appropriate to the dimension probed by the item. For instance, "How unusual do you think it is for a person to donate $100 or more to a presidential candidate? A great deal, some, very little, or none?" is a mismatch because "a great deal unusual," "some unusual," etc., are not grammatically acceptable responses and will confuse respondents.

  • Language differences. One must, of course, assure that items have the same meaning when the questionnaire is to be administered to populations speaking different languages. This is addressed by Behling and Law (2000). Simple direct translation of validated U. S. scale items into another language often will not create a valid scale in the other language. One must consider three dimensions of the problem: semantic equivalence, conceptual equivalence, and normative equivalence of items. An item may be acceptable in one dimension but not in another. Semantic equivalence is sought through the translation/back translation method, having independent translators translate from one language to another, and then back again, to see if the original and re-translated item remain the same. Conceptual equivalence may be sought by such methods as factor analysis, demonstrating that items in one language version have the same factor structure (load similarly on underlying factors) as in another language version. Behling and Law recommend caution in this and other statistical tests, however, urging that theory be used to validate each language version independently. Normative equivalence requires such strategies as developing close relations with respondents, using trusted agents for interviewing, providing assurance of anonymity or confidentiality, and pilot testing extensively. Behling and Law give numerous additional practical suggestions and insights into the translation problem and this text is strongly recommended for those who will face translation issues.

  
  
• Research on web surveys
  • Trends. There is a trend toward web-based and other self-administered forms of survey research due to the steadily increasing cost of interviewer-administered data collection, including the increased time interviewers need to locate, screen, persuade, and interview respondents. See Couper, 2005.

  • Self-administration, as in web surveys, CASI, and audio-CASI, improves the reporting of socially sensitive information relative to interviewer administration.

  • Response rates. The early claims that CATI and CAPI would result in improvements in response rates, or at least stem the decline, have proved unfounded. But fortunately, so too have the fears that the introduction of these technologies would hasten the decline in willingness to participate in surveys. While CASI has had demonstrable effects on reducing measurement error, its impact on nonresponse has been negligible. See Couper, 2005.

  • There is a strong trend toward opt-in surveys, often conducted via the Internet, and these are not probability samples. While producing large and diverse samples, even after proportion-adjusting web sample results, only some surveys approximate results of RDD or other random surveys. Others result in substantial differences from probability samples, even after statistical adjustment. See Couper, 2005.

  • More error and non-response in web surveys. A paper version of a survey obtained higher and more accurate response rates than did a computer version (Beebe, Harrison, Park, McRae, & Evans, 2006). Long web surveys and those of low salience to the respondent have greater non-response, but offering personalized feedback was effective in compensating for these problems. Offering a lottery incentive was only marginally helpful. (Marcus, Bosnjak, Lindner, Pilischenko, & Schütz, 2007).

  • No mode effect. The same individuals responding to similar web- and paper-based instruments do not give significantly different responses - there is little evidence of a mode effect linked to web-based questionnaires (Denscombe, 2006).

  • Coverage bias. The web survey studied exhibited significant coverage bias which was not correctable by usual statistical post-survey adjustments such as ratio-raking (rescaling the responses for known proportions in the true population) (Lee, 2006).

  • Age bias. Web surveys appeal to and may be biased toward a younger and different group. For instance, iIn an Australian study, most respondents given a choice preferred paper-based to web-based questionnaires. Web respondents were more likely to be young, male, middle status, and working in information technology related sectors (Diment & Garrett-Jones, 2007).

  • Lotteries are not much incentive. In a four-wave web survey, there was a direct positive effect of the lottery on response only at the first wave of the study. The lottery did not have any effect on retention (Göritz & Wolff, 2007). Cash lotteries relative to no incentives did not reliably increase response or retention; neither did it make a significant difference if one large prize or multiple smaller prizes were raffled (Göritz , 2006).

  • Radio buttons are better than drop-down pick lists. Format choice between radio button vs. drop down multiple choices did not significantly affect survey completions, number of nonsubstantial answers, or time to complete the survey. However, there was evidence that drop downs were associated with higher item nonresponse and longer response times. Furthermore, the 76% of respondents using Scroll Mice to complete the survey were prone to accidentally changing answers when presented with drop down formatted questions (Healey, 2007)

  
  

Assumptions

• Attitude is not action. Survey researchers often assume that attitudes lead to corresponding actions, but this is often not true, as discussed below.

Frequently Asked Questions

• How close a correspondence is there between people's opinion survey responses and their actual actions? Is there an attitude-action gap?
  In some areas, such as voting, there appears to be a close correspondence between how respondents say they will vote and how they actually do vote subsequently. In other areas, such as race-related attitudes, the discrepancy between what people say and what they do appears to be substantial. Tartar (1969) found 65% of respondents changed their behavior away from surveyed attitudes when confronted with actual situations requiring behavioral response. An early survey by Ehrlich (1969: 34) concluded, "Studies on the relation of attitudes and behavior have almost consistently resulted in the conclusion that attitudes are a poor predictor of behavior." Ehrlich gave seven reasons for this gap:
  1. There may be no clear way for an attitude to become expressed in a behavior.
  2. Attitudes may reflect fantasy or sublimation more than realistic responses.
  3. Individuals may not be willing to disclose honest expression of certain opinions.
  4. Individuals may translate opinions into action in unpredictable ways, such as suppression and inaction, displacement into other arenas, and rationalization.
  5. Individuals may lack the informational resources to act.
  6. Individuals may lack the opportunity to act.
  7. Individuals may lack the sense of efficacy to act.

• What are the pros and cons of using open-ended items?
  Open-ended items are frequently used as complements rather than alternatives to structured items. As a follow-up to a structured item, an open-ended item can bring to light unanticipated interpretations and variations in the meaning of responses. For instance, in a ranking item about job satisfaction, a structured response may show "income" to be the most important satisfaction factor, but a follow-up open-ended item may show some respondents mean "high income," whereas other respondents mean "steady income." Open-ended items are not necessarily more accurate, however. An open-ended question may elicit one range of preferences, yet when other choices are presented in a structured question, respondents may well alter their open-ended preferences to re-rank in the light of previously-unconsidered alternatives in the structured item. Ideally, one would use both, with an open-ended probe question after every structured item. This, however, would be prohibitive in time and fatigue costs of the interview, which is why random probe questions are used, as discussed above.

• Is it okay to force a choice, or is a 'no opinion' option necessary?
  It is very tempting to leave out 'no opinion,' 'not sure,' and 'don't care' options in structured questions. Forcing a choice not only yields the direction of preference, but also requires the respondent to give a more considered response. However, forcing a choice does distort the responses of individuals who genuinely have no opinion, are genuinely unsure, or who genuinely do not care. When the researcher can assume everyone really has an opinion, then forced choice is justified. If not, the researcher should retain the no preference options, or should consider filter and intensity questions. Research on the survey process supports the expectation that having a "don't know" category, while encouraging such responses, does not usually change the relative proportions of respondents in the other categories of the items.

  Filter questions precede an item and are of the type, "Here is a statement about X. Not everyone has an opinion on this. Here is the statement: ... Do you have an opinion on that?" If the respondent answers 'Yes' to the filter question, then a forced-choice item is presented. Intensity items follow forced choice items and ask "How strongly do you feel about that -- extremely strongly, very strongly, ...?" A separate intensity item is needed because extremity on the forced choice item is not equivalent to intensity: one may hold an extreme position non-intensely and may intensely hold a middle position. In using forced choice rather than offering a 'no opinion' option, the researcher is assuming that those who would have given 'no opinion' responses will be low intensity on the intensity item, but not necessarily in the middle on the forced choice item itself. Filter items also avoid the use of "no opinion," "not sure," and "don't care," all of which have a tinge of social favorability bias (we are supposed to have an opinion, be sure, and care). If a middle choice is offered, "neither agree nor disagree" is a better item than "neutral" because it covers divided opinions, whereas "neutral" implies an overt decision to avoid sides.

• How can I get a more valid response to highly sensitive questions?
  This issue is treated in depth by Fox and Tracy (1986). The randomized response strategy takes subjects "off the spot" by mixing randomized responses with real ones. The basic randomized response method: Consider the question "Have you ever used an marijuana?" Respondents can be told to flip a coin or use some other randomizing device, then if heads appear (not seen by the interviewer) respondents are told to answer the question honestly but to answer "Yes" regardless if tails appear. If the result for a survey is 60% "Yes," we can reason that the true response rate is 20% (half of true "Yes" responses will also be part of the 50% expected tails-determined "Yes" responses and half not, so the true percentage is twice the excess of the observed response [60%] less the random response [50%]). Fox and Tracy extend this to polytomous and quantitative responses and discuss various refinements as well as alternative approaches which have proved problematic.

• How do I handle the fact that not all respondents answer all items in my questionnaire?
  Andrew Gelman, Gary King, and Chuanhai Liu (1998) present a method of analyzing a series of independent cross-sectional surveys in which some questions are not answered in some surveys and some respondents do not answer some of the questions posed. The method is also applicable to a single survey in which different questions are asked, or different sampling methods used, in different strata or clusters. The method involves multiply-imputing the missing items and questions by adding to existing methods of imputation designed for single surveys a hierarchical regression model that allows covariates at the individual and survey levels. Information from survey weights is exploited by including in the analysis the variables on which the weights were based, and then reweighting individual responses (observed and imputed) to estimate population quantities. They also develop diagnostics for checking the fit of the imputation model based on comparing imputed to non-imputed data. They illustrate with the example that motivated this project --- a study of pre-election public opinion polls, in which not all the questions of interest are asked in all the surveys, so that it is infeasible to impute each survey separately. See Gelman, King, and Liu (1998).

• Where can I find out more about computer software for survey research?
  A standard but somewhat dated reference is Willem E. Saris, Computer-Assisted Interviewing, Quantitative Applications in the Social Sciences Series No. 80, Thousand Oaks, CA: Sage Publications, 1991. Saris's still-useful volume treats the pros and cons of computer-assisted survey research, and lists 18 software packages. See also Edith de Leeuw and William Nicholls II (1996) 'Technological Innovations in Data Collection: Acceptance, Data Quality and Costs' Sociological Research Online, 1(4), available online. A treatment dealing with specialized uses is "Computers, Survey Research, and Focus Groups in Public Administration Research and Practice," by Michael S. Vasu and Ellen Storey Vasu, pp. 196-219 in D. Garson, ed., Information Technology and Computer Applications in Public Administration, Hershey, PA: Idea Group Publishing, 1998. See also Computer Assisted Survey Information Collection by Mick Couper, based on the International Conference on Computer Assisted Survey Research, Survey Methodology Section, and published by John Wiley & Sons, 1998.

  The number of software options for survey research has become vast. Moreover, this is a moving target with new packages appearing with great regularity and existing software upgraded frequently. In this section a few illustrative resources are mentioned.

  • SPSS. SPSS, Inc., offers a wide range of survey research products and services. SPSS Data Entry[10] supports creation of surveys and forms using the “SPSS Data Entry Builder” module, and data entry using the “SPSS Data Entry Station” module. SPSS SmartViewer Web Server software may be used to distribute survey results online. More advanced survey research is supported by SPSS Dimensions, which is a web-enabled survey system for creation and data collection of surveys by web, phone, scan forms, or mobile devices, along with survey coding and analysis. Central data storage in Dimensions integrates with most popular databases and operating systems. Dimensions features include support strategies for privacy protection, increasing response rates, and creation of web-based interactive reports on survey results. SPSS Complex Samples is the SPSS module for stratified, clustered, multistage, and other complex sampling. SPSS also offers online survey hosting and online survey research services. .

  • SAS. SAS, Inc., while generally more comprehensive than SPSS, has no single produce oriented specifically to survey research. A book by Gravely (1998) discusses how Proc Tabulate and other modules in the SAS system can be mobilized to support survey research. Proc Surveyselect is for sampling (simple, stratified, clustered, multistage, other), with output being a data set containing the selected units, with selection probabilities and sampling weights. A variety of survey-related statistical procedures are available, including Proc Surveymeans, Proc Surveyreg (regression), Proc Surveylogistic, and Proc Surveyfreq.

  • Stata. Stata, Inc., software is a comprehensive statistics package which, in the survey research area, supports statistical analysis of sampling weights, multistage designs; stratified sampling; cluster sampling; complex survey designs; poststratification analysis of non-response; and a wide variety of statistical procedures. Indeed, the Stata Survey Data Reference Manual (Stata, 2005) is a valuable reference regardless of which software is used by the researcher. Stata Inc. at this writing does not have specialized modules for survey creation, survey administration, or survey data entry.

  • Survey Monkey is an online survey service whose free basic service includes considerable functionality and whose advanced options (ex., item skipping, unlimited responses) are available at modest price. The website is http://www.surveymonkey.com/. Survey Monkey has been widely used for instruction in survey methods, for dissertation research, and more.

  • SDA (Survey Documentation and Analysis). SDA is software from the Computer-Assisted Survey Methods program at the University of California, Berkeley.[13] SDA consists of a set of programs for the documentation and web-based analysis of survey data. The “Codebooks” module produces survey codebooks suitable for printing or web use, with data definitions for SAS, SPSS, Stata and DDI (XML) formats. The “Analysis” module provides basic, web-implemented statistical procedures for survey data. The “Subsetting” module has procedures for creating customized subsets of datasets. The website gives access to the demonstration SDA Archive, related documentation, and several datasets. The separate CASES software is designed for collecting survey data based on structured questionnaires, using telephone or face-to-face interviewing as well as self-administered procedures. CASES claims to be one of the most widely used packages for the purpose. CASES and SDA are not purchased but instead obtained by joining the Association for Computer-assisted Surveys (ACS), which requires a minimum one-year membership agreement and fee.

  • SUDAAN. SUDAAN is survey analysis software from the Research Triangle Institute, Research Triangle Park, NC. It consists of nine analytic procedures to adjust survey data analysis for stratified, clustered, and other complex samples, including repeated measures designs. It claims to be the only broadly-applicable software for analysis of correlated and weighted survey data.

  • IMPS and CSPro. IMPS (The Integrated Microcomputer Processing System) is freely downloadable software from the U. S. Census, designed for the major tasks in survey and census data processing: data entry, data editing, tabulation, data dissemination, statistical analysis and data capture control. CSPro ((the Census and Survey Processing System) is also available, designed for entering, editing, tabulating, and disseminating data from censuses and surveys. CSPro combines the features of the Integrated Microcomputer Processing System (IMPS) and the Integrated System for Survey Analysis (ISSA), and is intended to replace both. While it can process Census data, CSPro also can create user-defined data entry forms (screens) for data capture.

  • The Section on Survey Research Methods, American Statistical Association hosts an online listing of a large number of software packages for survey research.

  • Shareware, freeware, and other low-cost survey software options can be found by entering “survey” in the search box of http://www.download.com or http:/www.tucows.com, two of the most popular software download sites. There are a large number of such software packages. At this writing, Survey Power was listed as the most-downloaded of these options.

Bibliography

• Arksey, Hilary and Peter T. Knight (1999). Interviewing for social scientists: An introductory resource with examples. Thousand Oaks, CA: Sage Publications. Covers design, triangulation, human subjects protection, report writing, and other basic topics.
• Beebe, Timothy J.; Harrison, Patricia A.; Park, Eunkyung ; McRae, James A. Jr.; & Evans, James (2006). The effects of data collection mode and disclosure on adolescent reporting of health behavior. Social Science Computer Review 24(4), 476-488.
• Behling, Orlando and Kenneth S. Law (2000). Translating questionnaires and other research instruments: Problems and solutions. Thousand Oaks, CA: Sage Publications. Series: Quantitative Applications in the Social Sciences No. 133.
• Bourque, Linda B. and Virginia A. Clark (1992). Processing data: The survey example. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences Series No. 85. A non-technical work which covers designing forms, pretesting, training data collectors, data entry, and data preparation, including handling nonresponse, missing data, and outliers.
• Bourque, Linda B. and Eve P. Fielder (2002a). How to conduct self-administered and mail surveys, Vol. 3. Thousand Oaks, CA: Sage Publications. Covers pretesting, pilot testing, cover letters, tracking non-respondents, and other mechanics.
• Bourque, Linda B. and Eve P. Fielder (2002b). How to conduct telephone surveys, Vol. 4. Thousand Oaks, CA: Sage Publications. Covers types of phone surveys, instrument design, and other mechanics of telephone surveys.
• Converse, Jean M. and Stanley Presser (1986). Survey questions: Handcrafting the standardized questionnaire. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences Series No. 63.
• Couper, Mick P. (2005). Technology trends in survey data collection. Social Science Computer Review 24(3), 486-501.
• Czaja, Ron and Johnny Blair (2005). Designing surveys: A guide to decisions and procedures. Second ed.. Thousand Oaks, CA: Sage Publications. Covers item construction, survey design, testing, sampling, and reducing sources of error.
• Denscombe, Martyn (2006). Web-based questionnaires and the mode effect: An evaluation based on completion rates and data contents of near-identical questionnaires delivered in different modes. Social Science Computer Review 24(2), 246-254.
• Dillman, Don A. (1999). Mail and Internet surveys: The tailored method, Second edition. NY: John Wiley and Sons. Probably the most-used graduate guide to survey research, covers use of survey waves in analysis of nonresponse.
• Diment, Kieren & Garrett-Jones, Sam (2007). How demographic characteristics affect mode preference in a postal/web mixed-mode survey of Australian researchers. Social Science Computer Review 25(3),
• Ehrlich, Howard (1969). Attitudes, behavior, and the intervening variables. American Sociologist 4(1): 29-34. Cited in relation to the gap between attitudes and behavior.
• Fink, Arlene (2002a). The Survey Handbook, Vol. 1. Thousand Oaks, CA: Sage Publications. Covers qualitative surveys and survey ethics.
• Fink, Arlene (2002b). How to Ask Survey Questions. Thousand Oaks, CA: Sage Publications. Covers focus group questions, factorial questions, and conjoint
• Fink, Arlene (2002c). How to design survey studies, Vol. 6. Thousand Oaks, CA: Sage Publications. Covers validity issues in survey research as well as cross-sectional, cohort, and case-control common designs.
• Fink, Arlene (2005). How to conduct surveys, Third ed.. Thousand Oaks, CA: Sage Publications. Covers survey-based report-writing.
• Fink, Arlene and Jacqueline Kosecoff (2005). How to conduct surveys: A step-by-step guide, Third ed.. Thousand Oaks, CA: Sage Publications.
• Fowler, Floyd J., Jr. (2001). Survey research methods, Third edition. Thousand Oaks, CA: Sage Publications. Best-selling introductory text.
• Fox, James Alan and Paul E. Tracy (1986). Randomized response: A method for sensitive surveys. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences Series No. 58. Covers randomized response methods to encourage response to sensitive items.
• Gelman, Andrew, Gary King, and Chuanhai Liu (1998). Not asked and not answered: Multiple imputation for multiple surveys. Journal of the American Statistical Association, March, 1998. You can retrieve the paper from File Transfer Protocol (FTP) to ftp wizard.ucr.edu, then login as user: anonymous, password: , and cd to pub/polmeth/working_papers97, and retrieve gelma97c.zip.
• Göritz , Anja S. (2006). Cash lotteries as incentives in online panels. Social Science Computer Review 24(4), 445-459.
• Göritz, Anja S. & Wolff, Hans-Georg (2007). Lotteries as incentives in longitudinal web studies. Social Science Computer Review 25(1), 99-110.
• Healey, Benjamin (2007). Drop downs and scroll mice: The effect of response option format and input mechanism employed on data quality in web surveys. Social Science Computer Review 25(1), 111-128.
• Hess, Jennifer, Singer, Eleanor, & Groves, Robert M. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly 56(4): 475-495.
• LaPiere, Richard T. (1934). Attitudes vs. actions. Social Forces, Vol. 13 (December): 230-37. This is the classic article challenging survey research and empirically showing its limitations.
• Lee, Sunghee (2006). An evaluation of nonresponse and coverage errors in a pre-recruited probability web panel survey. Social Science Computer Review 24(4), 260-275.
• Marcus, Bernd; Bosnjak, Michael; Lindner, Steffen; Pilischenko, Stanislav; & Schütz, Astrid (2007). Compensating for low topic interest and long surveys: A field experiment on nonresponse in web surveys. Social Science Computer Review 25(3)
• Nesbary, Dale (1999). Survey research and the World Wide Web. Boston: Allyn and Bacon.
• Oishi, Sabine Mertens (2002). How to conduct in-person interviews for surveys, Vol. 5. Thousand Oaks, CA: Sage Publications. Covers interview scripting, transition statements, interviewer job descriptions, interviewer training, handling of response errors, data cleaning.
• Ostini, Remo and Ronald N. Forthofer (2006). Analyzing complex Survey data, Second ed.. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences, Vol. 71. Covers use of logistic regression.
• Peterson, Robert A. (2000). Constructing effective questionnaires. Thousand Oaks, CA: Sage Publications.
• Rea, Louis A., Richard A. Parker (Contributor), and Alan Shrader, ed. (1997). . Designing and conducting survey research: A comprehensive guide (Jossey-Bass Public Administration Series). San Francisco: Jessey-Bass.
• Rubin, Herbert J. and Irene S. Rubin (2004). Qualitative interviewing: The art of hearing data. Thousand Oaks, CA: Sage Publications. Covers open-ended in-depth questionnaires, develops the concept of "conversational partnerships," follow-up questions, transcript preparation, and writing up results of qualitative interviews.
• Salant, Priscilla, with Don A Dillman (1994). How to conduct your own survey. NY: John Wiley and Sons.
• Tartar, Donald (1969). Toward prediction of attitude-action discrepancy. Social Forces 47 (4): 398-404. Cited in relation to the gap between attitudes and behavior.
• Tourangeau, Roger, Lance J. Rips, and Kenneth A. Rasinski (2000). The psychology of survey response.. NY: Cambridge University Press.
• U. S. General Accounting Office (1991). Using structured interviewing techniques. PEMD-10.1.5, 7/91. Available online in .pdf format.
• Willis, Gordon B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications. Covers verbal probe questions, determining if probe questions are working, testing and trouble-shooting cognitive probe items.
• Winstein, Raymond M. (1982). The mental hospital from the patient's point of view. In Walter R. Gove, ed., Deviance and mental illness, Thousand Oaks, CA: Sage Publications. This famous article shows the pitfalls of participant observation research (ex., Goffman's and Rosenhan's classic studies) and demonstrates how systematic survey research captures the true experience of mental hospitilization.