Overview Research designs fall into two broad classes: quasi-experimental and experimental. Experimental studies are characterized by the ability to randomize subjects into treatment and control groups. This randomization goes a long way toward controlling for variables which are not included explicitly in the study. Because comparison groups are not true, randomized control groups in quasi-experimental studies, this type of study has to control for confounding variables explicitly through statistical techniques. For this reason, quasi-experimental studies are sometimes labeled correlational designs. As discussed below, not much can be concluded from one-point-in-time studies of a single group. It is much better to have a comparison (control) group, and better yet to have measurements before (pretest) and after (posttest) the treatment (the change in the causal variable or variables), and best of all to have multiple pretests and posttests. Any given method of data collection, such as survey research, could be used in an experimental or a quasi-experimental design, depending on whether there was a treatment and a control group, and if the participants were randomly assigned. Similarly, analysis of variance (ANOVA) studies may be experimental or quasi-experimental (but originated in the former). More on research design may be found in the sections on ANOVA, ANCOVA, MANOVA, and MANCOVA. In practical terms, however, some methods of data collection, such as case studies, are used in non-experimental designs.

Contents Key concepts and terms Experimental designs Quasi-experimental designs Non-experimental designss Assumptions Frequently asked questions Bibliography




Key Concepts and Terms

• Experimental Designs

  A design is experimental if subjects are randomly assigned to treatment groups and to control (comparison) groups. Cook and Campbell (1979) mention ten types of experimental design. Note that the control group may receive no treatment, or it may be a group receiving a standard treatment (ex., students receiving computer-supported classes versus those receiving conventional instruction). That is, the control group is not necessarily one to be labeled "no treatment."
  1. Classic experimental designs: randomization of subjects into control and treatment groups is a classic experimental method, amenable to a variety of ANOVA designs discussed separately. The two broad classes of classic experimental design are:
     • Between subjects designs:. In this type of design, the researcher is comparing between subjects who experience different treatments. There are different subjects for each level of the independent variable(s) (ex., for each different type of media exposure in a study of the effect of political advertising). Any given subject is exposed to only one level and comparisons are made between subjects' reactions or effects. The researcher relies on randomization of subjects among the treatment groups to control for unmeasured variables, though sometimes stratification of subjects is employed to guarantee proportions on certain key variables (ex., race).
       • Factorial designs use categorical independent variables to establish groups. For instance in a two factor design, the independent variables might be information type (fiction, non-fiction) and media type (television, print, Internet ), generating 2 times 3 = 6 categories. An equal number of subjects would be assigned randomly to each of the six possible groups (ex., to the fiction-television group). One might then measure subjects on information retention. A null outcome would be indicated by the average retention score being the same for all six groups of the factorial design. Unequal mean retention scores would indicate a main effect of information type or media type, and/or an interaction effect of both.
         • Fully-crossed vs. incomplete factorial designs. A design is fully crossed if there is a study group for every possible combination of factors (independent variables). An incomplete factorial design, leaving out some of the groups, may be preferred if some combinations of values of factors are non-sensical or of no theoretical interest. Also, when one of the factors is treatment vs. control (no treatment) and another factor is types/levels of treatment, the control subjects by definition will not receive types/levels of treatment so those cells in the factorial design remain empty.

       • Randomized block designs stratify the subjects and for each strata, a factorial design is run. This is typically done when the researcher is aware of nuisance factors that need to be controlled (example, there might be an air conditioned room stratum and a no air conditioning stratum) or if there were other mitigating structural factors known in advance (ex., strata might be different cities). That is, the blocking variables which stratify the sample are factors which are considered to be control variables, not independent variables as they would be in a simple factorial design. Randomized block designs seek to control for the effects of main factors and their interactions, controlling for the blocking variable(s).
         • In SPSS: Consider city to be the blocking variable and information type and media type to be the main factors. In a simple factorial design, city would be an additional factor and in SPSS one would ask for Analyze, General Linear Model, Univariate; the dependent variable would be retention score; city, information type, and media type would be fixed factors; the model would be "full factorial" (the default). In a randomized block design, one would ask for Analyze, General Linear Model, Univariate; the dependent variable would be retention score; information type, and media type would be fixed factors; the blocking variable, city, would be entered as a random factor; click Model and select Custom, then set "Build Term(s)" to "Main Effects" and move all three factors over to the "Model:" box; uncheck "Include Intercept in Model."; Continue; OK. Note that this procedure reflects the fact that in a randomized block design there are no interaction effects, just main effects. Later, for multiple comparisons, repeat this procedure but click the Post Hoc button and enter the main factors in the Post Hoc Tests box; also check the type of test wanted (ex., Tukey's HSD).

     • Within subjects (repeated measures) designs: In this type of design, the researcher is comparing measures for the same subjects (hence, "within subjects"). The same subjects are used for each level of the independent variable, as in before-after studies or panel studies. Since the subjects are the same for all levels of the independent variable(s), they are their own controls (that is, subject variables are controlled). However, there is greater danger to validity in the form of carryover effects due to exposure to earlier levels in the treatment sequence (ex., practice, fatigue, attention) and there is danger of attrition in the sample. Counterbalancing is a common strategy to address carryover effects: ex., half the subjects get treatment A first, then B, while the other half get B first, then A, so that the carryover effect washes out in the sense that it is counterbalanced in the overall sample. Keep in mind that counterbalancing does not remove all effects - for instance, if there is a practice effect in a test situation, with higher scores for the second-taken test, on the average both tests will score higher in the overall sample than they would otherwise, since for both tests half the sample had the benefit of a practice effect. Counterbalancing in this situation only seeks that both test scores are biased equally upward, not that bias in absolute scores is eliminated.

     • Matched pairs designs. Compared to between-subjects designs, within-subjects designs control for subject variables better but at the expense of greater threat to validity in the form of contamination from influences arising from subjects going from one experimental level (condition) to another. Another type of repeated measures design is matched pairs, where the repeated measurement is not of the same subjects but of very similar subjects matched to have like key attributes. While matched pairs designs avoid some types of invalidity of within subjects designs, such as the threat of subject fatigue across repeated tests, matched pairs designs control only for the matched attributes whereas same-subject within-subjects designs control for both explicit and unmeasured subject variables.

  2. Lottery designs: used when lotteries are expected, as in some communities' manner of assignment of students to magnet schools, this eliminates a major impediment to randomization in social science situations, where it is frequently considered unethical or even illegal, or contrary to community standards, to offer benefits (a public policy treatment) to some citizens but not to others.

  3. Mandated control designs: sometimes, as in the military, control levels are sufficiently high that random assignment to treatment and control conditions will be accepted as a matter of course.

  4. Waiting list designs: used when demand outstrips supply, which also legitimates giving treatment to some citizens but not others, since offering treatment to all is impossible due to limited supply (ex., studies of clients vs. a waiting-list group).

  5. Equivalent time series designs: used when treatment cannot be delivered simultaneously to all, also legitimating giving treatment to some citizens but, temporarily, not to others, as when all employees are to receive training, but in rotations, such that different types of training can be delivered to different groups.

  6. Spatial separation designs: when treatment groups are separated and have no experiment-relevant intercommunication, as when participative management is tried with a treatment group of new, randomly assigned employees in one location but not in another.

  7. Mandated change/unknown solution designs: when change is required but there is no one clear solution, random assignment of subjects can gain acceptance, as in random assignment of students to classes with alternative textbooks.

  8. Tie-breaking designs: in cases where receiving treatment is merit-based, as in some academic situations, those receiving tied scores on merit-based exams can be randomly assigned to a treatment and a control group.

  9. Indifference curve designs: the attractiveness of the treatment can sometimes be adjusted to a point where some people will be indifferent toward receiving or not receiving it, and thus may be randomly assigned to a treatment or a control group.

  10. New organizations designs: when a new organization is established, such as one providing job training, opportunities may well arise which permit random assignment of clients to a control and a treatment organization.

  
  

• Quasi-Experimental Designs

  • Nonequivalent Control Group Designs

    A design is quasi-experimental if subjects are not randomly assigned to groups but statistical controls are used instead. There may still be a control or comparison group. While subjects are not randomly assigned, they are either randomly selected (sampled) or are all the relevant cases. For instance, a random sample of cities with council–manager governments may be compared with a random sample of cities with mayor–council governments. Cook and Campbell (1979) outline 11 nonequivalent control group research designs. In each case, due to the nonequivalency of the comparison group, threats to validity are much more possible than in a randomized design and the researcher should consider checklist-style all the types of validity threats.
    1. One-Group Posttest-Only Design: Sometimes called the "one-shot case study," this design lacks a pretest baseline or a comparison group, making it impossible to come to valid conclusions about a treatment effect because only posttest information is available. The level of the dependent variable may be due to treatment, or may be due to any number of causes of invalidity such as history (other events coexisting with treatment), maturation (changes in subjects which would have occurred anyway), experimenter expectation (subjects seeking to provide responses known to be desired or simply reacting to the attention of being tested), or other biases discussed in the section on validity. If this design is used, information must be gathered on pretest conditions, if only through respondent recollections, which are often subjective and unreliable.

    2. Posttest-Only Design with Nonequivalent Comparison Groups Design: In this common social science design, it is also impossible to come to valid conclusions about treatment effect based solely on posttest information on two nonequivalent groups since effects may be due to treatment or to nonequivalencies between the groups. Strategies for improving validity center on trying to create equivalency between groups by random assignment of subjects or matched-pair assignment to groups. When such assignment is impossible, then attempts may be made to control statistically by measuring and using as covariates all variables thought to affect the dependent variable. Nonetheless, many of the same threats to validity exist as in one-group posttest-only designs: history (concurrent events affect the two groups differently), maturation (the two groups would have evolved differently anyway), testing (the two groups have different reactions to testing itself), regression to the mean (the two groups tend to revert to their respective means if starting from extreme levels), etc.

    3. Posttest-Only Design with Predicted Higher-Order Interactions: Sometimes the expectation of the treatment effect interacts with a third variable. Instead of the expectation that treatment group subjects will be higher on the dependent, one has the expectation that the subjects will be higher if in the upper half of third variable Y but lower (or not as high) if in the bottom half of Y. For instance, training may lead to greater productivity for high education employees but not for low education employees on the same tasks. The interaction creates two or more expectations compared to the simple one-expectation one-group posttest only design. Because there are more expectations, there is greater verification of the treatment effect. However, this design is still subject to possible challenges to validity due to such factors as history (subjects high in education had different experiences) -- it is just that the counter-argument has to be more complex to account for the interaction, and therefore may be somewhat less likely to be credible.

    4. One-Group Pretest-Posttest Design: This is a common but flawed design in social science. It is subject to such threats to validity as history (events intervening between pretest and posttest), maturation (changes in the subjects that would have occurred anyway), regression toward the mean (the tendency of extremes to revert toward averages), testing (the learning effect on the posttest of having taken the pretest), and most challenges discussed in the separate section on validity. Sometimes the pretest data is collected at the same time as the posttest data, as when the researcher asks for recollection data of the "before" state. This is know as a proxy pretest-posttest design and has additional validity problems since the pretest data are usually significantly less reliable.

    5. Two-Group Pretest-Posttest Design Using an Untreated Control Group (separate pretest-posttest samples design): If a comparison group which does not receive treatment is added to what otherwise would be a one-group pretest-posttest design, threats to validity are greatly reduced. This is the classic experimental design. Since the groups are not equivalent, there is still the possibility of selection (observed changes are due to selection of subjects, such as working with more motivated volunteers in a treatment group -- see two-stage least squares for a discussion of testing for selection bias). Much depends on the outcome. For instance, if the treatment group starts below the comparison group and ends up above after treatment, a stronger inference of a treatment effect exists than if both groups rise in performance, but the treatment group more so (this might well be due to selection). A strongly recommended modification to this design is to have more than one pre-test. Multiple pretests (at the same interval as between the last pretest and the posttest) help establish the performance trends in both the treatment group and the control group, and treatment should be revealed by a change in the trend line for the treatment group but not the control group.

    6. Double pretest designs. One can strengthen pretest-posttest designs by having two (or more) pretest measures. This can establish if there is a trend in the data independent of the treatment effect measured by the posttest. By seeing if there is a posttest effect over and above the trend, one controls for maturation threats to study validity.

    7. Four-group Design with Pretest-Posttest and Posttest-Only Groups. Also known as the "Solomon four-group design," this design has a treatment and control group with both pretests and postests and has treatment and control groups with posttests only. This design strengthens the two-group pretest-posttest design because, if the same effect difference is found for treatment vs. control groups in the pretest-posttest set as for the posttest-only set, then the researcher may rule out threats to validity having to do with repeated measurement (ex., learning effects from having taken the test before).

    8. Nonequivalent Dependent Variables Pretest-Posttest Design: In this design, the researcher identifes dependent variables related to the treatment-related variable, but where treatment is predicted to have no effect. Then, if the variable thought to be affected by treatment does in fact change in the predicted direction, but there is no change in the other related dependent variables, again as predicted, then the inference is made that the change in question is due to treatment, not some confounding cause such as test experience from the pretest.

    9. Removed-Treatment Pretest-Posttest Design: In some situations it is possible not only to introduce a treatment but also to remove it. If the dependent variable goes up after treatment and then goes down when treatment is removed, this is some evidence for the effect of treatment. Of course, if the variable goes up after treatment, it might come down on its own anyway due to a declining return or attrition effect. Cook and Campbell (1979) therefore recommend at least two posttests after treatment and before removal of treatment, in order to establish trend effects after treatment. The researcher also needs to beware of resentment effects due to treatment removal, as these also might cause a decline in the variable measured, depending on the situation.

    10. Repeated-Treatment Design: This design is similar to the preceding one but follows a pretest-treatment-posttest-removal of treatment-posttest-restoration of treatment-posttest pattern. The expected treatment effect is for the dependent variable to increase after treatment, decline after removal of treatment, then increase again with restoration of treatment. Even if this outcome occurs, inference is not foolproof as the decline phase may be due to resentment at removal of treatment rather than direct adverse affects of removal of treatment, and the subsequent rise may be due not to restoration of treatment but removal of the source of resentment. Also, subjects may more easily become aware of experimenter expectations in this design, and may seek to meet (or react against) expectations, thereby contaminating the study.

    11. Switching Replications Designs. In this research design, there are two comparison groups and three measures. Both groups are measured under pretest conditions. The treatment is given to one group but not the control group, and a first post-test measure taken. Then the treatment is given to the control group but not the first group, and a second post-test measure is taken.

    12. Reversed-Treatment Pretest-Posttest Nonequivalent Comparison Groups Design. This design is one in which the nonequivalent comparison group receives the opposite treatment (ex., the treatment group receives participative leadership while the comparison group receives autocratic leadership). The expectation is that the posttest will show increase for the treatment group and decrease for the comparison group. Cook and Campbell (1979) suggest adding a no-treatment group and even a placebo group where appropriate. Multiple pretests will improve this design by showing preexisting trends in the treatment and nonequivalent comparison group.

    13. Cohort Designs with Cyclical Turnover: This design refers to the study of groups as they evolve over time, as in the study of a fourth-grade class in year 1, the corresponding fifth grade class in year two, etc. The expectation is that the class average will increase in the posttest after treatment. This design is liable to the same challenges to validity as simple prettest-posttest designs, but it can be strengthened by partitioning the cohort into subgroups according to their exposure to the treatment. In a study of the effects of television violence, for instance, the cohort may be divided into groups of high, medium, and low exposure to violent television shows. The expectation is that the partitions exposed more will show more change on the dependent variable. Where partitioning is not possible, having multiple prettests and posttests can establish trends to rebut "it would have happened anyway" arguments about the validity of conclusions under this design.

    14. Regression-Discontinuity Design: One might hypothesize that if there is a treatment effect, then the slope of the regression line relating scores before and after treatment would be the same, but there would be a discontinuous jump in magnitude on the dependent variable immediately after treatment. This test requires verification that the relationship between prettest and posttest scores is linear, as two linear regressions (one before, one after treatment) on a curvilinear underlying relationship could spuriously appear to meet this test. Also, there may be a treatment effect taking the form of a steeper regression slope but no discontinuity at the point of treatment. Such a treatment effect is very difficult to differentiate from a simple curvilinear relationship.

    15. Regression Point Displacement Design. In this design there is a treatment group (ex., a county) and a large number of comparison groups (ex., other counties in the state). Consider the case where we wish to estimate the effect of an after-school intervention on juvenile crime. In the pretest condition we regress juvenile crime rates on, say, median income level and we note the position of the test county in the regression scattergram. In the posttest condition, after we have implemented the intervention program, we re-run the regression. If the location of the test county is displaced on the regression scattergram, we conclude that the intervention had an effect.

    
    

  • Interrupted Time Series

    Above, in the discussion of non-equivalent control group designs, it was suggested that pretest-posttest versions could be improved by having at least two pretests to establish linear tendencies apart from treatment. Cook and Campbell (1979) list six interrupted time series designs which extend this suggestion by having multiple pretests and posttests.
    1. Simple Interrupted Time Series Design. This is the one-group pretest-posttest design augmented with multiple pretests and posttests. The trend found in multiple pretests can be compared to the trend found in multiple posttests to assess whether apparent post-treatment improvement may simply be an extrapolation of a maturation effect which was leading toward improvement anyway. Since there is no control group, however, the researcher cannot assess other confounding factors such as history-type challenges to validity ( the possibility that other factors historically coterminous with the treatment actually led to the observed effect). There may be other problems such as failure to seasonally adjust data, confounding a seasonal effect with a treatment effect; selection bias, as due to non-random attrition of subjects in the posttest; instrumentation bias (the posttest is not equivalent to the pretest); and testing (there may be a learning effect from the pretest such that the observed effect is one a test artifact rather than a treatment effect).

    2. Interrupted Time Series with a Nonequivalent No-Treatment Comparison Group : This is the two-group pretest-posttest design using an untreated control group, but with multiple pretests and posttests. By having a comparison group, even if nonequivalent (not randomized), the same threats to validity can occur, but they usually occur in a more complex and hence more easily disproved way. For instance, if this design shows an improvement in the treatment but not comparison group, it may still be true that there is historical bias, but such biasing history factors must be unique to the treatment group for some reason not experienced by the comparison group. There could be seasonal bias, but only if the seasonal factors were thought to be uniquely associated with treatment. The researcher's main validity challenge is to show the two groups were equivalent on all causally important variables prior to treatmen (ex., in a study of a rehabilitation program's effect on recidivism, to show the two groups of prisoners were similar in crime record, age, etc.).. An alternative strategy, which only works for stronger effects, is to select a treatment group which would be expected to measure worse on posttreatment (ex., prisoners with worse criminal records than the comparison group, in a study of recidivism), on the theory that if the effect shows in spite of an adverse starting point for the treatment group, the treatment has an effect.

    3. Interrupted Time Series with Nonequivalent Dependent Variables : This is the nonequivalent dependent variables pretest-posttest design with multiple pretests and posttests. The object is to find dependent variables related to the dependent being studied, but where the related variables are not thought to be correlated with the treatment variable. Cook and Campbell (1979) give the example of influence on accident rates (the dependent) of breathalyzer tests (the treatment variable) given by police when bars are open weekend nights, but not given at other times. The dependent variable of interest is accident rates on weekend nights. The related dependents are accident rates on weekday nights when bars are open, and accident rates at times when bars are not open. The expectation was that accident rates would be significantly lower on weekend nights because of the presence of the treatment. Counter-explanations for lower accident rates (ex., safer cars, stricter court treatment of offenders) must explain not only the lower accident rate on weekend nights, but also the lack of effect at other times. Of course, confounding factors may well exist, but they must be unique to the dependent variable of interest.

    4. Interrupted Time Series with Removed Treatment : This is the removed-treatment pretest-posttest design with multiple pretests and posttests, including ones in between the original treatment and its removal, and hence is a more powerful test. For instance, the threat of history is reduced because any historical forces coincident with treatment would also have increase after treatment and decrease after removal, an unlikely circumstance. Ideally removal of treatment does not occur until enough observations have been taken to rule out any seasonal or other cyclical effects.

    5. Interrupted Time Series with Multiple Replications . This is simply the interrupted time series with removed treatment design, except that treatment and removal occur multiple times on a schedule. Circumstances rarely permit such a design, but it is stronger yet. By timing the replications randomly, the researcher is able to minimize contamination from cyclical factors. This design assumes one is dealing with a treatment effect which dissipates in a timely manner before the next replication, without carryover effects (otherwise there is "multiple treatment interference," meaning that receiving earlier treatments adds to or multiplies the effect of receiving later treatments).

    6. Interrupted Time Series with Switching Replications . This is a further refinement in which there are two groups, each serving as either the treatment or comparison group on an alternating basis, through multiple replications of treatment and removal.This requires an even higher level of control over subjects by the researcher but is a particularly strong design in ruling out threats to validity. It does not lend itself to studies where the treatment intervention has been gradual, or where treatment effect does not decay well.

  
  

• Non-Experimental Designs

  A design is non-experimental if the subjects are neither randomly assigned nor randomly selected. There may still be comparison groups.
  • Case study designs are discussed in a separate section.

  • Content analysis is discussed in a separate section. Under some circumstances, content analysis may also be part of a quasi-experimental design.

  • Ethnography is discussed in a separate section.

  • Focus groups are discussed in a separate section.

  • Narrative analysis is discussed in a separate section.

  • Network analysis and sociometry are discussed in a separate section.

  • Participant observation is discussed in a separate section.

    
    




Assumptions

• The researcher is assumed to have a research design!

• The researcher is assumed to have considered all threats to validity associated with the design.

• In the case of experimental designs, it is assumed that randomization of subjects controls for all unmeasured variables. However, the smaller the sample size, the less likely this is to be true.




Frequently Asked Questions

• Is a quasi-experimental design ever preferable to an experimental design?
• How do I handle the problem of sample attrition in designs which involve observations at two or more time periods?
  
  
  
  
• Is a quasi-experimental design ever preferable to an experimental design?
  In principle, no. However, it frequently occurs that an experimental design is not feasible, ethical, or even lawful. Data may need to be analyzed on the basis of existing archival information, it may be impossible to randomize subjects, and pretest data may be absent. Even when randomized experiments are undertaken, they may become flawed due to such factors as attrition in the treatment group, in which case the wise researcher will have a "fall-back" quasi-experimental design for purposes of analysis.

• How do I handle the problem of sample attrition in designs which involve observations at two or more time periods?
  There is no statistical "fix" for this problem. The researcher must report and discuss differences in means and variances for key variables, including demographic variables, which differentiate the samples at each observation time point. The researcher may not be able to generalize findings to the original sample as first intended. With larger sample sizes it may be possible to model attrition, such as by methods discussed by Duncan and Duncan (1994). The researcher may also wish to consider the pros and cons of weighting later samples in light of distributional characteristics of the initial sample.




Bibliography

• Cook, Thomas D. and Donald T. Campbell, Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton-Mifflin, 1979. A leading classic. See Shadish, Cook, and Campbell (2002).
• Creswell, John W. (2002). Research design: Qualitative, quantitative, and mixed methods approaches. Thousand Oaks, CA: Sage Publications.
• Duncan, S. C. and T. E. Duncan (1994). Modeling incomplete longitudinal substance use using latent growth curve methodology. Mulitvariate Behavioral Research, 29:
• Leedy, Paul and Jeanne Ellis Ormrod (2004). Practical research : Planning and design (8th Edition). A leading text.
• Levin, Irwin P. (1999). Relating statistics and experimental design. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences series #125. Elementary introduction covers t-tests and various simple ANOVA designs. Some additional discussion of chi-square, significance tests for correlation and regression. and non-parametric tests such as the runs test, median test, and Mann-Whitney U test.
• Pedhazur, E.J. and L. P. Schmelkin. (1991). Measurement, design, and analysis: An integrated approach. Lawrence Erlbaum Assoc. A widely used textbook.
• Shadish, W.R., Thomas D. Cook, and Donald T. Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin. An update of a classic by a third author.

Copyright 1998, 2008 by G. David Garson..
Last update 4/4/2008.