Experiments and quasi-experiments: methods for evaluating marketing options; hospitality managers could achieve greater success

Unfortunately, practical considerations often require compromises in experimental design. Such compromises are not a reason to dismiss the experimental results out of hand. Instead, marketers should assess the level of threat to statistical inference, internal or external validity posed by a compromise in experimental design, and adjust their confidence in the experimental results accordingly. Even an imperfect experiment can provide useful input into the selection of marketing options as long as the marketer is aware of what conclusions the experiment can support and what conclusions it cannot. We hope that this article will increase such awareness among hospitality marketers and will encourage them to make greater use of this research tool.

(1.) See: Robert J. Kwortnik, Jr., "Clarifying 'Fuzzy' Hospitality-management Problems with Depth Interviews and Qualitative Analysis," on pages 117-129 if this issue of Cornell Quarterly.

(2.) See: Matthew Schall, "Best Practices in the Assessment of Hotel-guest Attitudes," on pages 51-65 of this issue of Cornell Quarterly.

(3.) See: Kate Walsh, "Qualitative Research: Advancing the Science and Practice of Hospitality," on pages 66-74 of this issue of Cornell Quarterly.

(4.) Quirk's Marketing Research Review can be searched at www.quirks.com/articles/search.asp.

(5.) The experts queried were Stanley Plog, founder of Plog Research, and Peter Yesawich, president and CEO of Yesawich, Pepperdine & Brown. The differences in their estimates probably reflect differences in the work done by their respective firms.

(6.) Kevin J. Clancy and Peter C. Krieg, Counter-intuitive Marketing (New York: Free Press, 2000).

(7.) Ibid.

(8.) Sample Ad Track findings are reported in USA Today every Monday and can be found at www.usatoday.com/money/advertising/adtrack/index.htm.

(9.) This claim is based on the list of Ad Track findings available at www.usatoday.com/money/advertising/adtrack/index.htm.

(10.) See: David G. Myers, "Behavior and Attitudes," in Social Psychology, third edition (New York: McGraw-Hill, 1990), pp. 33-68; and A.W. Wicker, "Attitudes versus Actions: The Relationship of Verbal and Overt Behavioral Responses to Attitude Objects," Journal of Social Issues, Vol. 25 (1969), pp. 41-78.

(11.) See: Stephen P. Brown and Douglas M. Stayman, "Antecedents and Consequences of Attitude toward the Ad: A Meta-analysis," Journal of Consumer Research, Vol. 19, June 1992, pp. 34-51; and Gabriel Biehal, Debra Stephens, and Eleonora Curio, "Attitude toward the Ad and Brand Choice," Journal of Advertising, Vol. 21, September 1992, pp. 19-36.

(12.) Christine R McLaughlin, "Animals Gone Commercial: Do They Sell Products?," as viewed at animal.discovery.com/convergence/commercials/marketing_print.html.

(13.) See: David G. Myers, "Intuition: The Power and Limits of Our Inner Knowing," in Exploring Social Psycho logy (New York: McGraw-Hill, 1994), PP. 23-30; and R.E. Nisbett and T.D. Wilson, "Telling More Than We Can Know: Verbal Reports on Mental Processes," Psychological Review. Vol. 84 (1977), PP. 23 1-259.

(14.) See: Amy Oplinger, "Survey Says ...," Praxis, Fall 1998-Winter 1999, PP. 82-85; and Wilbert Jones, "New Wealth from Health," Restaurant Hospitality, Oct. 1999, pp. 98-102.

(15.) The comparison group in both experiments and quasi-experiments can be a different group or the treatment group at a different point in time.

(16.) Interested readers can find a discussion of many quasi-experiment designs in: William R. Shadish, Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi-experimental Designs for Generalized Casual Inference (New York: Houghton Mifflin 2002).

(17.) Eric Marder, The Laws of Choice (New York: Free Press, 1997).

(18.) If a marketer is interested in identifying the effects of more than 12 different treatments, experimentation may be less cost-effective than choice modeling. This is not the place for a detailed description of choice modeling. However, Cornell Quarterly readers can find an overview of one type of choice modeling known as discrete-choice analysis in: Rohit Verma, Gerhard Plaschka, and Jordan J. Louvirre, "Understanding Customer Choices: A Key to Successful Management of Hospitality Services," Cornell Hotel and Restaurant Administration Quarterly, Vol. 43, No.6 (December 2002), pp. 15-24. One thing to keep in mind is that experiments provide stronger evidence of cause-and-effect relationships than do choice models and, therefore, are preferable to choice models as long as their costs are nor prohibitive. However, if an experiment is nor possible or would be too expensive, then choice modeling is another causal method worth considering.

(19.) Academic researchers are concerned about a fourth type of validity--known as construct validity. A conclusion has construct validity if the variables being manipulated and measured in an experiment are correctly identified and labeled. This is a concern in basic science where researchers want to make conclusions about general, abstract constructs based on specific, concrete manipulations and measures. However, in applied marketing research, the variables researchers want to make conclusions about are generally defined by their operationalizations, so construct validity is not a potential problem.

(20.) For a thorough discussion of these types of validity, see: Shadish et al., op. cit.

(21.) A free, online program that calculates the needed sample size is available at http://calcularors.srat.ucla.edu/powercalc/

(22.) Robert Rosenthal, Experimenter Effects in Behavioral Research (New York: Appleton-Century-Crafts, 1966).

(23.) More precisely, random assignment makes groups comparable on the expected, pre-treatment level of the outcome variable. Essentially, it distributes the pre-treatment propensity to respond on the outcome variable evenly across groups.

(24.) Any sample size that produces statistically significant results in an experiment with random assignment is sufficient for random assignment to have worked. As long as subjects are randomly assigned, any pre-treatment differences in propensity to respond on the outcome variable can be due only to chance. Statistical significance means that the post-treatment differences on the outcome variable are too large to be attributed to chance, so the sample size was (by definition) large enough to rule out pre-existing chance differences between treatment groups. Samples of 20 to 30 subjects per treatment are common in academic psychological experiments. However, psychologists are more concerned about the existence of a treatment effect than about its exact size. Marketers interested in reliable estimates of treatment-effect sizes will need to use samples larger than 20 to 30 subjects per treatment. In addition, marketers that are studying insensitive or highly variable outcome measures may need to use large samp le sizes.

(25.) In other words, if the unit being randomly assigned is days or restaurants then the outcome measure should be a daily or restaurant average. There are statistical techniques that allow researchers to correctly analyze experiments where the units of random assignment and the units of outcome measurement are different, but those are new and sophisticated statistical techniques that are likely to be beyond the typical executive or manager's ability to implement. Thus, we advise randomly assigning and measuring the same units.

(26.) See: Laura A. Branon and Amy E. McCabe. "Time-restricted Sales Appeals: The Importance of Offering Real Value," Cornell Hotel and Restaurant Administration Quarterly, Vol. 42, No. 4 (August-September 2001), pp. 47-52.

(27.) Rachel Kennedy and Andrew Ehrenberg, "There Is No Brand Segmentation," Marketing Research, Spring 2001, pp. 4-7.

(28.) Corliss L. Green, "Differential Responses to Retail Sales Promotion among African-American and Anglo-American Consumers," Journal of Retailing, Vol. 71 (1995), pp. 83-92.

(29.) See: Myers, op. cit.; and A.W. Wicker, "Attitudes versus Actions: The Relationship of Verbal and Overt Behavioral Responses to Attitude Objects," Journal of Social Issues, Vol. 25, pp. 41-78.

(30.) STEP measurement involves giving subjects a booklet that describes all the major competitors in a product category and instructs subjects to distribute ten stickers among the competing options to reflect the likelihood that the subjects would buy the products as described. Each product description is on a separate page of the booklet. Product descriptions include a brand name, a picture, a price and a summary of product attributes and benefits (taken from real promotional materials on that product). The number of STEP stickers a person gives a product is related to that persons subsequent purchase behavior. Furthermore, the average shares of STEP stickers products receive correlate at .92 with the products' actual market shares. See: Marder, op. cit.

RELATED ARTICLE: Glossary of Terms

Quasi-experiments: A class of common field-research techniques in which at least one treatment is manipulated and there is at least one comparison. The difference between quasi- and true experiments is that in quasi-experiments consumers are not randomly assigned to treatments.

Random assignment: Assignment of consumers to treatments in such a way that each consumer has an equal chance of getting each treatment.

True laboratory experiment: A true experiment conducted in a model of the real world (a lab). Laboratory experiments are useful in basic research in consumer behavior because they can identify and explain the general conditions that influence consumer choices. While laboratory experiments are high in internal validity, they tend to be low in external validity.

Print Get the Mag Weekly Updates