Experiments and quasi-experiments: methods for evaluating marketing options; hospitality managers could achieve greater success

Hospitality executives have available a number of different research methodologies and tools to aid them in decision making. Each methodology is valuable in its own way, but no single technique can provide all the answers to decision makers' questions. Exploratory research (such as focus groups and depth interviews (1)) and descriptive research (such as surveys (2) or naturalistic observations (3)) can provide insight and understanding about business problems and opportunities and thereby guide decision makers' search for promising courses of action. These techniques do not, however, allow researchers to draw conclusions about cause-and-effect relationships. Consequently, these techniques are of limited usefulness in revealing how effective a specific potential action will be. That is the province of causal research methods, such as choice modeling and experimentation, which can help decision makers draw conclusions about the effects, benefits, and influences of their prospective actions. Exploratory, descrip tive, and causal research methods have a place in every functional area of hospitality businesses. In this article we focus on the use of causal-research methods in hospitality marketing.

Although systematic data on the use of different types of research in marketing are not available, several informal sources suggest that marketers often rely on exploratory and descriptive research, but rarely use causal research. For example, a web-based search by topic of Quirk'ds Marketing Research Review from 1986 to 2001 indicated that this magazine has published 162 articles on focus groups and 59 articles on telephone interviewing and mail surveys, but only 22 articles on choice modeling (i.e., conjoint or trade-off analysis). (4) "Experiments" was not even listed as a search topic. Assuming that the frequency with which different research methods are written about in marketing-research magazines roughly reflects the frequency with which those methods are used by marketing researchers, the data from Quirk's would indicate that 90 percent of marketing research is exploratory or descriptive and only 10 percent is causal. Similar estimates were obtained from a query of the founders of two large firms enga ged in marketing research for the hospitality industry. One estimated that 95 percent of hospitality-marketing-research expenditures are devoted to exploratory or descriptive research, while the other estimated that 80 percent of hospitality-research budgets are for exploratory or descriptive research. (5) It is clear to us that causal methods such as experiments are a rarity in hospitality-marketing research today.

In this article we advocate the increased use of experiments and quasi-experiments in hospitality-marketing research. The article is divided into three sections. Section one contains an explanation of why marketers should use causal research methods to evaluate the effects on consumers of different marketing actions. Section two contains a brief description of two causal-research methods--namely, true experiments and quasi-experiments--along with a discussion of their strengths and weaknesses. Section three contains a discussion of issues relating to conducting experiments and quasi-experiments and interpreting their results, which should give the reader an understanding of how to conduct and evaluate this type of research.

The Need for Causal Research

Ideally, hospitality marketers would first conduct exploratory and descriptive research to get an understanding of marketing problems or opportunities and would use this information to develop multiple courses of action that they believe will address those problems or capitalize on those opportunities. The proposed courses of action would then be systematically tested to discover whether they actually influence consumption behavior. Too often, however, marketers conduct only exploratory or descriptive research (as described above) and then develop just one course of action based on what they learn from those exercises. Kevin Clancy and Peter Krieg characterize this failure to develop and test several marketing options as a form of "death-wish marketing." (6) The problem with this practice is that the marketplace is so complex that no single course of action, even if well grounded in an understanding of the marketplace, is assured of producing the desired outcomes. In fact, marketers have a history of failing more than they succeed. Consider the following statistics compiled by Clancy and Krieg:

* the average brand loses market share each year,

* 90 percent of new products fail within three years,

* the average advertisement returns only 1 to 4 percent on the investment made in it,

* only 16 percent of trade promotions generate a profit, and

* the average firm satisfies less than 80 percent of its customers. (7)

These statistics suggest that there is enormous room for improvement in market research. If marketers rigorously tested various marketing options before settling on a course of action, we believe they could identify where failure will occur before encountering it first hand.

On the rare occasions that marketers do test different marketing options or evaluate specific marketing actions already undertaken, they often use exploratory or descriptive research methods that are poorly suited to support conclusions about the proposed actions' effects on consumer behavior. Focus groups and surveys, for instance, are used to get consumers' opinions about advertisements, frequency programs, new product ideas, and other marketing options under consideration. Those options that consumers report liking best are then implemented or continued. One example of this approach can be found in the Harris Ad Research Service, which surveys a national sample of adults about how much they like various ads being run in the marketplace and how effective they think those ads are. (8) Companies who subscribe to this service are told how consumer attitudes and opinions about their ads compare to the average of consumer attitudes and opinions about the other ads being evaluated. Presumably, companies use this information by continuing ads that score well and by discontinuing ads that score poorly. Among the hospitality brands whose ads have been evaluated using this service in the past several years are Avis, Burger King, Domino's, Hertz, Holiday Inn, KFC, McDonald's, Pizza Hut, Priceline.com, and Red Lobster. (9)

One of the problems with this use of descriptive research to evaluate marketing options is that it is based on incorrect assumptions about consumer psychology. Using consumers' attitudes and beliefs to predict how they will react to certain marketing initiatives assumes that their attitudes and beliefs strongly affect their consumption behavior. However, psychologists have found that behavior is affected by many factors and that specific attitudes and beliefs are only weakly predictive of how people will behave in any given situation. (10) For example, attitudes towards an ad are only weakly related to purchase intentions and brand choice. (11) The weak link between attitudes toward an advertisement and purchase behavior explains why popular ad campaigns, like Taco Bell's ads that feature a Chihuahua saying, "Yo quiero Taco Bell," often fail to increase sales. (12) Asking consumers to predict their own behavior is also unreliable. Psychologists have found that people are not aware of all the factors that affe ct their behavior and, most important, that they cannot accurately predict how they will react to events. (13) This is one reason that attempts to sell healthful or low-fat menu items in restaurants have generally failed despite consumers' reports that they want and will buy such items. (14)

To achieve company goals, marketers need to know what the effects of various marketing options or actions are on consumers' purchase behavior. While exploratory and descriptive research can provide information about consumer perceptions of, and attitudes toward, marketing options, these techniques cannot answer questions about how those options will affect consumer behavior. This is because the underlying causes of behavior are too complex to be accurately predicted from attitudes and opinions--even by consumers themselves. Fortunately, causal research methods can answer such questions. In the sections below we describe and discuss two causal research methods--namely, experiments and quasi-experiments.

Experiments, Quasi-experiments, and Their Limitations

Experiments are a type of research based on the following logic. If you identify two or more groups that are equivalent, expose those groups to different treatments, and subsequently observe differences between the groups on some dimension of interest, then you can reasonably conclude that those differences must be caused by the treatments. The following are characteristics of true experiments: (1) at least one treatment group and one comparison group, (2) at least one outcome measure, and (3) random assignment of subjects to treatments. Experiments can be used to test the effects of different prices, ad appeals, sales promotions, product changes, or any other marketing actions being considered on consumer attitudes and, most important, behavior.

For example, an experiment examining the effects of two menu designs on a restaurant's sales might randomly assign dining parties to receive one of the two menus by flipping a coin. A party sees one menu design when heads comes up and the other menu design when tails comes up. By keeping track of which dining parties received which menu design, the experimenter can compare the average check achieved with each menu design. Assuming that the samples involved are large, random assignment of dining parties distributes their characteristics evenly across the different groups and ensures that the groups seeing each menu design (or treatment) really are comparable. Thus, any subsequent difference in average check observed between the treatment groups can be attributed to the menu designs, and the experimenter can be confident that she knows which of the two designs will produce the largest sales in that restaurant.

Print Get the Mag Weekly Updates