Would you choose your preferred option? Comparing choice and recoded ranking experiments.

The most widely used elicitation formats in conjoint analysis (CA) applied to environmental valuation have been rating, ranking, and choice. As economists tend to prefer ordinal measures of preferences rather than cardinal measures, especially due to the more obvious interpretation in terms of random utility (Roe, Boyle, and Teisl 1996; Holmes and Boyle 2001), we focus on ranking and choice experiments. The differences found between these formats in previous studies are not particularly surprising because different statistical techniques are used, and violations of transitivity after the first rank have been observed both in experiments and in field applications (see Foster and Mourato (2002) and Bateman et al. (2007)). However, even if data obtained through a ranking exercise are recoded as a choice experiment by assuming that the option ranked first would be the option chosen, and analyzed using statistical techniques employed in choice experiments, differences in response have been found to persist between the two formats (see Boyle et al. (2001) and the remaining literature summarized in table 1). That is, according to previous research, saying that an option is your preferred option is not the same as saying that you would choose the option. The explanation provided in Boyle et al. (2001) is that the cognitive process is different, if we ask a person to state her (or his) preferred option rather than asking her/him to state the option that she/he would choose. Holmes and Boyle (2001) state that "This surprising result suggests that different cognitive processes were used in seemingly identical tasks (i.e., choose the most preferred profile from a set)." If we turn the argument upside down, one can argue that a method that yields different results with such similar tasks (choose one versus state your preferred option) has a fundamental problem.

As we feel that, if true, this would have profound implications not only for economic valuation through CA but also for economic theory, we decided to investigate whether this difference remains when some of the shortcomings of existing comparisons are removed. Although the details are discussed below, in our experiment we essentially ensured that the choice and the ranking surveys were identical in all relevant features: same experimental design, same number of alternatives, and same questionnaire.

The implications of this comparison are also relevant for CA practitioners. If the differences persist, we should probably recommend choice experiments because they are closer to real-market decisions (Adamowicz, Louviere, and Williams 1994). If the differences disappear, the use of ranking could be recommended because we can obtain the same results as when using choice, and we may be able to use the information provided by the subsequent ranks to develop an additional measurement.

Having this idea in mind we decided to use a pairwise comparison, plus status quo, to test differences between choice and recoded ranking experiments. We chose this design because it has not been used for comparing the two formats in previous studies in spite of being the most common format used for environmental valuation applications (Adamowicz, Louviere, and Williams 1994; Blarney et al. 2002; among others). In addition, this format can be seen as a benchmark case to test the theoretical concerns discussed above (convergent-validity of a choice and a ranking recoded as a choice when both are analyzed as a choice), because it has the minimum number of alternatives necessary to provide a meaningful ranking.

Using a split-sample design we presented a choice experiment to one half of the sample and a ranking to the other half. Although a ranking can be performed using a simpler design (Louviere 1988, p. 100), we chose the experimental design to be identical in both cases. In other words, we used for both subsamples the experimental design that we would need to use in a choice, because this design can also be used in a ranking.

Our results, the opposite of those obtained in Boyle et al. (2001) and the remaining literature summarized in table 1, show that the choice experiment and the ranking recoded as a choice provide statistically similar parameter vectors (same structural models). The same cognitive process was apparently used for seemingly identical tasks (i.e., choose the most preferred profile from a set, or state the profile you would choose from a set). Aggregated and per attribute welfare measures comparison tests also show statistically identical results in almost all cases. This holds for parametric tests as well as for bootstrapping tests. We completed our analysis by trying to detect "learning" and "fatigue" effects with a subsample analysis. We also used follow-up questions to study the effects associated with the information provided, the difficulty of the valuation task, the number of sets of alternatives, and the response effort. Only the respondent's reported difficulty with the valuation task turned out to be relevant.

Literature Review

Mackenzie (1993) made the first comparison of CA formats applied to environmental valuation. However, he performed a rating experiment and then simulated choices and rankings from the ratings (see also Anderson and Bettencourt 1993). In Roe, Boyle, and Teisl (1996) and Stevens, Barrett, and Willis (1997), ratings were also recoded to rankings and choices. Thus, the results of these studies are not relevant for our comparison purposes, because a choice experiment and a ranking recoded as a choice were not compared. The same holds for studies that analyze full ranks, because the differences reported can be explained by the different statistical techniques employed, and by inconsistency in the second and subsequent ranks (Chapman and Staelin 1982; Hausman and Ruud 1987; Ben-Akiva, Morikawa, and Shiroishi 1991; Foster and Mourato 2002; Siikamaki and Layton 2007). However, this should not pertain when the first rank only is analyzed as a choice.

Table 1 summarizes the main features of previous environmental valuation comparisons and cross-validity tests between independent samples of choice and rankings recoded to choice formats (i.e., we analyze only studies that focus on the first rank). Although results demonstrate differences between these formats, all the studies featured at least one of the shortcomings discussed below.

The experimental design of choice and recoded ranking experiments differ in some of the studies in table 1, because rankings can employ simpler experimental designs (e.g., Mogas and Riera (2001); see table 1). The number of alternatives offered for ranking is in some cases more than the alternatives provided to choose from (e.g., Morrison and Boyle (2001); see table 1). In comparison studies where this occurs, it is hard to discern whether the differences are caused by the process of stating preferences or by the different experimental designs inducing different results. Furthermore, when respondents face a high number of alternatives to rank, they may reduce the precision of their valuation process, or they may simply assign ranks randomly. This also affects the first rank. All the studies in table 1 provided four alternatives to rank.

The inclusion of a status quo alternative in all sets of alternatives is also relevant. As in the contingent valuation method, a reference level must exist to obtain adequate welfare measures (Roe, Boyle, and Teisl 1996). Most of the studies in table 1 did not always include the status quo in all sets of alternatives. Experimental designs in Boyle et al. (2001) and Holmes and Boyle (2001) were random in attributes, implying that the whole status quo appeared only in some of the sets of alternatives. In Mogas and Riera (2001), the choice experiment presented two alternatives plus status quo and the ranking presented four alternatives without status quo.

As table 1 shows, convergent validity was generally not obtained in parameters and only Morrison and Boyle (2001) found convergence when exclusively including respondents who stated that the valuation task was easy. As to welfare measures, results generally pointed out that they are statistically different (table 1).

Methodology

The CA exercise presented in this article was applied to the valuation, by public visitors, of a reforestation program with cork oak trees in Alcornocales Natural Park (ANP). ANP is a protected Mediterranean forest of 1,677 [km.sup.2] located in the south of Spain and it is covered by extensive woodlands where the main species is cork oak. Public visitors value its recreational environmental services highly (Campos, Caparros, and Oviedo 2007). The ANP forests currently face aging cork oak trees, due to natural mortality accentuated by diseases, and lack of natural regeneration due to overgrazing. Unchecked, this process will eventually result in the gradual replacement of the cork oak forest with shrublands. The failure of natural regeneration and private reforestation programs has led the regional administration to implement a policy providing subsidies to landowners that reforest their lands. This policy is currently being applied within the framework of the European Union Common Agricultural Policy. We decided to investigate whether social preferences, expressed through willingness to pay (WTP), are in alignment with conserving and increasing cork oak forest extent in ANE An analysis of policy implications of the results of these experiments can be found in Caparros et al. (2007).

Survey Logistics and Experimental Design