Evaluating collaborative planning: a case study of a Land and Resource Management Planning process.

Process management criteria summarized in Table 6 show that respondents believed that the process was well managed, with agreement ranging from 75% to 100% for most process criteria. Process criteria that received lower agreement were stakeholder design of process (69%), clear purpose of the process (63%), commitment to implementation (56%), and equality of power among stakeholders (56%). The process criterion with the lowest agreement (31%) was having realistic time limits. Unrealistically short time lines was also the second most frequently cited weakness in the open-ended responses (frequency of 33%), just behind inequality of stakeholders (frequency of 38%). This suggests that the government-imposed deadline that necessitated the use of a final offer selection process was viewed by respondents as a major weakness of the process.

Implications for Collaborative Planning

The findings of the evaluation provide important insights into collaborative planning. First, the results show that collaborative planning generates an array of benefits beyond achieving agreement on a plan. Even though the Lillooet process has not yet resulted in a plan approved by government, respondents still strongly agreed that it generated important social capital benefits such as improved relationships, knowledge, and skills for all participants. The process was also successful in significantly narrowing differences among stakeholders on protected area designation. Narrowing differences and improving stakeholder relations has helped move the land use conflicts closer to resolution. Second, respondents agreed that the collaborative process was superior to alternative approaches despite its failure to reach agreement.

The Lillooet experience provides some insights into the keys to success in collaborative planning. First, the success of a collaborative process is contingent on allowing sufficient time to achieve agreement. On average, the collaborative processes in British Columbia took between four and five years to complete, with a range from three years to ten years depending on the plan (Frame et al. 2004). In the case of the Lillooet process, a five-month deadline was imposed after four years of process in attempt to reach a decision prior to an upcoming election. The imposition of an unrealistic five month deadline was cited by respondents as one of the key barriers to achieving a successful outcome. To be acceptable to participants, collaborative processes need to be allowed sufficient time to reach a decision without the imposition of unrealistic deadlines.

Second, the success of collaborative planning is contingent on participation of all key stakeholders. In the Lillooet process, two key groups were absent: First Nations, who declined an invitation to participate as members of the planning table at the inception of the process and the mining industry, which withdrew part way through all provincial LRMP planning processes in 1999. Although both First Nations and the mining industry partially participated in the process, the absence of these key parties as fully active members of the planning table undermined support for the decision made by the government in 2001. Weak stakeholder support contributed to the rejection of the 2001 agreement by government. Consultation with First Nations and resource interests are the foundations of the current process to try to reach broadly-based social support for land use decisions.

Third, collaborative planning requires the stakeholders to work as a group on a joint solution. In the Lillooet process, the stakeholder table remained polarized into two major groups that were allowed to develop their own separate proposals. Once the separate proposals were developed by each group, the two groups remained attached to their own proposal and were unable to forge a consensus agreement. The use of final offer selection reinforced the division between the two groups and created a winner and loser, which is contrary to the fundamentals of collaborative planning. The creation of a winner and loser reduced support for the 2001 plan, which encouraged rejection of the plan by government.

Implications for Evaluation Methodology

The evaluation of the Lillooet planning process also provides some useful insights into evaluation methodology. First, the case study shows that it is important to use multiple evaluative criteria covering the spectrum of potential benefits. If the narrow criterion of achieving agreement on a plan acceptable to all parties was used, the planning process would be defined as a failure. The inclusion of other measures of success such as social capital provides a more accurate assessment of the merits of the process. Second, care must be taken to evaluate a process relative to feasible alternatives, not relative to just absolute measures of success. The Lillooet planning process did not produce an agreement and did not fully meet many of the evaluative criteria. While these findings are useful for identifying ways in which the process could be improved, the findings could be mistakenly used to reject collaborative planning as a preferred model. In the Lillooet case study, respondents were highly critical of the outcomes but they still strongly agreed that the collaborative model of planning was superior to alternative models. This finding illustrates that the key question in deciding whether to reject a program or process is not whether the process has deficiencies, but whether the deficiencies are greater or lesser than feasible alternative processes. Sound evaluation, therefore, must compare the program being evaluated relative to feasible options.

The case study evaluation also illustrates some of the challenges in evaluation. First, it is not possible to conduct a controlled experiment in which all factors are held constant to allow comparison of alternative planning processes. There are too many confounding factors. This study attempted to address this problem by comparing the collaborative model to alternatives by relying on stakeholder perceptions. Stakeholders, however, may have limited experience with other models of planning and be unable to make valid comparisons. Second, although it is important to use multiple criteria for evaluation, it is difficult to assess objectively the degree to which the criteria are met. The case study relied on stakeholder perceptions to assess the degree to which the criteria were met. However, stakeholder perceptions may be inaccurate or biased. Further, stakeholder perceptions may change over time, depending on the stage of the planning process. A single survey therefore may not accurately record stakeholder perceptions. Also, some stakeholder views may not be recorded because the stakeholder interests were not represented at the table. In the Lillooet evaluation, for example, the perceptions of First Nations and the mining industry are not recorded because neither group were members of the planning table.

These weaknesses in the evaluation of the Lillooet process can be addressed by modifying the evaluation methodology. Multiple surveys can be taken at various stages of the planning process to more accurately record stakeholder perceptions, stakeholders not at the table can be surveyed, and stakeholder perceptions can be complemented by other data sources using objective measures of success. These approaches were not used in this study due to limitations in resources, difficulty in identifying which stakeholders not at the planning table should be surveyed, and difficulty in identifying and interpreting so called objective measures of success. Objective measures of success, and the degree to which objective measures are met, require a degree of quantification that is not possible for many types of planning processes. Planning processes have diverse outcomes that are difficult to measure. Additionally, it is difficult to distinguish between outcomes due to the planning process and those due to other factors. For example, environmental indicators may show deterioration in ecological health in the planning area, but the deterioration may be due to exogenous factors such as climate change instead of failures in the planning process. Without clear definition of the measurable impacts of planning, it is also not possible to estimate net benefits or cost/effectiveness of the planning process. Consequently, evaluation of planning outcomes based on objective measures has major weaknesses. Nonetheless, attempts will be made to address these weaknesses in future evaluations of collaborative planning by the SFU research team.

Conclusion