Introduction: evaluation in resource and environmental planning.

Evaluations based on best practice standards compare plans to a theoretical ideal that may be impossible to achieve. Best practice evaluations are useful in indicating how the plan can be improved, but can lead to the erroneous conclusion that the planning model is deficient and should be rejected. For example, one of the case studies in this volume evaluates an innovative collaborative planning model that failed to reach a consensus agreement and was rated as a failure by the participants. This finding could lead to the conclusion that collaborative models are deficient. This would be an unfounded conclusion unless it could be demonstrated that an alternative planning model would have led to a successful outcome or be more likely to lead to a successful outcome. It is important to compare a plan to the feasible alternatives when assessing performance, rather than to a theoretical ideal.

Case Studies

The five case studies in this volume evaluate various aspects of environmental and resource planning. Each case study provides a framework for evaluation and evaluation findings. Two of the case studies deal with collaborative land use planning processes, two deal with protected areas planning, and one deals with water policy. The case studies are categorized by type of evaluation in Table 2.

Collaborative Planning: Lillooet Land and Resource Management Plan (Gunton et al.)

Collaborative planning has emerged as an increasingly popular model that is alleged to have significant benefits relative to other planning models. Collaborative planning delegates the responsibility for planning to stakeholders who engage in face to face negotiations to prepare a plan by consensus agreement. Despite collaborative planning's growing popularity, evaluation of collaborative planning is still in its infancy (Gunton and Day 2003). Without proper evaluation, the merits of collaborative planning relative to other planning methods and best practice guidelines for effective implementation of collaborative planning will remain uncertain.

Gunton et al. address the need for more evaluative research on collaborative planning by providing a case study evaluation of a collaborative planning process used to prepare a regional land use plan for the Lillooet region in British Columbia. The case study is part of a larger multiyear research program on collaborative planning in the School of Resource and Environmental Management at Simon Fraser University. The Lillooet process was chosen for a detailed case study because it is the only collaborative process under the Land and Resource Management Planning initiative in British Columbia that did not reach a consensus-based agreement. The Lillooet process also experimented with a unique final offer selection technique in an attempt to reach a decision.

The Lillooet evaluation utilizes an evaluation methodology developed as part of the larger collaborative planning research program at Simon Fraser University. The Lillooet evaluation faces several major challenges common to evaluating resource and environmental planning. The first challenge is identifying standards for comparison. The official goals for the collaborative planning process are too vague to provide a clear foundation for evaluation. Consequently the Lillooet case utilizes a comprehensive list of fourteen best practices process management criteria and eleven outcome criteria that were developed based on a review of the literature on collaborative planning. The importance of the evaluation criteria was tested by surveying stakeholders.

The next challenge is assessing the degree to which the evaluative criteria are met. For most of the criteria, such as neutrality of management staff, objective data for assessing the degree to which the criteria are met are not available. Consequently, an alternative approach is required. One approach is for the researchers to make a judgment using some type of rating scale such as met, partially met, or not met. The problem with this approach is that in many cases the researchers do not have a basis for making an accurate judgment. Consequently, researchers' judgment was rejected in favor of relying on a survey of stakeholders engaged in the process to rank the degree to which the criteria are met on a Likert-type scale, complemented by open-ended questions. Follow-up interviews were also conducted with stakeholders to elaborate on questionnaire responses. In this way, the evaluation used external evaluators to design the methodology and construct the evaluation framework, and internal evaluators to rate performance of the program.

The evaluation of the Lillooet planning process successfully identifies strengths and weaknesses and indicates how the process could be improved. For example, the findings suggest that: the process should have allowed the stakeholders more time to achieve a consensus outcome instead of using a final offer selection process; facilitators should have used a single text planning approach to discourage the preparation of competing plans from different stakeholder groups; and, increased effort should have been made to engage key stakeholder groups who were not part of the process. Therefore, the evaluation provides useful data for improving the process in the future.

Gunton et al. also point out that the evaluation method they used has deficiencies. The largest problem is that the evaluation relies on the perspectives of stakeholders, which may or may not be accurate. For some criteria, such as the neutrality of staff and the degree to which the outcome met the interests of each stakeholder group, this is not a problem because the perception of stakeholders is a sound basis for measuring performance. For other criteria, particularly outcome criteria such as the extent to which the process met the public interest and whether the process was superior to alternative methods of planning, stakeholder perceptions are less reliable. Stakeholder evaluations are also constrained by low response rates that make the interpretation of results challenging. The response rate in the Lillooet case study was 33%, which has a low confidence range for a small sample size. To remedy these problems with stakeholder surveys, the authors suggest developing more objective measures of outcomes to assess success. The survey is also based on a single snapshot of stakeholder opinion taken after the completion of a planning process. Accuracy would be improved by completing evaluation in stages over an extended time horizon.

Another limitation is that the evaluation is based on a single case study, which is too small a sample to provide reliable generic conclusions on the merits of the planning model used. The researchers address this limitation by pointing out that the study is part of a research program that is based on a much larger number of collaborative planning processes.

The authors emphasize several important lessons for evaluation from the case study. First, it is crucial to have multiple evaluative criteria. Relying on narrowly defined evaluative criteria may exclude important benefits of the plan and result in unjustified rejection of the planning model. For example, although the planning process did not achieve the desired outcome of a consensus agreement on a plan, the process did achieve important other benefits including improved stakeholder relations and improved stakeholder skills and knowledge. Second, the case study illustrates that it is important to compare the planning model to feasible alternatives as well as best practice ideals to assess merits. A planning model may not meet best practices criteria, but it still may be superior to all the alternatives. However, as the authors caution, comparing alternatives is challenging because the alternatives can rarely if ever be tested as part of a controlled experiment where all factors are held constant.

Ontario Resource Stewardship Agreements (Browne et al.)

Conflict between the tourism and forest industries in Ontario led to the signing of a memorandum of understanding by the Ontario government, the tourism industry, and the forest industry to use a new collaborative process called resource stewardship agreements (RSA) to help resolve land use conflicts. The second paper, by Browne et al., in this volume summarizes an evaluation of the RSA process.

The RSA evaluation sets out to answer two questions. The first question is whether the RSA process is meeting goals set by policy makers and the tourism sector. The second question is the whether the RSA process meets best practice requirements as defined in the academic literature for collaborative planning processes. Criteria used for evaluating the RSAs, were taken from three sources: government goals for the process, tourism industry goals, and academic literature on best practices.

The next step in this evaluation was to assess the degree to which the evaluative criteria were met. The study relies on two sources: stakeholder responses based on a mail survey and researcher assessments based on a review of relevant documents. In some cases, both sources were used to assess the same evaluative criterion. The combination of stakeholder assessments combined with researcher's assessments based on document analysis is an interesting approach that attempts to offset the deficiencies of relying on just one of the sources.