Introduction: evaluation in resource and environmental planning.

The RSA survey used a Likert-type scale to assess the opinion of respondents concerning the extent of agreement with statements describing the process. Responses were received from 116 stakeholders, for a response rate of 26%. This is another example of the low response rates common for stakeholder evaluations. The evaluative criteria were assessed as met, somewhat met, neutral or not met based on the average response.

The RSA case study provides useful evaluation results for decision makers by identifying strengths and weaknesses of the process. The RSA case study states clearly the limitations of the evaluation. First, the evaluation is done from the perspective of only one stakeholder group: the tourism sector. This is a stated objective of the research that is justified on the grounds that the purpose of the RSA is to meet the interests of the tourist sector and therefore should be evaluated from the tourist sector perspective. The researchers suggest that the government should consider broadening the RSA planning process to include all relevant stakeholders. The researchers also caution that the results are a snap shot of stakeholder views assessed early in the implementation phase. As such, the results help identify how the process can be improved, but do not provide definitive results on the success of the program. More evaluations over an extended time horizon are required.

Canadian Bulk Water Export Policy (MacNab et al.)

The third case study, by MacNab et al., evaluates an innovative approach termed "guided federalism" to develop bulk water export policy in Canada. Guided federalism is based on the federal government encouraging adoption of a national policy by the provinces by providing a recommended policy framework that provinces are encouraged, but not required, to adopt. Guided federalism is designed to address the challenges of developing consistent national policy in a federal state in which two levels of government have overlapping jurisdictions.

Concern over bulk water exports led to calls for development of a national bulk water export policy. The challenge is that both the federal and provincial governments have authority to regulate water under the Canadian constitution. To address the need for development of a consistent national policy while recognizing the rights of the two levels of government, the federal government developed the Accord for the Prohibition of Bulk Water Removal from Drainage Basins (Accord) in 1999 that outlines a proposed policy that the provinces are encouraged to adopt. The purpose of the case study evaluation is to assess whether the guided federalism approach was successful.

The case study uses eight evaluative criteria structured in the form of questions that are based on the goals of the Accord to regulate bulk water exports. The questions are defined specifically enough to allow for yes or no answers. The answers to the questions are based on a review of legislation, other relevant documents and public statements.

The evaluation concluded that guided federalism did not achieve its objectives in the bulk water case based on the fact that none of the jurisdictions meet all eight evaluative criteria. An unavoidable limitation of the study is that it is not able to compare the guided federalism policy against feasible alternative strategies--such as a compulsory national strategy--because no alternative strategy has been implemented and therefore no alternative can be evaluated. Thus the study shows that guided federalism has not met the objectives but does not show that other feasible strategies would have been more successful.

Protected Area Selection (Paridaen et al.)

The case study on protected area selection by Paridaen et al. evaluates the process for selecting the protected areas that were designated in British Columbia to achieve a more than doubling (by area) of the park system between 1990 and 2002. The purpose of the evaluation is to determine: 1) what criteria for the selection of protected areas were deemed to be important by stakeholders; and 2) whether the criteria deemed important were actually used in the selection process.

The first step in the evaluation was to complete a literature review to identify protected area selection criteria. Twenty-four criteria were selected and grouped into three categories: environmental, social, and economic. Next, a survey was developed to test the significance of the criteria by using a five point Likert-type scale ranging from very important to not at all important. The survey questionnaire was designed as a self-administered, mail back survey.

The next step was to select survey respondents. Four case study regions were chosen to represent a cross section of completed land and resource planning processes that designated protected areas. Surveys were distributed to all 170 stakeholders who participated in the protected area selection process in the four regions and responses were received from 46 participants for a response rate of 27%. Respondents were asked to rate the generic importance of the criteria, and the importance of the criteria in the actual protected areas selection process that they participated in, on the five point Likert scale. The average respondents' ranking for each criterion was calculated. The generic importance ratings were then compared with the actual ratings to evaluate the extent to which the selection process was based on the generic importance criteria.

The study found that environmental criteria were ranked as the most important for protected area selection, followed by social, and economic. The ranking of the criteria actually used in the selection process was similar to the generic ranking. However, the use of social and economic criteria was lower in the actual selection process than warranted by the generic ranking and some specific criteria such as increasing employment had a much lower role in the actual selection process than warranted by their significance ranking. The researchers recommend that future protected area selection processes be designed to give due weight to all criteria.

The protected area selection study provides one of the first attempts to rank the importance of criteria and then assess the degree to which the criteria were used based on a survey of stakeholders engaged in the selection process. This evaluative method, in essence, uses stakeholders to develop the evaluation criteria and then assess the extent to which the criteria are met. Given that policy is intended to meet democratically determined goals, the methodology of stakeholder surveys to set evaluation criteria and assess the degree to which the criteria are met clearly has merit. However, this approach is limited to policy cases in which stakeholders are actively engaged and are therefore well informed. The researchers also observe that their study may suffer from selection bias because it is based on only four case studies of over twenty potential cases.

Protected Area Planning Process (Ronmark et al.)

The objective of the final study in this volume, by Ronmark et al., is to develop and test an overall methodology for evaluating planning processes. The first step is to identify best practices criteria for planning based on a literature review. Thirty-five best practices criteria are identified and grouped into three categories: planning process criteria, planning outcome criteria, and planning implementation criteria. Next, a survey is used to test the importance of the evaluative criteria and the degree to which the criteria are met in the planning process. The importance of each criterion is ranked on a four point Likert-type scale ranging from not important to very important. The extent to which the criteria are met is assessed by a five point Likert scale ranging from strongly agree to strongly disagree with statements describing the planning process. Multiple statements are used for each criterion and the average of the responses to the multiple statements is calculated to assess the degree to which each criterion is met.

The case study used to test the methodology is the preparation of park master plans in British Columbia. The survey was sent to two types of participants in the park planning process. One group consisted of the provincial government park planners and the other group consisted of non-governmental stakeholders with an interest in protected area planning. Due to logistical constraints, respondents were limited to eleven park planners representing different regions of the province and one representative from each of the fifteen non-governmental stakeholder organizations that have a stated interest in provincial park planning. The response rates from park planners and non-governmental stakeholder organizations were 82% and 67%, respectively.