Introduction: evaluation in resource and environmental planning.

The results confirmed that all the criteria identified in the literature review were ranked as important to very important, except for one (independent facilitation) that was ranked as only somewhat important. The testing of the degree to which the criteria are met identified strengths and weaknesses in the planning process that need to be mitigated. An interesting finding in the study is the wide discrepancy between the ratings of the government planners and the non-governmental stakeholders, with the park planners providing more positive ratings for achievement of the best practices criteria. The researchers point out that while it is not surprising that the planners would rank the outcomes more highly, the discrepancy illustrates the need to include all relevant stakeholders in plan evaluation to counter the "internal evaluator" bias of the planners. While the researchers caution that the specific findings in the case study should be interpreted with caution because of the small sample size, they conclude that the case study application confirms the feasibility and utility of the plan evaluation methodology.

Conclusion

Effective environmental planning is contingent on comprehensive evaluation. Based on the case studies and evaluation theory we can identify eight keys to successful evaluation.

1. Evaluation should use a comprehensive set of evaluative criteria that include explicit policy goals and best practice standards that have been empirically verified.

2. The cases being evaluated should represent a large enough sample to provide reliable results.

3. Evaluation should include all components of the program including program theory, design, implementation, outcomes, and efficiency.

4. Evaluation should occur on an ongoing basis over multiple time periods at critical steps in the process.

5. If a summative evaluation is being done to assess whether the program should continue, the program should be evaluated against feasible alternatives.

6. Care should be taken in concluding causality between programs and outcomes.

7. Evaluation should incorporate the views of stakeholders in the assessment, as well as those of external evaluators.

8. Evaluation reports should clearly state the limitations of the evaluation.

In the real world of evaluation, meeting all eight criteria is extremely difficult due to resource constraints and methodological challenges. None of the case studies in this volume meet all criteria. The case studies show that identifying clear and measurable program outcomes, conducting multiple period evaluations over an extended time horizon, comparing programs against feasible options, and determining causality between programs and outcomes are particularly difficult criteria to meet. However, the case studies in this volume illustrate how useful evaluations can be conducted in the complex field of resource and environmental planning in the face of these resource and methodological constraints. In particular, the case studies illustrate the techniques and benefits of incorporating stakeholder views to verify best practices criteria and assess the degree to which best practices criteria are met. The case studies also illustrate the importance of using multiple evaluation criteria to assess program performance. Hopefully, the case studies in this volume will stimulate ongoing research in this important field of evaluation in environmental planning.

Acknowledgements

We would like to thank SSHRC for funding support for this research and the anonymous referees for their helpful suggestions.

References

Bellamy, J.A., G. T. McDonald, G.J. Syme, and J.E. Butterworth. 1999. Evaluating Integrated Resource Management. Society and Natural Resources 12: 337-353.

Day, J.C., Thomas I. Gunton, and T. Frame. 2003. Towards Rural Sustainability in British Columbia: The Role of Biodiversity Conservation and Other Factors. Environments 31(2): 21-39.

Clark, Alan and Ruth Dawson. 1999. Evaluation Research. Thousand Oaks, California: Sage Publications.

Esty, Daniel C., Marc A. Levy, Tanja Srebotnjak, Alexander de Sherbinin, Christine H. Kim, and Bridget Anderson. 2006. Pilot 2006 Environmental Performance Index. New Haven, Conn.: Yale Centre for Environmental Law and Policy.

Gunton, Thomas.I. 2003. Natural Resource Megaprojects and Regional Development: Pathologies in Project Planning. Regional Studies 37(95): 505-519.

Gunton, Thomas I., and J. C. Day 2003. The Theory and Practice of Collaborative Planning in Resource and Environmental Management. Environments 31(2): 5-19.

Gunton, Thomas I., and Chris Joseph.2006. Toward a National Sustainability Strategy for Canada: Putting Canada on the Path to Sustainability within a Generation. Vancouver: David Suzuki Foundation.

Gunton, Thomas I., Ken Calbick, Anita Bedo, Emily Chamberlin, Andrea Cullen, Krista Englund, Aaron Heidt, Matthew Justice, Gordon McGee, Sean Moore, Carolyn Pharand, Ian Ponsford, Jennifer Reilly and Ian Williamson. 2005. The Maple Leaf in the OECD: Comparing Progress Toward Sustainability. Vancouver: David Suzuki Foundation.

International Institute for Sustainable Development (IISD). 2004. National Strategies for Sustainable Development: Challenges, Approaches and Innovations in Strategic and Co-ordinated Action. Winnipeg: IISD.

Joseph, Chris, Thomas I. Gunton and J.C. Day. 2007. Planning Implementation: An Evaluation of the Strategic Land Use Planning Framework in British Columbia. Journal of Environmental Management (in press).

Leach, W.D., N. Pelkey and Paul Sabatier. 2002. Stakeholder Partnerships as Collaborative Policymaking: Evaluation Criteria Applied to Watershed Management in California and Washington. Journal of Policy Analysis and Management 21(4): 645-670.

National Academy of Public Administration (NAPA). 2001. Evaluating Environmental Progress: How EPA and the States can improve the Quality of Enforcement and Compliance Information. A Report by a Panel of the National Academy of Public Administration. [Accessed on 15 June, 2007].

Organization for Economic Cooperation and Development (OECD). 2004. OECD Environmental Performance Reviews: Canada. Paris: OECD.

Rossi, Peter, Mark Lipsey and Howard Freeman. 2004. Evaluation: A Systematic Approach. Thousand Oaks, California: Sage Publications.

Suvedi, Murari, and Shawn Morford. 2003. Conducting Program and Project Evaluations: A Primer for Natural Resource Program Managers in British Columbia. FORREX-Forest Research Extension Partnership. Kamloops, B.C. FORREX Series 6. [Accessed on 18 April, 2007].

Weiss, Carol. 1998. Evaluation Methods for Studying Programs and Policies. Second Edition. Upper Saddle River, New Jersey: Prentice-Hall.

Thomas Gunton is a professor in the School of Resource and Environmental Management and Director of the Resource and Environmental Planning Program at Simon Fraser University. He has held numerous senior positions in government including Assistant Deputy Minister of Energy and Mines for the government of Manitoba and Deputy Minister of Environment, Lands, and Parks for the government of British Columbia. His research focuses on environmental mediation and dispute resolution and resource and environmental planning. He can be contacted at tgunton@shaw.ca

Murray Rutherford is an Assistant Professor in the School of Resource and Environmental Management at Simon Fraser University. He is a policy scientist and planner whose research focuses on policy analysis and evaluation, ecosystem-based management, and human values and attitudes toward nature and the conservation of biological diversity. He can be contacted at mbr@sfu.ca

Peter Williams is a professor in the School of Resource and Environmental Management and Director of the University Centre for Tourism Policy and Research at Simon Fraser University. His research relates to the use of land and resources for sustainable tourism. He can be contacted at peterw@sfu.ca

Chad Day is professor emeritus and founding director of the School of Resource and Environmental Management at Simon Fraser University. His research focuses on institutions for integrated land and water management and environmental planning. He can be contacted at jday@sfu.ca Table 1. Evaluation Options Issue Option Who conducts evaluation 1. Internal

2. External Purpose of evaluation 1. Program Improvement (formative evaluation)

2. Program justification (summative evaluation)

3. Generic knowledge (theoretical evaluation)

4. Hidden agenda (ulterior evaluation) Timing 1. Before implementation

2. During program operation

3. After program completion Timing Scope 1. Single snap shot

2. Multiple period Content Scope 1. Program theory

2. Program design

3. Program implementation

4. Program outcomes

5. Program efficiency Methodology 1. Qualitative

2. Quantitative Evaluative Criteria 1. Program goals

2. Best practices

3. Social welfare (benefit-cost)

4. Efficiency (cost-effectiveness)

5. Time series trend