Introduction: evaluation in resource and environmental planning.

Abstract

Evaluation systematically assesses performance of policies in meeting goals. The primary purpose of evaluation is to provide information to help improve programs. There are many approaches to evaluation, each with its own strengths and weaknesses, and many types of common errors in evaluation methodology. Keys to successful evaluation include: using comprehensive evaluation criteria, using a large enough sample size, evaluating all facets of the program over an extended time horizon, comparing the program to feasible alternatives, clearly identifying causality, incorporating the views of stakeholders, and clearly stating methodological limitations.

Resume

L'evaluation permet d'examiner systematique-ment le rendement des politiques relativement a l'atteinte des objectifs. Le principal objet de l'evaluation est de fournir de l'information qui permette d'ameliorer les programmes. Il existe de nombreuses approches a l'evaluation, chacune possedant ses propres forces et faiblesses et de nombreux types d'erreurs communes en matiere de methodologie devaluation. Les cles d'une evaluation reussie sont, notamment: l'utilisation de criteres d'evaluation complets, l'utilisation d'un echantillonnage suffisamment grand, l'evaluation de tous les aspects du programme sur une longue periode de temps, la comparaison du programme avec les alternatives realisables, l'identification precise de la causalite, l'integration du point de vue des intervenants et l'enonce precis des limites methodologiques.

Key Words

Policy evaluation, resource and environmental management

Introduction

Resource and environmental planning can be defined as a decision making process involving six steps (Figure 1). Evaluation plays a key role in two of these steps: evaluating options and evaluating outcomes. Given its importance in planning and policy-making, it is not surprising that evaluation is institutionalized as a legal requirement in many jurisdictions and has become a distinct field with its own body of theory and methodology.

The purpose of this volume of Environments is to explore recent developments in evaluation applied to resource and environmental planning through a series of case studies. In this introductory article we provide an overview of the field of evaluation and discuss some of the challenges in evaluating resource and environmental planning. We then review the contributions of the case studies to evaluation and identify keys to successful evaluation.

Evaluation Theory

In her landmark text on evaluation, Carol Weiss (1998: 4) provides a definition of evaluation that is commonly accepted in the evaluation literature. According to Weiss, evaluation is "the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy."

Weiss emphasizes the following four key attributes of evaluation in her definition.

* Evaluation is systematic, meaning it utilizes methods of analysis that meet rigorous scientific standards and produces results that can be replicated by other analysts.

* Evaluation focuses on the operation of a program and/or program outcomes.

* Evaluation assesses the degree to which the process and/or outcomes meet plan goals.

* The purpose of evaluation is to inform managers on the strengths and weaknesses of the program and identify ways in which the program can be improved and/or whether the program should be continued.

A wide variety of alternative approaches to evaluation exist (Table 1).

Internal versus External

The first decision in evaluation is whether the evaluation should be conducted internally within the organization that is managing the program or externally by an independent evaluator. Internal and external evaluations have their strengths and weaknesses (Clark and Dawson 1999, Weiss 1998). Internal evaluators may have a bias in favor of the program that they are managing, which reduces their likelihood of producing an accurate evaluation. This bias may be strategic, in that internal evaluators want the program to appear successful, or may simply arise because their thinking has been affected by close involvement with the development and implementation of the program. An advantage of using external evaluators is that they are less likely to have a bias in favor of the program. However, external evaluations also have potential weaknesses. External evaluators are less likely to understand the program and program goals. Results from external evaluations may also be less likely to be used by the program managers to improve the program because the external evaluators may not be trusted or believed, and in any event are not retained within the organization to assist in the implementation of recommendations. Some researchers suggest a way of achieving the combined benefits of external and internal evaluations is to use a hybrid approach, in which external and internal evaluators work together to assess the program (Suvedi and Morford 2003).

[FIGURE 1 OMITTED]

Purpose

A second decision is determining the purpose of evaluation. Four purposes are identified in the evaluation literature (Weiss 1998, Rossi et al. 2004). The most common purpose of evaluation is to assist in the improvement of a program by identifying strengths and weaknesses. This type of evaluation is referred to as formative evaluation. A second purpose is to determine whether the program is justified or not. This type of evaluation, normally referred to a summative evaluation, is often undertaken to fulfill accountability requirements for public spending. A third purpose is pursuit of general knowledge. This type of evaluation, referred to as theoretical evaluation, is normally undertaken by academics and is not linked directly or indirectly to program managers. A fourth purpose of evaluation may be to fulfill a hidden agenda such as justifying a decision that has already been made to eliminate a program or initiate a project. In these cases, the evaluation, which can be referred to as an ulterior evaluation, is managed to achieve a predetermined outcome.

Timing

A third decision in evaluation is timing. Evaluations can be done prior to policy or program development to assess options, during program operation, and/or after program completion. Evaluations can provide single snapshots of program performance or be ongoing over a longer time frame to analyze trends and foster continuous improvement. A comprehensive evaluation process should include evaluations at all key stages of a program's life (Weiss 1998). Another useful time-frame for evaluation is to compare the evaluation results after a program has been implemented (ex-post evaluation) with the forecast results used to select a program (ex-ante evaluation). Ex-post/ex-ante comparisons are especially useful for assessing accuracy of initial evaluation assessments and indicating how ex-ante evaluations can be improved. While ex-post/ex-ante evaluations are rarely undertaken, the limited studies that have been completed show a systematic optimistic bias in ex-ante evaluations that contributes to poor program selection and design (Gunton 2003).

What to Evaluate

The fourth decision in evaluation is deciding what component of the policy or program to evaluate. Evaluations can focus on one or more of five program components: program theory, which is the underlying logic of the program including the causal model of the problem it is intended to address, program design, program implementation, program outcomes, and program efficiency, which measures the outcomes per unit of resources required to operate the program. A comprehensive evaluation normally should include all five components in the evaluation (Weiss 1998).

Methodology

The fifth decision in evaluation is selecting a methodology. First, the evaluator must choose between quantitative and qualitative analysis. Quantitative analysis describes program performance in numerical terms that can be statistically analyzed. Qualitative analysis evaluates programs by observation, interviews, and document analysis and assesses performance in verbal as opposed to numerical terms. Within quantitative analysis there are also options. The preferred technique is randomized experiments. Randomized experiments attempt to identify program impacts by comparing groups or areas where the program is applied to a control group where all relevant variables other than those related to the program are held constant. Randomized experiments attempt to overcome one of the primary challenges in evaluation: distinguishing outcomes due to the program from outcomes due to other factors. The use of randomized experiments is constrained in resource and environmental management due to the complexity and diversity of the systems being evaluated and political and ethical considerations. There are, however, a range of alternatives available between pure experimental methods and pure qualitative methods (Clark and Dawson 1999).

Evaluation Criteria