Methodological frontiers of public finance field experiments.

INTRODUCTION

An economics graduate student starting her dissertation today would do well to follow the example of Heather Ross and think big. After working in 1965 and 1966 as an intern with the President's Council of Economic Advisers and the U.S. Department of Health, Education and Welfare, Ross became interested in income maintenance policy, particularly in how the behavior of low-income people might respond to transfer payments. In 1966, Ross, then a visiting research fellow at the Brookings Institution, wrote a paper entitled A Proposal for Demonstration of New Techniques in Income Maintenance, which was subsequently submitted to the Johnson Administration's anti-poverty agency, the Office for Economic Opportunity. The experiment that resulted from this proposal--the New Jersey Income Maintenance experiment--is generally considered to be the first large-scale social experiment ever conducted. Ross's thesis proposal instigated a project that cost over $30 million in 2006 dollars, and sparked a wave of large-scale experiments in health, housing, and welfare. (1)

Over the past 40 years, many of the essential goals of public finance economists in assessing policy interventions have remained the same as they were for Ross. We would like to have a credible strategy for identifying the causal effect of the intervention, and we would like to have data about situations that are as similar as possible to that in which the policy would occur in the future. In the case of income maintenance programs, there was little data in 1966 that could be used to meaningfully forecast the potential effects of the negative income tax proposals that were being made. The compelling logic of Ross's proposal was that in addition to collecting contextually relevant data, one could use the power of random assignment in an experimental framework to assess how otherwise similar groups of individuals responded when assigned to different policy interventions. The difference in outcomes between an intervention group and a comparison group in a social experiment could be used as an estimate of the effect of the intervention in a setting similar to that in which a policy intervention might be enacted.

The purpose of this article is to demonstrate how a rich array of methods can be applied to increase the relevance of field experiments in public economics. Given that the essence of analyzing experiments typically involves looking at the difference between two sample means, one might ask whether experiments involve more than rudimentary research methods, and even whether there are any important methodological frontiers in this area. I focus on advances in the research methods used in conducting field experiments, using examples from government taxation and expenditure, public goods, and externalities. The examples are intended to illustrate methods with general applicability, and this is not an exhaustive survey of particular studies. In this article I assume that the reader is familiar with the basics of how to design an experiment and the conditions under which it identifies the causal effect of a policy intervention, as discussed by Orr (1999) and others. The focus here is on research in North America. (2) To emphasize frontiers of research, the examples are primarily drawn from studies using data collected within the past few years.

To characterize the public finance field experiments I discuss, the circumstances are generally intended to be close to an actual application of a policy intervention as possible--although some are clearly closer than others. (3) In focusing on public finance field experiments, I also adopt criteria delineated by Greenberg and Shroder (2004) for identifying social experiments, which are: random assignment, policy intervention, follow-up data collection, and analysis. I use the term "field experiments" here mainly because some authors, including Ferber and Hirsch (1978), have defined "social experiments" as those being conducted with government funds or by government agencies. For the purposes of this article, I will consider all social experiments to be public finance field experiments.

In this article I will not focus on the lengthy and informative debate over the value of public finance field experiments relative to other types of research. Many of the potential issues were identified early on by Rivlin (1974) and others. For instance, an experiment may not be a good estimate of the effects of a policy if people behave differently when they know they are being observed, if they believe the experimental intervention is temporary but the policy would be permanent, or if the policy created spillovers when implemented at scale that were not present in the experiment. Each experiment attempts to minimize the impacts of such issues, and naturally there is variation in the success of these attempts. The issues have been ably discussed at length by Burtless (1995), Heckman and Smith (1995), Harrison and List (2004), Moffitt (2004) and others. Rather than reviewing this terrain again, I will start from the premise that most experiments have value and focus on methodological approaches to making public finance field experiments more useful. (4)

This article is organized around the sequence of activities involved in conducting an experiment and discusses related methodological innovations. First, topics are selected. Second, a conceptual framework is developed that motivates hypotheses about the potential impacts of interventions. Third, a field experiment is designed. Fourth, the experiment is implemented and data is collected. Fifth, data is analyzed. Lastly, results are interpreted.

TOPICS

In selecting a specific topic for study, there will always be value in working in areas where there is interest in taking action based on results, and in areas that are understudied. Sometimes it will make sense to go directly from idea to field experiment. As complementary approaches, ideas for the topics of study for field experiments may be generated from earlier stages involving exploratory research methods that can be implemented more quickly and less expensively. The value of field experiments can be enhanced by imbedding them in a program of related research, so that the experimental work can interact synergistically with research of scholars involved and with the broader literature in order to build a deep, coherent, synthesized body of knowledge. This section illustrates the methodological technique of building field experiments at early stages that use two different approaches: non-experimental research and laboratory research.

Building on Non-Experimental Research

Analyses of participation in a Tax Deferred Account offer an example of entrepreneurship in collecting new data and of creating a program of research building directly on earlier non-experimental research. This line of research focused on analysis of the decision to enroll in a savings plan by university staff, and an attempt was made to study the spillovers from colleagues in the same department on individual decisions. Duflo and Saez (2002) used an instrumental variables strategy for identification, assuming that average wage or tenure in a department did not directly affect individual decisions (conditional on individual wages or tenure), in order to use average wages or tenure as instruments for average participation. They later followed up on this initial work to implement an experiment in which some individuals in some departments received incentives to visit a benefit fair to learn more about the savings plan. The effects of spillovers could be assessed by comparing the outcomes of the non-incentivized individuals in incentivized departments to individuals in non-incentivized departments. In their analysis of the experiment, Duflo and Saez (2003) found that the estimates of the social effect were smaller in the experiment than in the preceding work using instrumental variables, but that small changes in the environment induced by the experiment did have a noticeable impact on plan participation.

Building on Laboratory Research

Studies of matching and rebate subsidies for charitable contributions have been the subject of a sustained program of research in both the laboratory and the field. The U.S. tax code provides a form of a rebate for charitable contributions by making them tax-deductible. As an example of economic equivalence, a 20 percent rebate on one dollar of contribution means that the gift would cost the donor 80 cents. An 80 cent donation with a 25 percent match would also cost the donor 80 cents and raise one dollar for the charity. The economically equivalent rebate may not be as successful as a match in encouraging charitable giving--perhaps because the rebate is more complicated, less certain to be given, or just that the rate associated with the rebate (20 percent) is lower than the match (25 percent). Eckel and Grossman built on their earlier laboratory experiments (2005) examining this issue by conducting field experiments (2006a, 2006b) with two different charities in Minnesota where potential donors were offered rebates or matches. Although each experiment had complications, the preponderance of the evidence was that the matches consistently raised more funds for the charities in both the laboratory and field experiments. Thus, it may be possible for the government to implement a match received directly by charities having the same overall budgetary cost as the current deductions, but resulting in greater donations. (5)

Summary