Disaggregating education production.(ORIGINAL PAPER)

Introduction

A particular weakness of the education production function literature is that production is measured after a series of treatments and reflects aggregation across treatments. The literature has generally utilized data based on student performance over an entire course. Studies of this nature involve aggregation across the stages of production. In a large lecture course, the stages of production are traditionally the reading assignment, lecture, follow up study and test preparation. The benefit of disaggregation across these treatments is the ability to identify which treatments are effective and which are not. This study meets a need to extend the perspective of this literature to a disaggregated view.

In the next section, we will discuss the literature relevant to our investigation. We will draw from previous studies a consensus involving three constructions of the dependent variable and the predictors that are used to estimate production. We will explain the data and provide some analysis of the descriptive statistics. We will also provide a profile of the multistage production process through the distributions of scores produced across stages. Finally, we present the empirical estimation of multistage education production functions.

Review of the Literature

The education production function literature falls into two distinct groups, those that use a standard group of independent variables and have a more general focus and those focused more narrowly on a unique experimental variable. Studies of the latter type include Borg and Shapiro (1996), who focused on student personality type, Cohn et al. (1995), who focused on measuring working memory, Anderson et al. (1994), who focused on a myriad of detailed academic indicators, Charkins et al. (1985), who focused on teaching styles versus learning styles and Fraas (1982), who focused on instruction using simulations. Because of their limited application, we have excluded these studies from our focus. We also excluded other studies that did not employ all the independent variables we deemed to be fundamental to the consensus. These included Douglas and Sulock (1995) (GPA and ACT were not included.), Bonello et al. (1984) (ACT was not included.) and Crowley and Wilton (1974) (GPA and ACT were not included.).

Using the criteria noted above, we focus on 16 studies in this literature. The majority of these studies use some form of a pretest/post-test framework. (1) The consensus independent variables include pre-test score, ACT score, cumulative GPA and some measure of student effort. We will discuss each of these in turn.

The pretest score as a regressor is often included as a potential measure of human capital acquired by the student since beginning university study. Including pretest scores will also allow for depreciation of knowledge between the pretest and the posttest. O'Neill (2001), Grimes and Nelson (1998), Watts and Bosshardt (1991), Schmidt (1983), Manahan (1983), Miller (1982), Prince et al. (1981), Kelly (1975) and Buckles and McMahon (1971) all include a pretest score.

The ACT score represents the student's aptitude upon entering university study. It is universally viewed as a measure of human capital accumulated up to that point. The student's accumulation of human capital subsequent to entering university study is the omission implicit in measuring human capital exclusively by the ACT score. This is a rationale for including GPA and/or the pretest score as a supplementary measure of human capital.

GPA does not enter the regressor lists of 8 of the 16 studies noted. Studies by Watts and Bosshardt (1991), Schmidt (1983), Manahan (1983), Miller (1982), Prince et al. (1981), Soper and Thornton (1976) and Kelly (1975) omit this regressor. There is a concern that GPA may be collinear with ACT and that the student's average grade is a predictor of the subsequent grades that a student will earn. This strong relationship exists outside of the theoretical construct of education production. Therefore, GPA may be multicolinear with the entire regressor list as a general structure generating all of the student's grades. This may disrupt the capacity of regression analysis to capture the effects of the other regressors. In this study, we will take precautions on this point by performing our analysis both with and without GPA.

Effort is the labor variable in an education production function. In 9 of the 16 studies noted, there is no measure of labor inputs. Omission of an effort measure seems to have been a matter of available data rather than any doubt about the desirability of such a measure. Limited data availability has also led to use of the McKenzie and Staaf (1974) measure of effort in some of the literature. This construction involves a juxtaposition of the concepts of achievement and aptitude. Dividing some measure of achievement by some measure of aptitude generates an effort measure. In general, the determinations of the measures of achievement and aptitude used to form the McKenzie-Staaf variable have been a matter of available data and researcher choice. Studies by Krohn and O'Connor (2005), Okpala et al. (2000), Borg et al. (1989), and Schmidt (1983) measure effort using this approach. Only studies by Miller (1982), and Buckles and McMahon (1971) use student time-on-task, the most direct measure of effort.

We measure effort with student time on task. A distinct advantage of the data collection process used for this study of disaggregated production is that it also provides an opportunity to collect effort data at each stage of production. This should provide for a much more accurate measure of effort and thus more accurate results in estimating the impact of effort on production.

Description of the Experimental Model

The consensus of the literature utilizes three fundamental specifications of the education production function. The score on the posttest (S1) is one of three fundamental specifications of the dependant variable in an education production function. The other two are referred to as gain (S1-S0) and gap ([S1-S0]/[I-S0]). We will use each of these three specifications in turn. It was noted that the regressors will be pretest score (S0), ACT and eflbrt with and without GPA as an additional regressor. Also included in our estimations will be a dummy variable controlling for instructor differences.

Because the student does not enter the production stage tableau rosa, gain rather than score represents the outcome of production. Let -dS0 represent the loss from depreciation of pretest cognition that takes place over the horizon of the production stage. Gain measured as net gain is given by equation one.

1. Gain = -dS0 + [a.sub.0] + [a.sub.1]ACT + [a.sub.2]GPA + [a.sub.3]instructor + [a.sub.4]effort + [epsilon]. The score version of the model, equation two, would be trivial except that it may better predict a student's score. The gap version of the model is shown as equation three.

2. Score = (1-d)S0 + [a.sub.0] + [a.sub.1]ACT + [a.sub.2]GPA + [a.sub.3]instructor + [a.sub.4]effort + [epsilon]

3. Gap = -dS0 + [a.sub.0] + [a.sub.1]ACT + [a.sub.2]GPA + [a.sub.3]instructor + [a.sub.4]effort + [epsilon]

Data and Descriptive Findings

The empirical testing of all the models presented in the previous section utilizes student quiz data and survey data collected in macroeconomic principles classes at a public university in Kentucky. Data collection took place over three semesters spanning from the spring semester of 2003 to the spring semester of 2004. The two instructors administered the same examinations, spent the same amount of class time covering material and utilized the same textbook. (2) Some data points were lost due to the failure of the student to take one of the quizzes or because the student's transcript information was unavailable.

Our goal was to acquire separate data on each of the stages of production: reading, lecture, post lecture study and test preparation. We gave the students five quizzes: a pretest, a reading quiz, a lecture quiz, a next meeting quiz and a posttest. This five-way treatment was repeated over four topics. These data were then pooled across topics for each stage. Consequently, each stage of production was bracketed by a preceding quiz, that stage's pretest, and the immediately subsequent quiz, that stage's posttest.

Table 1 reports scoring statistics for each stage in the production process. The means, gains and standard deviations for student scores as a group are shown in the upper part of the table. The distributions of scores and gains by student percentiles are shown in the lower part of the table. Distributional effects are outside of our principle inquiry. Nonetheless, they are part of understanding how scoring varies across the stages of production and are of acute interest.

The average pretest score for all students was .44. Although some student's may have had previous coursework that contributed to this average, the pretest score indicated student performance prior to any educational treatments in the concurrent semester. The gain from the reading assignment (.21) was much larger than at any other stage of production. The gain from lecture was second largest at .12. We note that the reading assignment was the first contact treatment for most students and consequently might be expected to produce the largest gain. The average student score droped from a high of .77 immediately following the lecture to .75 at the next class meeting as a result of follow up study. A greater drop in student performance from the .75 average to an average of .71 occurred in the test preparation stage of production. Therefore, two of the treatments produced gains and the other two treatments produced losses for the average student.

Print Get the Mag Weekly Updates