How can we tell if any of this actually has made any difference and what kind of guidance would this experience provide from the point of view of developing a de novo or revitalised SME policy? There are general things one can ask, such as 'is the SME sector any more dynamic, innovative, or likely to train or export more now than it was, say, at the beginning of the 1990's or 1980's when this policy band wagon began to roll?' Or you can ask questions about the impact of specific programmes. I want to do a bit of both.

First I am going to talk about evaluating particular projects, and I'm going to do this with reference to a particular R&D program, this is the Small Firms Merit Award for Research and Technology (SMART). It is one of the longest running UK programmes. It began in 1988 and it has had quite a long history of evaluation. I was involved in one of these evaluations and I want to talk about that because it exposed a number of the problems that arise in thinking through whether a scheme like this actually works.

The SMART scheme had a number of components essentially involved with encouraging small firms to advance from R&D to the commercialisation of projects. The biggest parts of the budget we evaluated were concerned with feasibility studies to convert R&D programmes into products and development projects to take things through from R&D to development stages. The rationale for this programme is the typical, well established market failure argument that small firms have difficulty in getting access to high risk venture capital and consequently need support, in this case by awarding financial grants. The SMART scheme has disbursed over 250 million [pounds sterling] to over 5000 firms since 1988.

The major issue with evaluating a program like this is that you do not know what the firms would have done if they had not received a grant. This is the problem of establishing a counterfactual position to assess performance. One approach is to form a control group of similar firms who did not get an award. This is complicated, however, by the possibility that firms that get into a scheme are quite likely to have characteristics that would have made them do well in any case and these characteristics may be too numerous to control for them all by matching. A multivariate alternative is to carry out a selection regression model to identify these factors and correct for this selection bias when explaining the impact of participation in the scheme on performance. Another approach is to ask the people themselves to perform a subjective counterfactual assessment. In principle this provides a much more sophisticated counterfactual because the individuals can take into account everything that they think relevant. Basically scheme participants express their view of what would have happened if they had not got the award.

In the SMART evaluation we did matched control groups, we did selection modelling, and we did subjective counterfactuals and case studies. But in setting up this investigation, and this is a very important problem from the point of view of business policy design, there was a lack of comprehensive contextual information about the firms. The amount of information they were required to give about their prior performance and their subsequent performance was limited. This is a classic problem facing those trying to evaluate a scheme. It involves going back and creating data whereas it should be--in my view--a simple requirement that there be minimum reporting requirements before a firm receives a benefit, and monitoring afterwards. The information required should be specifically designed to allow an effective assessment of scheme objectives

There are also some particular problems which arise when dealing with an innovation programme. All kinds of things impinge on small business performance so trying to establish causal linkages around innovation is a little like searching for a needle in a haystack. Moreover, enterprise innovation and growth performance is highly skewed. As a result I would argue that public policy administrators have to behave like venture capitalists. This means not just focussing on average point outcomes but also looking at their distribution. There will be many failures and a few big winners.

Finally, it is important to note that typically firms have multiple objectives in seeking support through a scheme such as SMART, not all of which are necessarily always the same as the scheme's objectives. We found that the vast majority of firms that went into the scheme did not always focus on finance alone. However a majority reported that, in their view, most of their multiple objectives were met by the scheme. In addition their subjective estimates of additionality were also positive in the sense that the firm's project, or programme, would not have gone ahead without the scheme or, if it had gone ahead, it would have been much later and much smaller. These subjective assessments of project additionality are positive but we sought to identify evidence of effects in terms of growth in productivity and exports.

When we looked at these variables before and after performance of the grant, for recipients and for the control group who didn't get awards, the recipients looked much better. But this conclusion is potentially contaminated by sample selection biases, basically because the grantee firms are better firms before the award than the control group firms. When we corrected the results for this bias we got positive but insignificant impacts.

Thus most of the results shown in Figure 5 are positive but they are rarely significant. This result matches what the subjective counterfactual analysis told us in the sense that 57% of firms said the grant had no impact on their turnover, 70% reported no impact on exports, 64% reported no impact on employment and 46% reported no impact on profitability.

Now if you assess a scheme on the basis of this average impact you would have to say this scheme was not very successful. But that conclusion would be deeply misleading because this is an innovation scheme. What you have to look at is the total output from the scheme rather than estimates like these that look at individual units and just think about them as average effects. The same problem arises with the econometric analysis, which misses the enormous skewness in the outcomes. In actuality, 5% of the growing firms accounted for over half of all the sales generated by this SMART scheme, and the top 20% of firms accounted for 80% of sales, so the outcomes show a Pareto distribution (see Figure 6).

Only a small number of firms account for this massive increase in the proportion of sales. This means that we have to do a different sort of calculation. That is, we should aggregate outcomes across the scheme as a whole and look at what the total amount of improved activity has been and recognise that the typical individual firm result may be no effect. If we treat the outcomes as a group effect and relate the aggregate performance to the expenditures on the scheme, then this scheme looks better value for money. The important lessons here are that first of all you have to correct for selection bias, and then you have to allow for the skewness in the outcomes in evaluating the scheme. On this basis this scheme was successful.

Viewing it in this way is like treating the scheme as a venture capital portfolio. One corollary of a venture capital approach is to tie progressive grant instalments to milestones. I think we should build staged awards into innovation programmes. And, finally, it is important to make sure that you build into the design of schemes the provision of the information you know you are going to require when you want to evaluate it.

[FIGURE 6 OMITTED]

I will turn now from individual programmes to the consideration of aggregate economic effects. These are also inherently difficult to assess since it is very difficult to construct counterfactuals about what the SME sector would look like now if there had been no programme of policy intervention. But we can ask some broad questions about what has happened to the size, scale, shape and the performance of firms in this sector. In doing this I am going to refer to some official data about trends in SME activity over time (see Figure 7). This is the UK business stock from 1984, the days of Lord Young and the conversion to DTI from the Department of Enterprise, through the Lawson Boom and then to the 1990's. I think it's a very striking picture because it shows that the UK business stock now is not at the levels achieved at the end of the Lawson Boom.

[FIGURE 7 OMITTED]

This was an increase in business formation linked to house price rises which increased collateral for loans and made borrowing easier. The small business sector peaked in numbers in 1991. Then it collapsed, followed by a more gradual recovery. Most of the government expenditures in support programmes have accrued in this post-peak period and they have produced a slow increase in the business stock, but nothing like the earlier peaks.

This growth in the number of firms since the early 1990's is almost entirely driven by firms in finance, property and business services; manufacturing businesses have slowly declined. The increase has mostly occurred in the very tiny firms--it has occurred in the firms employing between one and ten people. Most of the rest of the SME sector, the 10-249 employee firms, have small shares in the growth. So what has happened in aggregate terms is a massive increase in the micro sector.

Now I want to say something performance. At the Centre for Business Research at Cambridge we have carried out seven surveys, one in each of the years where there is a grey bar in Figure 8.

We have made sure we have asked similar questions over time so this enables us to take matched groups of firms over these surveys, matched by size, age and sector, and to compare answers to a variety of questions about firm performance. I am going to compare 1991, 1997 and the latest survey for 2004. The overall time series survey sample comprises about 3,400 firms, with a sample of over 1,000 firms drawn from each of the 1991, 1997 and 2004 surveys, all matched in terms of size, age and industrial sector. In comparing these years it is important to note the major changes in monetary conditions since 1991 shown in the diagram above. It is likely that answers to questions about access to finance will be influenced by these macro conditions, and reported improvements in access to finance since then may reflect these macro changes rather than the effects of support policies.

Print Get the Mag Weekly Updates