SME survey methodology: response rates, data quality, and cost effectiveness.

Cost effective data collection is an important methodological issue for small and medium enterprise (SME) researchers. There is a generally held view that mail surveys are the most efficient means of collecting empirical data, despite the potential difficulties associated with low response rates. To enhance the usefulness of mail surveys, researchers have suggested a variety of strategies aimed at improving response rates. While previous studies have examined the effect on response rates of many of these strategies, their impact on data quality and on the cost effectiveness of data collection is less well understood. This study evaluates four response-inducing strategies (printing the survey instrument on colored paper, telephone pre-notification, payment of a monetary incentive, and a follow-up mailing) in terms of their effect on data quality, response rates, and cost effectiveness for a population of SMEs.

Introduction

Small and medium enterprise (SME) researchers are frequently confronted by the need to collect substantial quantities of empirical data, often from a reluctant population. Despite potential difficulties with non-response bias, mail surveys are commonly used for such purposes. For example, an examination of Entrepreneurship Theory and Practice, the International Small Business Journal, the Journal of Business Venturing, and the Journal of Small Business Management from 1991 through 2000 revealed that one in three published articles were based on a mail survey.

A myriad of strategies have been suggested to stimulate questionnaire returns and increase external validity. While the impact on response rates for these various strategies has been reasonably well researched and documented in broader consumer and industrial populations (Yammarino, Skinner, & Childers, 1991), their influence on data quality and the cost effectiveness of data collection has seldom been examined (Fox, Robinson, & Boardley, 1998). This is particularly true for the narrower SME population where there have been relatively few studies that have tested the impact of response-inducing strategies on response rates, let alone on data quality and cost effectiveness. This lack of research within the SME setting is quite surprising given Turley's (1999) view that these effects may be population specific, and that the rationale for studying SMEs is that they are expected to be different (Welsh & White, 1981).

This article makes an important contribution to the SME literature, and possibly the broader consumer and industrial literature, by evaluating the impact of a variety of suggested mail survey response-inducing strategies on response rates, the quality of questionnaires returned, and the cost effectiveness of the process. We hope our findings will help guide future SME researchers intending to use mail surveys.

The next section provides a brief overview of the literature concerning response-inducing strategies. This is followed by a discussion of the methodology employed in this article. Next we present and discuss our results. We conclude with a summary of our findings and some suggestions for future SME researchers intending to use a mail survey.

Literature Review

Given the paucity of research on the effect of response-inducing strategies for SME surveys, this review relies primarily on studies of the broader consumer and industrial populations. We have classified the strategies into four types: attention-seeking, questionnaire completion, questionnaire return, and incentives.

Attention-seeking

Various attention-seeking strategies have been designed to help the survey instrument pass through the recipient's "attention filter" (Diamantopoulos, Schlegelmilch, & Webb, 1991). Of these, telephone pre-notification has been found to consistently increase response rates (Schlegelmilch & Diamantopoulos, 1991); but there have been mixed results for pre-notification by letter (Albaum & Strandskov, 1989; Duhan & Wilson, 1990) and by postcard (Murphy, Dalenberg, & Daley, 1990; Murphy, Daley, & Dalenberg, 1991). Similarly, personalization strategies have also generated mixed findings (Kanso, 2000; Martin et al., 1989). None of these strategies appear to have resulted in any statistically significant differences in data quality (as measured by item omissions, see Duhan & Wilson, 1990; Jobber, Allen, & Oakland, 1985; Murphy et al., 1991; Wunder & Wynn, 1988).

Postcard pre-notification appears to be more cost effective than either telephone pre-notification (McLaren & Shelley, 2000) or a no pre-notification control (Murphy et al., 1991). Using a letter to pre-notify has, generally, been more costly than no prenotification (Albaum & Strandskov, 1989), but has been cost effective compared to a commitment card (Duhan & Wilson, 1990). The use of telephone pre-notification has not been found to be cost effective compared to a no pre-notification control (Mitchell & Nugent, 1991).

Questionnaire Completion

There have also been strategies formulated to promote questionnaire completion. In general, university sponsorship has been found to increase response rates without affecting item omissions, whether compared to a "no sponsorship" group (Greer & Lohtia, 1994); a commercial operator (Schneider & Johnson, 1995); or a research oriented firm (Faria & Dickinson, 1992). Assurances of anonymity have generally proved successful in terms of response rates (Kanso, 2000; Tyagi, 1989), but do not seem to have increased data quality (Albaum, 1987). Mail surveys with colored print on colored paper have been found to increase response rates (LaGarce & Kuhn, 1995), but this has not extended to the use of colored paper alone (Buttle & Thomas, 1997). Using double-sided rather than single-sided, or short rather than long questionnaires (Jobber, 1989), appears to have had limited impact on response rates. Producing the survey instrument in a booklet format does not appear to affect the number of questionnaire items left unanswered (Jobber, Birro, & Sanderson, 1988).

Questionnaire Return

Follow-up techniques designed to encourage the return of completed questionnaires have usually proved successful. The combination of mail and telephone reminders in Dillman's (1978) Total Design Method (TDM) appears pre-eminent (Paxson, 1992). However, many surveys have employed a modified "reduced reminder" form of the TDM, consistent with Westhead and Cowling's (1998) finding that the statistical significance of any follow-up comes from the first reminder only. The effectiveness of single follow-ups in significantly increasing response rates is also supported outside of the TDM, whether it be via a letter with questionnaire (Jobber & Sanderson, 1983), a letter only (Martin et al., 1989), or a postcard (Kanso, 2000). Although prior research suggests that Dillman's (1978) TDM is cost effective (Paxson, 1992), we found no studies that addressed data quality.

Incentives

Prepaid monetary incentives, both nominal (Angur & Nataraajan, 1995) and substantial (James & Bolstein, 1992), appear to increase response rates. By way of contrast, the promise (rather than pre-payment) of a monetary incentive to participants has generally not been effective, even when the offered amount is substantial (James & Bolstein, 1992). However, when the promised payment is to others (a charity, for example), small value incentives seem to be helpful (Faria & Dickinson, 1992). The chance to win a substantial monetary prize appears to have a mixed impact on response rates (Angur & Nataraajan, 1995; Paolillo & Lorenzi, 1984).

Where there was a promised monetary incentive of $10, Goetz, Tyler and Cook (1984) reported a significant reduction in item omissions and a significant increase in the length of text answers. James and Bolstein (1990) found that small pre-paid incentives significantly increased the length of text answers, but did not affect item omissions. Pre-paid monetary incentives have proved cost effective in some studies (James & Bolstein, 1992), but this has not generally been the case (Brennan, 1992). The cost ineffectiveness of monetary incentives has been especially prevalent where the incentive was promised rather than pre-paid (Dommeyer, 1988).

In terms of non-monetary incentives, gifts of low value appear to have increased response rates (Hansen, Tinney, & Rudelius, 1983). However, the offer (or promise) of results does not appear to increase response rates (McCline, Bhat, & Baj, 2000; Morris & Jones, 1999). The chance to win a substantial non-monetary prize (London & Dommeyer, 1990), or the promise of survey results (Dommeyer, 1985), does not appear to have affected item omissions. We are not aware of any study that has assessed the cost effectiveness of non-monetary incentives.