Perceptions of organizational politics: a demonstration of the reliability generalization technique.

Although validity evidence in its many forms (e.g., content, construct, criterion-related) is an admittedly large part of the psychometric pie, reliability is also a critical piece (Nunnally and Bernstein, 1994). As stated by many psychometricians, "A test may be reliable without being valid, but it cannot be valid without being reliable" (e.g., Aiken and Groth-Marnat, 2006: 97). Therefore, reliability is a necessary condition for validity. There is widespread acknowledgement (Gronlund and Linn, 1990; Henson, 2001; Thompson, 1994; Vacha-Haase, 1998) that reliability is actually a feature of the scores obtained in a specific sample from the use of an instrument, and not a feature of the instrument itself. Specifically, reliability values (such as Cronbach's alpha) obtained using the same instrument may vary from sample to sample. In fact, the APA Task Force on Statistical Inference declared that "reliability is a property of the scores on a test for a particular [italics added] population of examinees" (Wilkinson and APA Task Force on Statistical Inference, 1999: 596). Since researchers have come to realize that "perfect reliability is only a handy fiction" (Nunnally, 1967: 218), a certain amount of measurement error is expected in the social and behavioral sciences. However, poor reliability can result in attenuated construct relationships, such as the zero correlation described by Reinhardt (1996) for scores that are perfectly unreliable on at least one variable. Other detrimental effects of poor reliability of scores include reduced statistical power (Onwuegbuzie and Daniel, 2004).

Because reliability is such a central issue in empirical research (e.g., Nunnally and Bernstein, 1994), many (e.g., Henson and Thompson, 2002; Vacha-Haase, 1998; Vacha-Haase et al., 2002) have argued for an increased focus on the topic of reliability in the form of reliability generalization (RG) studies. The purposes of this study were, first and foremost, to demonstrate the technique of reliability generalization so that other management researchers might use it, and secondly to provide researchers investigating perceptions of organizational politics with information that allows them to understand: (1) the typical reliability of scores on the Perceptions of Organizational Politics Scale (POPS) so they can place their own reliability measures in context, (2) the amount of variability in reliability coefficients so they can understand the robustness of reliability estimates, and (3) the sample and/or test elements that systematically contribute to variations in reliability so they can understand how generalizable measures of political perceptions are across diverse samples. Politics are sometimes described as dysfunctional aspects of organization life whereby employees compete for scarce resources usually at the expense of each other (Kacmar and Baron, 1999). The perception of organization-level politics implies that organizational members view life in the organization as tainted by behavior rooted in self-interest (Ferris et al., 1989). Thus, the measurement of phenomena like politics in the workplace is of potential value to organizational intervention efforts and organizational development. To accomplish these goals, this study examines factors affecting variance in score reliability across studies that employ measures of perceptions of organizational politics, using reliability generalization, a meta-analytic framework described by Henson and Thompson (2002), Vacha-Haase (1998), and Vacha-Haase et al. (2002).

RELIABILITY GENERALIZATION

Reliability generalization as an analytic tool was first described by Kennedy and Turnage (1991) as an extension of validity generalization (Hunter and Schmidt, 1990; Schmidt and Hunter, 1977; Schmidt et al., 1985). However, the technical sophistication of reliability generalization was greatly expanded upon by Vacha-Haase (1998) as she described reliability generalization as an analytic method of characterizing mean measurement error variance and the sources of variability in scores. Sources of variability include aspects of the measurement instrument itself (or its administration) and characteristics of the sample. Moreover, Hen son and Thompson suggest that researchers can "tailor their studies to maximize reliability" (2002: 114) through the choice of scale, method of administration, and management of sample characteristics. However, Vacha-Haase et al. (1999) and Yin and Fan (2000) found that many researchers tend not to report reliability of scores on their focal variables. Instead, they rely on reliability induction (Vacha-Haase et al., 2000), whereby the study authors report the reliability of scores obtained in previous studies or test manuals rather than the reliability of scores obtained from their own sample. For example, Beretvas et al. (2002) found that less than nine percent of articles using the Marlowe-Crowne Social Desirability Scale reported reliability scores for their own samples. The use of reliability induction suggests an overconfidence in the possibility of scales resulting in scores showing acceptable reliability, as well as, perhaps, an ignorance that reliability is not associated with a particular scale, but rather with the scores on the scale (Gronlund and Linn, 1990; Pedhazur and Schmelkin, 1991; Thompson, 1994; Vacha-Haase, 1998).

For study authors who do report reliability scores for their samples, Vacha-Haase (1998) suggests that the reliability generalization analytic technique can explore the variables contributing to score variation in reported reliabilities. Such variables include sample characteristics such as sample size, gender, age, ethnicity, and other features of the sample from which scores are obtained. Other variables include aspects of the administration of the scale or characteristics of the scale itself such as the type of scale used (e.g., Guttman, Likert) and the number of items on the scale, especially if alternative forms are used. Most reliability generalization studies have used multiple regression to analyze the variance accounted for and the significance of each independent variable in the prediction of reliability values across studies, though other techniques since Vacha-Haase (1998) have been introduced (e.g., Beretvas and Pastor, 2003).

To our knowledge no one has evaluated the ability of POPS to yield reliable scores across situations and samples. Though many researchers have reported reliability estimates for scores obtained using their specific samples, no one has assessed the typical score reliability (Vacha-Haase, 1998) for POPS across multiple samples or the extent of variability of scores across different samples using POPS. It is possible that POPS produces more or less reliable assessments of perceptions of organizational politics within certain situations or with specific sample characteristics. Moreover, if the reliability of scores is related to certain situational characteristics, one can only assume that it would be valuable to determine the contributing factors to the variability of reliability values. Given the proliferation of interest in perceptions of organizational politics as a research area, the psychometric concerns of validity and reliability are imperative to explore. Since POPS in its many forms is the de facto standard of measurement for political perceptions, this study undertakes an analysis of the scale using the reliability generalization technique. Essentially, a reliability generalization study identifies the typical score reliability for a given measure across multiple samples, the variability of score reliabilities across multiple samples, and what factors contribute to the score reliability. With this framework in mind, an overview of the various measures of POPS is important to understand.

DEFINING AND MEASURING PERCEPTIONS OF ORGANIZATIONAL POLITICS

Politics in organizations are generally regarded as pervasive, necessary for normal business functioning, and a simple fact of organizational life (Ferris et al., 1996b; Greenberg and Baron, 1995; Pfeffer, 1981; Pinto, 1997; Vigoda-Gadot et al., 2003; Williams and Dutton, 2000). Organizational politics are ubiquitous and considered necessary for normal business functioning (Pfeffer, 1981; Pinto, 1997); however, in the scientific literature there are differing notions (e.g., Allen et al., 1979; Bacharach and Lawler, 1980; Mintzberg, 1983; Pettigrew, 1973; Porter et al., 1981; Tushman, 1977) of what constitutes organizational politics. Most organizational researchers agree that perceived organizational politics may be described as the perception of intentional actions, sometimes performed at the expense of others, that are either covertly or overtly performed in an effort to advance one's position (Allen et al., 1979; Andrews and Kacmar, 2001; Ferris and Kacmar, 1992; Kacmar and Baron, 1999; Kacmar and Ferris, 1991). Ferris et al., (1989) suggest that organizational politics is a social influence process in which behavior is strategically designed to maximize long-term and short-term self-interest, that is either consistent with or in opposition to others' interests. Such self-interest maximization includes the prevention of negative outcomes and the attainment of positive outcomes. Adams et al. (2002) suggest that even though political behavior may have positive outcomes, employees' perceptions of politics are nearly always negative.

In an effort to understand the nature of perceptions of organizational politics in the workplace, Kacmar and Ferris (1991) published a measure known as POPS. Its popularity and promising empirical support have generated many evaluations of the components of the theoretical model of organizational politics put forth by Ferris et al. (1989), providing valuable insight into the antecedents and consequences of politics in organizational settings. Ample empirical evidence exists that politics can be conceived of negatively and that political perceptions have moderately strong relationships with key workplace outcomes (Miller et al., 2008). To understand the politics perceptions construct, a basic understanding of the development of instruments used to measure it is helpful.

Print Get the Mag Weekly Updates