Mobile Edition
Print
Get the Mag
Weekly UpdatesIt is almost a given that some jobs have the potential to change rapidly over time. Some of these changes are the result of new or developing technology. That is, technology appears to be changing at ever-increasing rates as new materials are being used in production settings, information is being generated and managed in new ways via computerization, and so forth. Clearly, these changes in technology influence applied psychology and human resource management systems. (1, 2) Jobs also may be changing given an increased emphasis on teamwork in organizations, (3) empowerment of employees, (4) or other managerial interventions such as quality management systems. (5)
These changes and interventions may subsequently influence the nature of the knowledges, skills or abilities involved at work. In all of these instances, researchers and human resource practitioners are often faced with concerns about the currency (or "up-to-dateness") of their existing job analyses.
However, there is a marked contrast between what is written about the dynamic nature of organizations and the literature addressing changes in job analysis. That is, there is a substantial body of literature suggesting that some jobs might be changing in significant ways, but little guidance on how to cope with such changes when examining or using job analyses.
It is also clear that job analysis plays a pivotal role when designing organizational interventions, such as selection systems. (6) In turn, test developers and users sometimes need to confirm that the underlying job analysis information is still current. For example, suppose a thorough job analysis took place several years ago, but a new selection system is being designed and there is an underlying concern that the job may have changed in some manner. The up-to-dateness (currency) of the job analysis information is important to organizations for at least two sets of reasons. First, organizations desire to maximize the effectiveness of their organizational interventions and processes. For example, valid selection systems that identify the best possible workers might incorporate an up-to-date understanding of the job, as such information could influence the content validity of the exam. A second, related reason that currency is important might occur if an organization incurs legal action against one or more of its human resources systems. For example, a job analysis is often used as a foundation for establishing the validity (and particularly content validity) of a personnel selection system. The nature of the job analysis could influence how convincingly the system is presented to regulatory agencies and defended in court. (7) We therefore also examined legally oriented documents for delineation of job analysis currency methods.
The purpose of this article is to review existing procedures and/or writings and then to present the development and implementation of a systematic protocol for assessing the "up-to-dateness" (currency) of job analysis information collected at some prior point in time. The protocol also provides a systematic way of identifying related changes, if any occurred. As will be noted later, there are clear variations to the method that might be viable in certain situations, but it is believed that this protocol provides an important first step in initiating common, systematic procedures and methods for checking on currency.
Prior Literature and Professional Standards
Prior literature. As noted, the literature on the currency of job analysis information is quite scant. Reviews of job analysis techniques typically do not even mention when, or how, to check for currency of job analytic information. (8, 9,10,11,12)
A few sources briefly mention job analysis currency issues. Brannick and Levine (13) indicate it is important to check that job analyses are not "stale." Morgeson and Campion (14) indicate that accuracy of job analyses may be difficult to assess if jobs change over time. Sackett and Laczo (15) mention job analyses for "future" jobs and suggest asking subject matter experts (SMEs) to "think about what aspects of jobs are most likely to change and what skills or attributes are important to those aspects" (p. 33); the concept of a "recently changed" job is mentioned but not operationalized. Dunnette (16) suggested that the federal government's Dictionary of Occupational Titles was limited because of difficulties in keeping the information current. Also, Cascio's (17) human resources management text mentions the notion of a "periodic audit" of tests/manuals.
There is also a limited literature on the shelf life of tests (18, 19) but these efforts do not directly address the currency of underlying job analyses (e.g., other factors such as lack of security might affect shelf life). Also, this literature does not address how to check for job analysis currency.
Given the limited findings above, we also asked a variety of professionals in the field about methods for checking the currency of job analysis information. There was general consensus that checking for currency might be important in some situations, but a common theme in the professionals' remarks was that no established procedures for doing so currently existed. Further, these individuals generally indicated that they simply asked some SMEs if anything had changed, either in the job or in relation to the underlying knowledges, skills or abilities (KSAs).
Governmental Guidelines. We searched the Uniform Guidelines on Employee Selection Procedures (20) (referred to as the Uniform Guidelines) and found very little about job analysis currency. Section 5K of that document (review of validity studies) states, "... additional studies need not be performed until such time as the validity study is subject to review as provided in section 3B above. There are no absolutes in the area of determining the currency of a validity study. All circumstances concerning the study, including the validation strategy used, and the changes in the relevant labor market and the job should be considered in the determination of when a validity study is outdated."
Note that section 3B of the Uniform Guidelines (21) concerns the search for alternatives. Included in that section is, "If a user has made a reasonable effort to become aware of such alternative procedures ..., the use of the test or other selection procedure may continue until such time as it should reasonably be reviewed for currency [italics added for emphasis]." Both of the above quotes are rather vague about when to check for currency, and they appear to offer little advice about how to check for currency.
Professional Standards. Another reference source for professionals in the field has been the Society for Industrial and Organizational Psychology (SLOP) Principles, 3rd ed. (22) Discussion of currency and comparability of job analyses in those principles tends to apply to validity generalization issues. For example, the principles say, "When a systematic new job analysis is not completed, the researcher should compile reasonable evidence which establishes that the jobs in question are similar in terms of work behavior and/or required knowledge, skills, and abilities." (p.5)
The SIOP Principles (3rd ed.) (23) also indicate, "There should be a periodic audit of the use of the selection procedures. Departures from established procedures often occur over time and new findings in psychological or psychometric theory may be relevant to one or more of the assessment procedures in use. A systematic plan for review should be followed to avoid reliance on misused or obsolete procedures" (p. 33). Again, the idea of a periodic, systematic check is mentioned, but the "when" and "how" are not operationalized.
The SIOP Principles have been recently rewritten in a fourth edition. (24) Again, there is little mention of job analysis currency issues. Our review of that document finds only one directly related statement: "The researcher should consider whether the work and the worker requirements are reasonably stable" (p. 21).
In sum, although some jobs may be changing quickly, and although job analyses are the foundation for many human resource systems, there is scant guidance on how academics and practitioners should check to see if their job analysis information is up to date.
Proposed Systematic Protocol for Assessing Currency
Below, we present a four-stage protocol for assessing the currency of job analytic information. An overview/outline of this protocol is presented in Table 1 and is described below in more detail. In describing the protocol, we assume that an organization desires to assess the currency of a job analysis (i.e., we focus this article on how to assess currency and not necessarily how often). We also assume that an organization has previously conducted a thorough job analysis and that systematic information was previously collected. We further assume that the job analysis procedures are sound; i.e., questions about accuracy, appropriateness of subject matter experts, etc. have been resolved,(25) as our focus is on currency. For example, in the protocol below, we assume that professionals working with the organization previously generated a list of important tasks (activities within work behaviors), as well as a list of knowledges, skills, and abilities (KSAs) that are important for successful job performance.
A few themes should be kept in mind. First, we note that changes to the task or KSA lists do not necessarily mean that the human resource system (e.g., selection or compensation system) that followed from the job analysis is necessarily less valid. This is a separate, follow-up consideration (although information obtained with this protocol clearly informs such a consideration). We point out how such issues might get resolved in one of our implementation examples below.
FEED