Career planning validity of self-estimates and test estimates of work-relevant abilities.

Although several published instruments obtain ability self-estimates, the above studies reporting the career planning validity of test estimates and self-estimates of ability appear to be unique. However, there have been a number of studies comparing the validity of ability self-estimates and test estimates as predictors of school and college grades. In a yet-to-be-replicated review of such studies, Baird (1976) found conclusive evidence favoring self-estimates.

The purpose of the study reported in this article was to compare the career planning validity of test estimates of six abilities typically included in ability test batteries with the validity of the six test estimates plus self-estimates for nine additional abilities. In effect, the study determined whether adding self-estimates of abilities to test estimates resulted in greater validity, that is, higher hit rates for occupational groups. The occupational groups consisted of 1,620 college students assigned to Holland (1997) types on the basis of certainty-screened occupational choice. Study analyses extend those cited above in that the previous analyses involved ability composites rather than the full set of 15 work-relevant abilities listed in Table 1.

Method

Sample

The initial sample consisted of 7,008 1st-and 2nd-year students (66% female, median age of 28) attending 14 colleges (12 were community colleges) in 8 primarily midwestern states. All had completed the ability tests and ability self-estimates in the Career Planning Program (CPP; American College Testing, 1988). Because community colleges have open admissions, they typically use the CPP in orientation programs, career planning courses, workshops, and career centers. Hence, it is reasonable to assume that students were motivated to do well on the CPP tests. It was also in their interest to be insightful and open when they provided self-estimates. Across the six CPP tests, the mean stanine scores of sample members ranged from 4.6 to 5.4. The Grade 12 national norm group has a mean of 5.0.

Prior to completing any of the CPP ability estimates, students were asked to scan a list of 140 occupational titles (ACT, 1989) arranged according to 23 job families. Then they were asked to identify the occupation that came the closest to the one they were considering. Finally, they were asked the following question: "How sure are you that the occupational choice you selected ... will still be your first choice one year from now?" (p. 2). Response options for the question were very sure, fairly sure, and not sure. Students choosing either of the last two responses were excluded from the study analyses. The remaining 1,620 students (the final sample) were assigned to Holland-type groups on the basis of occupational choice. Their distribution across Holland (1997) types was as follows: R (21%), I (21%), A (13%), S (21%), E (11%), and C (14%).

Variables

Test estimates of ability. The CPP obtains test estimates for Abilities 1-6 in Table 1. The number of items and time limits for the tests are as follows: Reading (40, 20 minutes), Numerical (32, 18 minutes), Language Usage (64, 11 minutes), Spatial Perception (35, 9 minutes), Clerical (35, 5 minutes), and Mechanical Reasoning (30, 12 minutes). These time limits total to 75 minutes. Test content and psychometric support were described by ACT (1988).

Internal consistency reliability coefficients for five of the CPP tests (Clerical excluded) ranged from .81 to .91 (median of .89) for a sample of 2,238 12th graders. The test-retest reliability coefficient (2-week interval) for the speeded Clerical Test was .74 for a sample of 590 11th graders.

Two divergent types of validity data are available for the CPP tests: (a) correlations between the tests and high school grades, and (b) the extent to which mean score profiles show expected differences among persons pursuing various occupations. Four samples of 9th graders (N = 932) and four samples of 11th graders (N = 913) completed the CPP in late winter. Across the eight samples, the median correlation between the CPP tests and end-of-year grade-point-average was .42. For a composite score based on the three academic ability tests, the median correlation was .63.

Mean score profiles for the six CPP tests were obtained for 2,021 persons who had been grouped into 15 job families on the basis of the occupations they were pursuing 6 years after completing the tests. Generally, the profiles were in line with expectations. For example, persons in the Clerical and Secretarial Work job family scored highest on the Language Usage Test and the Clerical Test, whereas persons in the Construction and Maintenance job family scored highest on the Mechanical Reasoning Test and the Spatial Perception Test. Taken together, the two types of validity evidence appear to provide good support for the CPP tests.

Self-estimates of ability. As part of the CPP, students provided self-estimates for Abilities 7-15 in Table 1. First, they read the description of an ability. For example, "sales ability" was described as "Influencing people to buy a product, service, or take a suggested course of action"; "manual dexterity" was described as "Making or repairing things easily and quickly with one's hands" (ACT, 1989, p. 9). For each of the abilities, students then indicated whether they considered themselves to rank in the lower 25%, middle 50%, or upper 25% compared to persons their own age. These response categories were assigned scores of 1, 2, and 3 in the analysis. The untimed completion of the self-estimates required about 2 minutes.

Because the nine ability self-estimates are not used, separately, for career planning, reliability data are not available. In any case, validity is not possible without reliability. Validity data were provided by the five studies cited above; each used self-estimates for the same abilities. A different type of validity data is provided by the self-estimate mean scores of occupational choice groups. As reported by ACT (2001), Grade 12 students in two samples (Ns = 1,049 and 1,503) provided self-estimates for all 15 IWRA abilities. The self-estimates were used to obtain ability composites for Holland (1997) types. Holland-type occupational group membership was determined on the basis of occupational choice screened on certainty. In both studies, all six occupational groups scored highest on the ability composite corresponding to the group's Holland type. Finally, Prediger (1999a) found that the factor structure underlying self-estimates for all 15 abilities (two samples) was similar to the Data/Ideas and Things/People Work Task Dimensions underlying the hexagonal arrangement of Holland's (1997) types. Thus, relevance of the abilities for Holland's types was empirically demonstrated.

Analyses

Discriminant analysis (Norusis, 1990) was used to determine occupational group (Holland type) differentiation and to obtain predictions of group membership (hence, hit rates). The group differentiation achieved by the ability estimates was expressed in terms of Wilks's lambda and the more usual "percent of variance accounted for" (VAF), which is one minus lambda. Bayes's rule (Norusis, 1990) was used to obtain predictions of occupational group membership (hence, hit rates) based on the ability estimates.

Discriminant analyses were conducted separately for the test estimates of Abilities 1-6 and for test estimates of Abilities 1-6 together with self-estimates of Abilities 7-15 (a total of 15 abilities). Both analyses were conducted on two random subsamples (1 and 2) stratified by college, educational plans, gender, and Holland-type of occupational choice. A cross-validation design was used with Subsamples 1 and 2. That is, Subsample 1 was used to develop equations to predict occupational group membership, and those equations were used to predict group membership for Subsample 2. Then, equations based on Subsample 2 were used to predict group membership for Subsample 1. This design controls for hit rate inflation, which is crucial when the validities of 6 and 15 predictors are being compared.

Results

Table 2 reports results averaged across the two subsamples. The results were nearly identical for the two subsamples (see Table 2 Note). The Wilks's lambdas for both sets of predictors were highly significant, statistically. The VAF for test estimates plus self-estimates (49%) was substantially higher than the VAF for test estimates alone (23%). The cross-validated occupational group hit rates were 37% and 28%, respectively. Thus, the addition of self-estimates to test estimates resulted in a 32% proportional improvement [(37%-28%)/28%] in the hit rate. The 15-variable hit rate is similar to those typically reported in interest inventory validation studies (e.g., see Swaney, 1995, for a summary).

The Rationale section noted that, in a study comparing the career planning validity of self-estimates and test estimates for the same six abilities (Abilities 1-6 in Table 1), the results favored self-estimates. Because the CPP provided the test estimates, those results suggest that, had self-estimates replaced the test estimates for Abilities 1-6 in the study reported here, the hit rates for the 15 abilities would have been at least as high. By implication, the ability test battery would have made little, if any, contribution to career planning validity. In any case, study results show that assessing a broad range of work-relevant abilities via briefly administered self-estimates can substantially add to the career planning validity of the abilities typically assessed by tests.

In order to obtain perspective on which of the 15 work-relevant abilities did the best job in differentiating Holland types, univariate Fratios (Norusis, 1990) for the abilities were examined. When the Fs were averaged across the two subsamples, the top five abilities were as follows: Manual Dexterity, Mechanical Reasoning, Helping Others, Scientific, and Creative-Artistic. The scores for only one of these abilities (Mechanical Reasoning) were based on a test estimate. In contrast, three of the six test estimates of ability ranked among the bottom five (Numerical, Clerical, and Reading). As on the typical paper-and-pencil test, Clerical was narrowly defined. The other two abilities would appear to be relevant to occupations in all of Holland's (1997) six types. Hence, lack of differentiation would be expected. Two of the nine ability self-estimates (Organization and Leadership) ranked among the bottom five, possibly because they mainly apply to the Enterprising type.

Print Get the Mag Weekly Updates