Mobile Edition
Print
Get the Mag
Weekly UpdatesThis article summarizes 12 presentations in Group 7 of the 2007 joint symposium of the International Association for Educational and Vocational Guidance, Society' for Vocational Psychology, and National Career Development Association held in Padua, Italy, that focused on procedures for verifying the efficacy of vocational guidance programs. Three themes, or general tensions, emerged from the group's discussion of the presentations: (a) quantitative versus qualitative evidence, (b) public policy needs versus local needs, and (c) comprehensive versus cost-effective programs. On the basis of these core themes, recommendations are made for 3 potential directions for research that address certain threats and opportunities for the field.
Conscious of an era emphasizing evidence-based practice and empirical standards of accountability, Group 7 of the 2007 joint symposium of the International Association for Educational and Vocational Guidance, Society for Vocational Psychology, and National Career Development Association held in Padua, Italy, focused on procedures that can enable practitioners, consumers, and stakeholders to evaluate the effectiveness of vocational guidance programs. In other words, how do practitioners know that vocational interventions achieve their intended outcomes? What are the criteria that should be adopted to judge the credibility of methods used for measuring such outcomes? After 12 presentations by scholars and practitioners from the international community of vocational guidance practitioners, the group's discussion led to more questions than answers. In this article, we summarize three thematic tensions that emerged during the course of the discussion. We also discuss threats, which were identified by the group participants, to the sustainability and adaptation, within shifting cultural contexts, of vocational guidance programs that are responsive to the needs of diverse populations across diverse settings.
Theme 1: Quantitative Versus Qualitative Evidence
Perhaps it was unavoidable that a tension between quantitative and qualitative methods of program evaluation would emerge. This tension became evident when Steven Brown of the United States responded, during the presentation of a paper (S. D. Brown, 2007), to a question about how researchers should demonstrate the efficacy of vocational guidance programs. His recommendation was to conduct a meta-analysis. S. D. Brown clarified that his idea should not be taken as seriously as it might have come across; yet, the audience thought that it was a crucial point, for it brought into play the relative priority that scholars should place on deductive, statistical findings versus contextualized, inductive findings. Although meta-analyses can help to establish the overall effectiveness of interventions, their relevance may be limited in guiding programs to meet the specific concerns of populations presenting with a constellation of risk factors and/or cultural matters that do not correspond to (or are not represented by) the prevailing samples, problems, and constructs on which the meta-analyses are based. Indeed, this limitation has previously been noted by Whiston (2002) concerning efficacy literature as a whole among career counseling interventions.
Cinamon (2007) highlighted the challenges of merging standardized quantitative outcome measures with locally constructed narratives in her study of Israeli youth at risk for school dropout (i.e., Russians, Ethiopians, Israeli Arabs) across seven high schools. In her school-to-work project, which was designed to help schools build the capacity to carry out their own guidance programs, the author noted that common outcome variables, such as career decision-making self-efficacy, do not necessarily offer a useful assessment of program efficacy because they do not represent the same meanings to different children, depending on their ethnic experiences. Cinamon suggested that important outcomes of interest should not be restricted to quantitative measures, but should be complemented by qualitative findings, which, in her research, illustrated the value of assessing meanings and motivations that derive from outcome expectations. For example, Arab children born in Israel expected negative outcomes (despite exhibiting high levels of self-efficacy) because career success would mean, as one child expressed it, "going against my people" in a historical context of ethnic discrimination. At the 2004 joint symposium of the International Association for Educational and Vocational Guidance and National Career Development Association held in San Francisco, a similar concern of exclusive adherence to standardized quantitative outcomes was voiced in a discussion group that focused on cross-cultural perspectives of career assessment (Watson, Duarte, & Glavin, 2005).
One of the most polemical points of view on the relative value (or utility) of quantitative procedures was offered by Bright (2007) of Australia, who advocated a chaos theory approach to career counseling. Echoing the notion of positive uncertainty articulated by Gelatt (1989), the author proposed what some scholars in the field might call a paradigm shift from conventional career counseling models that portray an adaptive decider as someone who uses a rational, independent, and methodical decision-making style; in other words, he challenged the traditional notion that the decision maker, under ideal conditions, should be free of emotional distractions and should systematically gather, evaluate, and use information to maximize his or her future career goals, while calculating or weighing various career alternatives (see Phillips, 1997). In this respect, Bright also critiqued experimental designs and statistical models that have dominated program evaluation efforts, although he did not offer any clear alternatives to assessing program efficacy.
From a mixed methods perspective, Perry (2007) of the United States expanded on Cinamon's (2007) suggestions of moving toward blending narratives with numbers. In his study, which evaluated a career education program targeting urban minority youth, Perry described how qualitative data can be converted into quasi-statistical findings, and thus achieve a certain degree of transferability, through quantifying the developmental strength of internal variables (e.g., self-efficacy) using coding protocols built on consensual qualitative methods (Hill, Thompson, & Williams, 1997). Perry also encouraged a social action approach, whereby programs are not only demonstrated to be efficacious, but also can be transformed into viable solutions that are embedded in a community. Although statistical procedures are vital for diagnosing career concerns--and thus appropriately intervening--as exemplified through S. D. Brown's (2007) meta-analytic factor analysis of correlation matrices that concern making vocational choices, they are not sufficient if the goal of efficacy research also entails altering contexts and transforming how systems regularly operate.
In the final analysis, discussion group participants concluded that quantitative and qualitative evidence are both important sources of efficacy verification and, consequently, are both needed in future research in the vocational guidance field. But in terms of how these two contrasting paradigms of inquiry could be integrated or mutually enhance each other, no straightforward, feasible solution was identified other than the consensus that the more ways that program efficacy can be demonstrated, the better.
Theme 2: Public Policy Needs Versus Local Needs
Another tension that became evident in Group 7 discussions was the need to persuade the policy makers when the objectives of vocational guidance efficacy research may come into conflict with the needs and values of local change agents. For organizations to make an investment in a model of vocational guidance, a rationale that is unsupported by the hard language of results (no matter how intuitively pleasing it may sound) may not be enough to convince local officials that a project is actually worth pouring millions of dollars into year after year. For better or for worse, behavioral {e.g., employment levels, grades, job retention rates) and economic (e.g., annual income, tax revenues) outcomes tend to be a preferred language of change for many power brokers who like to see logic models accompanied by long-term benefits, not just short-term changes in attitudes (e.g., career maturity). This dilemma presented significant challenges, leading the group participants to consider the question of which parties in society should be called on to define what are the most credible or important criteria for demonstrating program success, including the controversial issue of which criteria are more important than others.
Rossier (2007) from Switzerland pointed out several caveats in claiming that a program works in the absence of conducting follow-up assessments that may span from a month to an entire year, or even longer. According to his evaluation of a vocational guidance program serving college-educated young adults and another program serving adult job seekers, postintervention gains were not necessarily sustained. Aside from longitudinal designs, several presenters raised the issue of the difficulty of trying to satisfy public policy criteria when the purposes of a vocational guidance program (and needs of local stakeholders) do not coincide with those criteria. Ingstad, Warska, and Carey (2007) from Norway described a pilot project that developed local partnerships between businesses and secondary schools in Norway and Holland. The authors provided a practitioner-oriented structure of guidelines for school administrators, counselors, mentors, and parents, based on a social action approach, to facilitate the entrepreneurial spirit and vocational decision making of young people. Their project was in its early stages of development, and it was not designed to demonstrate statistical gains per se, but to forge the partnerships that would ultimately lay the foundation for its sustainability.
FEED