Mobile Edition
Print
Get the Mag
Weekly UpdatesINTRODUCTION
In a few months students from almost every single country will be taking high-stakes tests such as the Test of English as a Foreign Language (TOEFL), International English Testing System (IELTS), Business Language Testing Service (BULATS), and others in order to achieve a score that allows them to pursue their studies abroad. Students who want to pursue graduate studies in the United States will be able to choose from a number of tests to prove their proficiency level, but, up to now, the most important test has been the TOEFL. The TOEFL, like other exams (1), has been progressively converted into a computer-based test (CBT), first, and into an Internet-based test (iBT) later. Most students today take a computer version of this test. The promptness of the rating system as compared to the traditional paper-and-pencil format have helped to popularize the CBT and iBT versions of TOEFL. Many of the other high-stakes tests have followed or are following this change. Probably, the second most important high-stakes exam is IELTS, whose computerized version is currently being used experimentally in several countries as a first stage of implementation.
The degree of computerization varies according to the test. It runs from an implementation of the four skills (iBT TOEFL) to only the least communicative sections (BULATS), with some exams using a combination of human and computer interaction such as the Basic English Skill Test Plus (BEST Plus). The continuous evolution of the Internet and computer devices, exam takers' better training, and the reduced costs of implementing computerized tests will eventually facilitate the transition from the current paper format to the electronic one.
This paper will present the features, and a brief comparison, of some of the most well known high-stakes exams. Since some of these tests are not well known or are not easily accessible to the general practitioner, this paper will serve as a first approach to a field that will undoubtedly change significantly in the next five years.
COMPUTER TEST DESIGN
Computer tests have been operative for at least 20 years. However, although the potential of computer assisted language testing was foreseen from the mid-eighties, it has been necessary to develop the means to include audiovisual prompts in the tests. As with most general learning platforms, testing platforms have evolved significantly in the past ten years. Some of these changes have been reflected in this journal (Roever, 2001) but, overall, the greatest challenge is not preparing the test or producing a testing platform that can test faster and more efficiently than teachers do today, but rather, first, being able to develop specific items only designed for computers and, second, studying how the results and correction of online tests can lead to learning (Chapelle & Douglas, 2006). Figure 1 shows the interface of a low-stakes testing platform where the students can see corrections. The student can clearly see different codes according to the mistake produced and, if a list of color codes is provided, will be able to recognize the type of mistake that was made. Additionally, some testing platforms include comments from the raters but it is not clear whether these comments lead to language learning.
[FIGURE 1 OMITTED]
The fact that it was necessary to state clearly if certain features are common or even necessary when designing computer interfaces for tests led Fulcher (2003) to provide a set of guidelines that most testing platforms follow. These guidelines relate to hardware/software configurations, navigation, page layout, toolbars and controls, text, text color, icons and graphics, multimedia, delivery systems, score retrieval, database storage, test rubrics, familiarity (and ergonomics) technology, and trialing. The guidelines are currently used by the designers of tests such as TOEFL or BULATS but very few (if any) studies have proven their validity.
TYPES OF ONLINE OR COMPUTER-BASED TESTS ITEMS
From a traditional perspective, computer-based tests are usually composed of multiple-choice questions. In the last few years, many low- and high-stakes tests have included different types of exercises that can be found in many e-learning platforms. For example, the long-standing DIALANG project (an adaptive low-stakes testing platform) includes a section where users can suggest new types of items. Today, however, while multiple-choice items have preference in high-stakes exams, low-stakes exams, like those distributed by many universities all over the world tend to include more free writing sections (Bernhardt, Rivera & Kamil, 2004). In relation to the interface organization, formats vary, although most tests follow the patterns described below:
Multiple-choice items
Traditionally set as a vertical list of questions with answers, today most exams present only one question at a time. This allows for both a bigger font size for questions and answers as well as larger displays of any media when they occur as prompts for a question. Multiple-choice tests tend to be adaptive. Adaptive tests are based on the results (right/wrong) for each test item and have the capacity to place students in the appropriate proficiency level by using an item bank (Meunier, 1994; Brown, 1997). Generally, this type of test tends to reduce the time necessary to take the test and facilitates correction and reporting.
Writing
Interfaces for writing usually show the retrieval device and provide a box to write in the answer. The key issue in these interfaces is whether this box should be large enough to let students see most of their test or whether simply scrolling up and down is an acceptable format for writing an essay comfortably (Figure 2).
[FIGURE 2 OMITTED]
Speaking
Multimedia reproduction devices usually occupy most of the screen. The idea is that the conversation should be as natural as possible. A larger picture is supposed to produce in the viewer an effect of increased identification with the task. (Figure 3)
[FIGURE 3 OMITTED]
Tests have tended to follow three types of format. Undoubtedly, these formats affect the students and the strategies used in the preparation for the test. The most significant difference is in the oral section. Some tests do not include a speaking section while others have integrated the oral section in the platform through semi-directed interviews and descriptions. Overall, there are four approaches:
1. There is no speaking section. The test includes some listening tasks, but the student is not expected to speak.
2. The platform is used for grammar, reading, and listening (and sometimes writing. There is an individually-administered speaking test administered by a human tester.
3. The testing platform is used to provide input (in the case of BEST Plus, adaptive input), but there is a test administrator present while the test takes place.
4. The speaking section is included in the computer test, and there is no test administrator specifically in charge of giving the speaking test (for example, the new iBT TOEFL).
REVIEW OF HIGH STAKES TESTS
Although recent literature has paid special attention to high-stakes tests (Stoynoff & Chapelle, 2005), very few journal publications have addressed the use of computers in high-stakes test deliveryAccording to some research, the difference between a paper-and-pencil test and a computer-based test is minimal (Breland, Lee, & Murak, 2005; Choi, Kim, & Boo, 2003). However, anecdotal evidence gathered from teachers at the Polytechnic University of Valencia suggests there are broad differences between students in their computer skills. Thus, there are probably some differences between the two ways of taking an exam.
This paper looks at eight high-stakes tests in order to review their features. They are classified in the following fashion: tests that only include multiple-choice questions, tests that include writing and multiple-choice questions, and tests that include speaking questions. The tests reviewed are: BULATS, IELTS, TOEFL, ACT ESL placement test, BEST Plus, Computerized English Skills Assessment (CELSA), WebCAPE (Computerized Adaptive Placement Exam), and iBT TOEFL. It is important to mention that no discrimination has been made between online tests and other computer-based tests (on CD ROM or Intranet systems) because as technology progresses, it is expected that most of these and other tests will go online.
Tests with no speaking section
ACT ESL Placement test (originally a variation of the COMPASS testing system, http://www.act.org) is a computer adaptive testing system for mathematics, reading and writing, and ESL. The type of questions included in the test can be found at http://www.act.org/esl/sample.html. For ESL, it includes sections on language use, reading, and listening that can be used singly or in combination. Additionally, Compass has developed a new writing test called ESL e-Write that will go online very soon (http://www.act.org/compass/announce/preview.html). This test can be used to evaluate high school and university students.
WebCAPE Computer-Adaptive Placement Exams (formerly ESL-Computerized Adaptive Placement Exam, https://www.softstudy.com/products/CAPE.cfm), developed by the Humanities Research Center, Brigham Young University, is a low-stakes exam usually used in commercial hiring. The test can be taken in ESL, Spanish, German, French, and Russian. The 15-20 minute test includes language use, listening, and reading and can be tailored according to the needs of the testing institution. Until recently this test could be obtained from Brigham Young University but now is run by SoftStudy, Inc. A free version of WebCAPE is available at https://www.softstudy.com/products/trial.cfm.
FEED