Reading comprehension exercises online: the effects of feedback, proficiency and interaction.

To investigate the effects of these different treatments (English proficiency level, Manner of study, and Type of feedback) on the comprehension of the text during the first comprehension exercise, all students were given 20 minutes to complete a second comprehension exercise related to the same text. This time, however, all students received KCR feedback. The score on this second exercise was the dependent variable. All input data were stored in a database and analyzed quantitatively. Furthermore, transcripts of the video sessions were written by the students themselves. The transcripts were then checked and formatted by the researcher before being analyzed qualitatively. The results and their implications are discussed below.

QUANTITATIVE RESULTS

Scores on the second comprehension exercise were stored in the computer log. A three-way ANOVA was then performed. Results are listed below in Table 1.

A quantitative analysis of the results in this study shows that there is no support for Hypothesis 1 as the main effect of Type of feedback was not statistically significant (F (1,217)=0.01, p>.05). Therefore, Elaborative feedback was found to be equally as effective as KCR feedback. Students receiving Elaborative feedback (M=10.39, SD=2.00) in the second comprehension exercise scored slightly higher than, but not significantly higher than, those receiving KCR feedback (M=10.33, SD=2.15). The quantitative analysis also shows that there is no support for Hypothesis 2 as the main effect of Manner of study was not statistically significant (F(1,217)=1.19, p>.05). Therefore, individual work (M=10.24, SD=2.21) was found to be equally effective as pair work (M=10.46, SD=1.95). In contrast, the main effect of the level of English proficiency level was found to be statistically significant (F(1,217)=29.19, p<.05). Hypothesis 3 is supported, therefore, as higher proficiency students (M=11.04, SD=1.93) performed significantly better than lower proficiency students (M=9.66, SD=1.98) on the second comprehension exercise.

The interaction between Type of feedback and Manner of study (see Figure 1 below) was found to be statistically significant (F(1,217)=4.93, p<.05). Therefore, Hypothesis 4 is supported as it made a difference to scores on the second comprehension exercise whether students received Elaborative or KCR feedback, and whether they worked individually or in pairs during the first comprehension exercise. When receiving Elaborative feedback, students who worked in pairs (M=10.77, SD=1.96) scored higher than those who worked individually (M=9.89, SD=1.95). For those receiving KCR feedback, students who worked in pairs (M=10.16, SD=1.91) scored lower than those who worked individually (M=10.54, SD=2.39).

[FIGURE 1 OMITTED]

Hypothesis 5 is not supported as the interaction between Type of feedback and English proficiency level was not statistically significant (F(1,217)=2.96, p>.05). In other words, there was no significant difference between students' scores on the second comprehension exercise whatever the feedback and whatever the proficiency level. When receiving Elaborative feedback, higher proficiency students (M=10.83, SD=2.03) scored higher than lower proficiency students (M=9.94, SD=1.88). For those receiving KCR feedback, higher proficiency students (M=11.23, SD=1.83) scored higher than lower proficiency students (M=9.39, SD=2.06).

Hypothesis 6 is also not supported as the interaction between English proficiency level and Manner of study was not statistically significant (F(1,217)=2.05, p>.05). It did not make a significant difference to scores on the second comprehension exercise whether higher or lower proficiency students worked alone or in pairs. Higher proficiency students scored higher when working alone (M=11.12, SD=2.00) than when working in pairs (M=10.98, SD=1.88). Lower proficiency students scored lower when working alone (M=9.31, SD=2.06) than when working in pairs (M=9.94, SD=1.90).

Finally, Hypothesis 7 is not supported as there was no statistically significant interaction between Type of feedback, Manner of study and English proficiency level (F(1,217)=0.91, p>.05). For Elaborative feedback, the highest score in this study was achieved when higher proficiency students worked in pairs (M=10.94, SD=2.24) and the lowest score was when lower proficiency students worked alone (M=9.13, SD=1.85). For KCR feedback, the highest score was achieved when higher proficiency students worked alone (M=11.45, SD=2.15) and the lowest was when lower proficiency students worked in pairs (M=9.31, SD=1.93).

To summarise the findings of this quantitative analysis of the data obtained, significant results were obtained for: (1) the main effect of English proficiency level and (2) the interaction between Manner of study and Type of feedback. To help clarify the interpretation of these results, focus is now placed on examples of the recorded interactions which are analysed qualitatively.

QUALITATIVE RESULTS

The desirability of quality interaction between students around a computer was discussed earlier. Therefore, in order to provide a suitable context for interaction, the volunteer students were asked to work in pairs and the sessions were recorded and transcribed. To investigate the perceived degree in which students are: "(a) behaving cooperatively or competitively and (b) engaging in the critical reflection or in the mutual acceptance of ideas" (Mercer 2004, p. 146), this study employs the notion of Exploratory talk (Mercer, 1995). Furthermore, to investigate the quality of the resulting interactions and the degree to which students engage in the activities, reference is also made to Wegerif et al.'s (1998) pragmatic ground rules for promoting Exploratory talk and Wegerif's and Mercer's (1996) IDRF (Initiation, Discussion, Response, Follow up / feedback) structure. Examples of interactions illustrating the effects of the different kinds of feedback are discussed below.

A Typical Scenario--Non-Elaborative Feedback

A common observation in reading classes is that students often copy answers to comprehension questions directly from answer sheets without actually considering why their own answers are different and / or incorrect. The following excerpt highlights this typical dilemma when two students (Student 1 & Student 2) receive non-Elaborative feedback, in this case, online. Having completed answering all the questions,

Student 2 suggests that they should get the computer to check their answers. Following the feedback,

Student 1 comments that it is only possible to check their answers once with KCR feedback. There was no attempt to self-correct all five errors, having received the KCR feedback, and the students considered the exercise finished.

Interaction with Elaborative Feedback 1

The transcript below comprises part of the interaction between two students as they read and progress through the first comprehension exercise. Following the first check of their answers, Elaborative feedback (lines 403-405) provided by the computer refers students back to relevant paragraphs in the reading text for three incorrect answers. Students negotiate with each other and attempt to identify which questions are wrong. Students manage to identify one of the incorrectly answered questions (lines 439-440) and decide together to change one of their incorrect answers (line 443).

Interaction with Elaborative Feedback 2

In the next transcript, students interact as they discuss another potentially incorrect answer. The point of the question is for students to find out that it gets darker earlier in Japan than in England in the summer. As part of the interaction, students negotiate the meaning of fading light. They locate their second error (lines 552-556) and change their answer accordingly. However, both agree that they are not sure what the third incorrect answer is so they agree to check their answers once more (lines 568-572) and receive the next round of feedback.

Interaction with Elaborative Feedback 3

The next transcript comprises the interaction as students receive the second round of feedback. Having correctly changed two of their three incorrect answers, one error still remains. Following Elaborative feedback (lines 573-575), students are delighted to identify which question is answered incorrectly (line 578-580). They negotiate the meaning of the hint and also the information inferred in the text about what time shops usually close. The point of the question is for students to determine that shops generally close at 5:30pm in England, much earlier than in Japan. Students then correct their third error and check their answers for a third time (lines 580-596). Finally, all questions are correctly answered (lines 597-599).