Effect and Affect

*This research was supported by a technology initiative grant by College of Humanities at the University of Arizona in the spring of 2000. An earlier version of this project was presented on March 16, 2000 at the 34^th TESOL Convention in Vancouver, British Columbia, Canada.

This study investigates whether differing commenting and interaction modes (technology-enhanced versus traditional) make a difference in the area (global versus local), the type (evaluation, clarification, suggestion, and alteration), and the nature (revision-oriented versus non-revision-oriented) of comments by peers reviewers in L2 writing, and what impact the observed differences have on students’ revisions. The findings show that the overall number of comments, the percentage of revision-oriented comments, and consequently the overall number of revisions made by the technology-enhanced group was larger than those by the traditional group. While the participants in the technology-enhanced group tend to find MOO interaction affectively more appealing, a closer look at interaction modes suggests that face-to-face communication is more effective than MOO communication because of the nonverbal communication feature that is indispensable in intercultural communication in a peer review setting. In lieu of the differential effects within each commenting mode (Word editing versus pen and paper) and interaction mode (MOO versus face-to-face) we suggest that the use of Word editing in an electronic peer review mode combined with face-to-face interaction in the traditional peer review mode may serve as a two-step procedure for effective peer review activities in L2 writing classrooms.

Introduction

Over the past decade, the biggest concern second language writing teachers have expressed about using peer review activities is whether peer comments in fact help students write better papers. This concern raises two fundamental questions: First, do second language writing students have sufficient linguistic, content, and rhetorical knowledge to give their peers constructive feedback on their drafts? Second, are these students able to modify their texts based on their peers’ comments?

Some researchers (Amores, 1997; Leki, 1990; Nelson & Murphy, 1992, 1993) have stated that peer review is a problematic task for many second language writers for a number of reasons. First, students sometimes focus too heavily on “surface concerns” (Leki, 1990, p. 9), and sometimes they tend to neglect larger revising issues, and provide vague and unhelpful comments. Second, students sometimes could be hostile, sarcastic, overly critical, or unkind in their criticisms of their classmates’ writing. Moreover, the interactions of the group are at times unpleasant, with students being overly critical of each other’s writings (Nelson & Murphy, 1992). In fact, the nature of responding to peers’ drafts sometimes generates a sense of discomfort and uneasiness among the participants. Generally speaking, the students could become rather defensive when their work is criticized, especially by their peers (Amores, 1997). In doing peer response activities, some students might feel uncertain about the validity of their classmates’ responses, and some might struggle with their own listening comprehension skills due to the accents of their peers. Due to the lack of L2 formal (rhetorical) schemata, some students might have inappropriate expectations about the content and structure of peers’ texts, resulting in counterproductive feedback that leads writers further away from the expectations of their teachers.

On the other hand, some second language writing researchers found that peer comments “enhance a sense of audience, raise learners’ awareness of their own strengths and weaknesses, encourage collaborative learning, and foster ownership of text” (Tsui & Ng, 2000, p. 147). Some researchers have found that students, especially those who have been trained in peer review, are quite capable of making useful suggestions about their peers’ drafts (i.e., Berg, 1999; Hedgcock & Lefkowitz, 1992; Paulus, 1999; Stanley, 1992; Villamil & de Guerrero, 1998). Students, especially those who have been trained in peer review, are able to give specific comments and advice on their peers’ writing and point out problems with content and rhetoric (Berg, 1999; Hedgcock & Lefkowitz, 1992; Lockhart & Ng, 1995; Mendonça & Johnson, 1994; Stanley, 1992; Villamil & de Guerrero, 1998). Analysis of the interaction in peer response groups has shown that students actively ask questions (both requests for information and comprehension checks), give and ask for explanations (unclear points, opinions, and content), restate, give suggestions, and correct grammar mistakes (Mendonça & Johnson, 1994). In fact, peers may give better content feedback than teachers if the students are paired based on fields of study (Belcher, 1990). In addition, peer response groups can be a fruitful environment for students to negotiate meaning and practice a wide range of language skills (Lockhart & Ng, 1995) which are integral to their development not only as second language writers but also for the development of all four language skills in a collaborative environment (Vygotskii, 1978).

In spite of the controversies, a fundamental question all L2 writing researchers and teachers have about peer review activities is: Will students be able to incorporate peer comments into their revisions? There is no definitive answer to this question. Some research indicates that students utilize only a small percentage of their peers’ comments in revision (i.e., 5% for the students in Connor and Asenavage’s 1994 study; see also Nelson & Murphy, 1993; Partridge, 1981; Tsui, 1999), but much of the research has shown that students utilize all three types of feedback: self, peer, and teacher (i.e., Caulk, 1994; Chaudron, 1984; Cheong, 1994; Mendonça & Johnson, 1994; Paulus, 1999) even if they at times rely more on their teacher’s feedback for revision (Cheong, 1994). In fact, research on revisions based on teacher versus peer feedback has shown that both helped (Caulk, 1994; Chaudron, 1984; Paulus, 1999), especially since peer feedback seemed to be more focused on specific concerns, whereas teacher feedback was more global. A study comparing the revisions of one group with teacher feedback only and one group with peer feedback only found that there were no significant differences in revision--both groups performed equally well (Hedgcock & Lefkowitz, 1992). In addition, peer feedback has also been found to instigate further revision after the peer response activities ended (Paulus, 1999; Villamil & de Guerrero, 1998) indicating that students keep considering their peers’ comments when revising their drafts on their own.

Another interesting area of research on peer review which is directly related to the effects of peer review activities and the extent students’ drafts are revised is Computer Mediated Communication (CMC). CMC refers to both asynchronous communication such as e-mail and virtual synchronous conversation in Multi-User domains Object-Oriented (MOO’s).[1] CMC’s afford an additional and possibly less anxiety-provoking means of learner to learner communication within language classrooms than face-to-face interaction. In addition, it extends the interaction possibilities beyond the classroom walls, hence “beyond its time constraints and the usual limited circle of interlocutors of classroom pair and group work” (Belcher, 1999, p. 255). This aspect of convenience is often cited as being an important advantage of MOO’s for classroom communication, as it enables participants to collaborate outside of the classroom on their own time (Egbert, Chao, & Hanson-Smith, 1999; Snookes, 1995). Research in second language classrooms indicates that networked computers do indeed enhance opportunities and motivation for authentic interaction and meaning negotiation (Kern, 1995); reduce anxiety and produce more talk (Fanderclai, 1995; Harris, 1995; Kern, 1992; Reid, 1994); and improve linguistic proficiency and increase self-confidence (Beauvois & Eledge, 1996). In ESL/EFL writing classes, observations of the use of CMC in various formats also indicate students’ enhanced confidence in writing and increased quantity in both peer and teacher feedback (Braine, 1997). Researchers (Sproull & Kiesler, 1991; Rheingold, 1993) also suggest that CMC has great potential for leveling the playing field in the multilinguistic, multicultural classroom, thus empowering minority students or nonnative English-speaking students in composition classrooms with linguistically and culturally diverse backgrounds (Belcher, 1990). As oral classroom participation is not encouraged in many countries where attentive listening and students’ silence are expected (Liu, 2000), peer review via CMC seems greatly facilitated by synchronized interaction. Warschauer (1996) found much more equitable conversations in the CMC mode than in face-to-face as the less vocal students seemed to participate more. This finding of increased participation was confirmed by Sullivan and Pratt (1996) whose study showed full student participation in electronic discourse as compared to 50% participation in face-to-face interaction. In terms of the quality of interaction, Warschauer (1996) reported that the participants in his study tended to express their own ideas during electronic communication rather than directly answering questions posed to them, suggesting increased creativity. Also, the electronic discussions involved significantly more complex sentence structures than face-to-face discussions.

In the latest work on peer response, Liu & Hansen (2002) conclude that research in this area thus far has already observed and identified both quantitative and qualitative differences between electronic interactions and face-to-face interactions in peer review activities. What is needed is a close examination of whether the differences in peer review activities in both commenting modes (technology-enhanced versus face-to-face) will lead to differential effects on students’ revisions. To address this issue, the present study investigates whether differing commenting modes make a difference in the areas, types, and the nature of comments made and how these comments affect revisions.

The study

As mentioned above, CMC is a rapidly growing tool used for peer review in L2 writing. There has been little research, however, that directly compares the effectiveness of traditional peer review versus this electronic format. Given the rapid increase in computerized classrooms at universities across the nation and the increasing numbers of courses which rely on distance learning, the use of CMC in performing peer review is becoming a crucial tool for the teaching of ESL writing. The purpose of this study is to directly compare the effectiveness of traditional peer review (i.e., students writing their peer comments on their partners’ papers and then meeting face-to-face in class to discuss those comments) versus electronic peer review (i.e., students making comments on computers using features of Microsoft Word (discussed further in data description) and then communicating electronically via a MOO). The rationale for examining asynchronic (either paper or electronic) and synchronic (either face-to-face or MOO) peer review is that this study seeks to reflect the usual classroom procedure for peer response, which does not normally use either asynchronic or synchronic peer review in isolation.

To evaluate the extent to which students make revisions on their drafts as the result of peer comments either in the electronic mode or in the traditional mode, we must first of all bear in mind that to quantify only the ratio of overall comments versus overall revisions is misleading as students’ revisions of their drafts are constrained by a number of factors, such as the areas in which the comments are made (global versus local) and the types of comments made in each area (evaluation, clarification, suggestion, and alteration). Some comments (e.g., “This is a good thesis.”) are not revision-oriented, and some comments might be too vague to be taken into account (e.g., I’m not sure what this means but it is probably o.k.”). Therefore, we should not only identify and describe the areas and the types of comments made in peer review, but also differentiate the nature of comments (revision-oriented versus non-revision-oriented). Only in this way can we examine the effects of peer comments on revisions with great precision. This study investigated how the different commenting modes (technology-enhanced versus traditional) affect the nature of comments made in both areas across and types, and how the differential comments affect the revisions. We examined the following three research questions:

1. Does the use of a technology-enhanced asynchronic commenting mode (Word editing) versus a traditional commenting mode (paper and pencil) result in a differential distribution of peer comments in both areas (global versus local), types (evaluation, clarification, suggestion, alteration), and the nature of each type of comment (i.e., revision-oriented versus non revision-oriented)?

2. Does the use of a technology-enhanced synchronic interaction mode (MOO) versus a traditional synchronic interaction mode (face-to-face) result in a differential distribution of peer comments in both areas (global versus local), types (evaluation, clarification, suggestion, alteration), and the nature of each type of comment (i.e., revision-oriented versus non revision-oriented)?

3. To what extent do students revise their papers based on peer comments made in the traditional versus technology-enhanced commenting and interaction modes?

The participants

The participants for this study consisted of 48 students taking second semester freshman composition at a large southwestern university in the United States during the spring semester of 2000. The students ranged in age from 17 to 28 years, with the average age being 20. Two classes, each with 24 students, took part in the study. The first class (traditional) contained only non-native speakers of English, while the second class (technology-enhanced) was a mixed section containing 7 native speakers of English (all from the United States) and 17 non-native English speakers. Combined, the group represented 16 different languages and 17 countries. The non-native students had studied English for an average of 9 years, and all of the students had attended the university the previous semester. The same instructor taught both sections involved in the study and used the same readings, assignments, and grading rubrics for both groups. The instructor was a Ph.D. student specializing in teaching English as a second language and had taught writing at the university for three years.

In each class, students were divided into six peer review groups. In the traditional class, all the students were non-native English speakers. In the technology-enhanced class, only two groups consisted of all non-native English speakers. While complete data was gathered from all 48 students throughout the semester, we only focused on 8 non-native English–speaking students (1 group of 4 from each class) for this study in order to obtain more in-depth data for analysis. The two groups were purposefully chosen to reflect the diversity of the overall non-native population, with each group containing four different languages and both males and females. The traditional group consisted of three females (Japanese, Indonesian, and Singaporean) and one male (Mexican), while the technology-enhanced group contained two females (Indonesian and Sri Lankan) and two males (Peruvian and Greek). While the participants varied in the number of years they had studied English, all were placed into the class based on their performance on a university-specific writing placement exam. In addition, all students admitted to the university must have a score of at least 500 on the TOEFL exam. The detailed demographic information can be seen in the following table (Table 1).

Group	Name	1st Lang	Country	Sex	Age	Years of English
Traditional	Ayako	Japanese	Japan	F	25	12
	Jane	Indonesian	Indonesia	F	18	6.5
	Orlando	Spanish	Mexico	M	18	4
	Belinda	English	Singapore	F	19	15
Technology Enhanced	Christina	Indonesian	Indonesia	F	18	11
	Christian	Spanish	Peru	M	17	11
	Michael	Greek	Greece/Cyprus	M	20	9
	Vayomi	Singhala	Sri Lanka	F	20	12

Data description

This study contains data from multiple sources. First, a personal information sheet was collected from each student at the beginning of the semester, consisting of students’ demographic information, their attitudes about peer review and the use of technology in the classroom. Second, we collected data from one of the three required student essays[2] for the course including three drafts for each student, peer review comments made on the initial draft, two follow-up questionnaires, informal interviews with the students, and transcripts of classroom peer review interaction. To ensure that the revisions observed in the second drafts were derived only from the peer comments, teacher comments were not made on the papers until the second drafts had been turned in.

The peer review for this paper took place in fifth week of the semester. The students in the technology-enhanced group attended class in a computer lab throughout the semester (see illustration I). The two classes followed the same basic syllabus, but the activities for the computer-enhanced class made use of the computers, including using Microsoft Word for in-class writing assignments, using the commenting features in Word on journal assignments, and using the MOO for class discussions. Before the students turned in their rough drafts, there was a two-part training session in order to ensure that all participants were well informed about the rationale and the procedures of the peer review process. First, there was an in-class discussion for both groups regarding the benefits of peer review. This discussion was based upon the format laid out by Berg (1999). Each group was also given a peer review sheet for this particular assignment devised by the instructor, which included questions on the introduction and thesis statements, organization of the paper, use of topic sentences, idea development, textual evidence used in the paper, and grammar issues. The rationale for the questions on the sheet was discussed with the students by comparing the elements emphasized for peer review with the actual grading rubric utilized for the class. For this section of the training students were also given a copy of an essay (rough and final draft) written by a student during a previous semester and the peer review comments for that essay. Students were asked to comment on how the essay changed and why. The students also discussed what comments seems to be useful or not useful and why.

The second part of the peer review training was customized for each group and took place over a 75-minute class period. In the traditional group, a section of a sample essay was photocopied for the students and they were asked to comment on it with a partner in class, writing their answers on the peer essay itself and on the review sheet provided, which was identical to the sheet used in part I of the training. The pairs then spent approximately 15 minutes in class commenting on the sample. Next, the groups shared their answers to the peer review questions and the class discussed what they believed to be the most useful responses generated in the class. In the technology-enhanced group the same writing sample was used and a peer review sheet with the same questions was utilized, but the writing sample was given to the students on disc, and the training session took place in a computer lab, as seen below:

The students were reminded by the instructor (through the use of a computer projector) how to use the “track changes” and “insert comments” features of Microsoft Word[3] and the students then practiced this feature with a partner for several minutes using the writing sample on disc provided by the instructor. The students were then asked to use the “track changes” and “insert comments” features to answer the questions from the peer review sheet, but this time with a partner, and using the computers in the lab. The students in the technology-enhanced group were instructed to make all comments on the computers. After 15 minutes of commenting, they exchanged their answers and reflected on the use of technology in peer review.

For the actual assignment in this study, there were three drafts required for each student. In the traditional group the first drafts of the assignment were handed in on paper, while the technology-enhanced group handed in their drafts on disc. In each case, the students were also required to give their peer review teammates copies of their papers in the appropriate format (i.e., on paper for the traditional group and on disc for the technology-enhanced group). All the students received comments from their peers on their first drafts and used those comments to make their second drafts which then received teacher comments in their respective formats. After the students exchanged their comments with their peers and had a chance to read the comments, they were divided into groups and engaged in peer review interactions with the technology-enhanced group on the MOO and the traditional group sitting in a circle. The students in each class were given a set of three guiding questions related to the peer review sheet as starters, but were encouraged to focus on any elements of the papers which they believed to be important for the assignment. This section of the peer review process took place over a 75-minute period. The interaction of the traditional group was tape-recorded and later transcribed (see a segment in Appendix 1) while the MOO discussion was recorded and transcripts were automatically generated for later analysis (see a segment in Appendix 2). The peer review process for both groups is summarized in Table 2:

The overall framework for data analysis in this study is based on the model from Wolcott (1994) on his continuum of data description, analysis and interpretation. Our analysis and interpretation focused on how the comments from the students were distributed in both global areas (feedback with regards to idea development, audience and purpose, and organization of writing) and local areas (feedback with regards to copy-editing, such as wording, grammar, and punctuation) (McGroarty & Zhu, 1997); how the comments in each area were made across four types: Evaluation (comments on either good or bad features of writing), clarification (probing for explanations and justifications), suggestion (pointing out the directions for changes), and alteration (providing specific changes); and what the nature of comments is in each type and in each area. We developed the following grid for analysis (Table 3).

The analysis procedure consisted of five steps. First, each researcher coded the peer comments made on all eight students’ papers. In the traditional group, this coding covers marginal peer comments written on the papers themselves, end comments, and comments written on the peer review sheet provided by the instructor. In the technology-enhanced group, on the other hand, the coding covers peer comments inserted electronically via the Microsoft Word “insert comments” feature, changes made to the papers via Word’s “track changes” feature, marginal comments typed into the paper, and end comments made regarding the paper or the peer review sheet. In order to avoid ambiguities in analysis, the coding was based on sentential meaningful units. When the meaning of a comment was complex, we determined the entry by its meaningful units[4]. Second, we inserted those comments into our analysis grid. In addition, comments made at the end of the papers and/or on the peer review sheets were inserted into the grid as well. Third, we discussed those comments we had disagreement on in order to maintain inter-rater reliability. We used Kuder-Richardson Approaches (KP21) for determining internal consistency between the two raters, and our reliability estimate is .90. Fourth, the overall comments made by the students were calculated by both researchers for each section of the grid and the number of revision-oriented comments in each section of the grid was also determined. Finally, the students’ revisions in their later drafts were then compared to the total number of revision-oriented comments in their first drafts. The same analysis procedure applied to the data collected during the MOO or face-to-face interaction. After transcribing the face-to-face interaction and printing out the MOO transcripts, each was coded and put into the analysis grid.

Findings and interpretation

Based on the analysis grid above, we will discuss our findings according to the three research questions for this study.

Research Question One: Does the use of a technology-enhanced commenting mode (Word editing) versus a traditional commenting mode (paper and pencil) result in a differential distribution of peer comments in both areas (global versus local), types (evaluation, clarification, suggestion, alteration) and nature (revision-oriented versus non revision-oriented)?

As revealed in Figure 1, the number of global comments (feedback with regards to idea development, audience and purpose, and organization of writing) made by each group was similar, although global comments were the minority for both as well. The number of local comments (feedback with regards to copy-editing, such as wording, grammar, and punctuation), on the other hand, varied widely between the two groups, with local comments clearly dominating in the technology-enhanced group. As the figure illustrates, although the technology-enhanced group made 75% more comments that the traditional group (316 versus 180), this difference was mostly due to the much larger number of local comments made by the former. Because the questions on the peer review sheet were primarily global in nature (e.g., Take a look at each paragraph. Does each paragraph introduce and develop one idea?), this suggests that the technology-enhanced group not only made global comments based on the peer review sheet, but they also went beyond that and made substantial local comments on screen due to the convenience of the technology-enhanced commenting features. Many of these local comments in the technology-enhanced group also pertained to problems in grammar and spelling on the papers. It is likely that the automatic grammar and spelling check provided by Microsoft Word also contributed to this preponderance of local comments in this group.

As demonstrated in Figure 2, the largest difference in the types of comments made by the two groups had to do with alteration comments. While this type of comment contributed only 7.2% of the feedback in the traditional group, it accounted for almost 47% of the comments made by the students utilizing technology-enhanced peer review. As discussed above, this difference may be largely due to the influence of the grammar and spelling check provided by Microsoft Word on the technology-enhanced group. The traditional group, on the other hand, seemed hesitant to provide alteration comments to their peers, perhaps due to a lack of self-confidence regarding their own grammar and spelling skills. While it certainly would have been possible for the students in the traditional group to check in a dictionary for misspelled words, or a grammar resource for problems at the sentence level, this would have required that they first notice that a problem existed and then they would have needed to consult such outside resources. In contrast, the technology-enhanced group was given guidance (albeit far from perfect) from Microsoft Word regarding potentially problematic areas at the local level via red underlining (signifying spelling errors) or green underlining (signifying potential grammar problems). Instead of having to consult an outside resource to make changes for these problems, the peers only had to right-click on the mouse to see options for alternative spellings and/or sentence structures. In short, local level alterations were quite easy to make for the technology-enhanced group in comparison to the traditional group.

While the percentage of clarification and suggestion comments made by both groups was relatively similar, the other large difference between the two groups had to do with evaluation comments. Whereas nearly 60% of the comments made by the traditional group were evaluative, only 25% of the technology-enhanced comments fell into this category. This difference could be explained by the fact that the traditional group heavily relied on the peer evaluation sheet, which did not require them to ask any clarification questions, and they tended to make fewer specific comments in the text itself, instead often writing their comments on the peer review sheet. The technology-enhanced group was given the same peer review sheet, but the peers in that group wrote almost none of their comments on the sheet, instead choosing to type their answers. Moreover, almost all of the evaluation and suggestion comments made by the traditional group were global in nature. This could also be explained by the fact that the questions in the peer review sheet played an important role in influencing this group in the types of comments they made. For instance, the question “Does the introduction lead into the thesis well?” elicits more global/evaluation comments, while the question “Is there anything missing? If so, please specify.” encourages the reviewers to suggest additions or changes to the introduction. It is interesting to note that neither the technology-enhanced nor the traditional group made any global/alteration comments, suggesting the nature of global comments does not lead itself to alteration because this would require the peer reviewer to rewrite substantial sections of the paper, and this is usually not feasible nor desirable in peer review.

As discussed above, the technology-enhanced group made a larger number of comments overall (316 versus 180) and the comments made by the two groups also differed both in area (global versus local) and type (evaluation, clarification, suggestion, and alteration). One important question that has usually been ignored in previous studies, however, is the nature (revision-oriented versus non revision-oriented) of the comments made. Some types of comments (i.e., alteration) are by definition revision-oriented, as they involve actually giving the author an alternative word or sentence to replace the original. On the other hand, evaluation comments may easily be revision-oriented (e.g., I think that your thesis statement is not clear) or non revision-oriented (e.g. Yep, really nice introduction).

As illustrated in Figure 3, 92% of the comments made by the technology-enhanced group were revision-oriented in nature, compared to 75.6% for the traditional group. As implied above, much of this difference is due to the large number of alteration comments made by the technology-enhanced group. As all of the alteration comments for both groups were revision-oriented, and as the technology group had a much larger percentage of alteration comments compared to the traditional group (47% versus 7%), this finding should not be surprising. However, if the alteration comments are ignored for both groups, the percentage of revision-oriented comment made by the technology-enhanced group is still 11% larger, as shown in Figure 4.

As shown above, if the alteration comments are eliminated from both groups, the overall number of comments made per group differs by only one. The 11% difference in the percentage of revision-oriented comments still evident in Figure 4 may be largely explained by the greater percentage of evaluation comments made by the traditional group. As a number of the evaluation comments made by both groups were positive in nature, they did not encourage revision.

Research Question Two: Does the use of a technology-enhanced synchronic interaction mode (MOO interaction) versus a traditional synchronic interaction mode (face-to-face interaction) result in a differential distribution of peer comments in both areas (global versus local), types (evaluation, clarification, suggestion, alteration), and the nature of each type of comment (i.e., revision-oriented versus non revision-oriented)?

Although most instructors who use peer review in their classes have two components in the process (i.e., written comments and later face-to-face discussion about those comments), previous research typically only focuses on the written comments. In order to better reflect the peer review process used in the classroom, this study has the traditional group engage in a face-to-face classroom discussion of the peer comments and has the technology-enhanced group participate in a virtual discussion on a MOO regarding their reviewed papers. The following table reflects the overall quantity of comments between the face-to-face and MOO interaction in area, type, and nature.

As seen in Figure 5, the face-to-face interaction produced a larger number of comments overall, with a large majority of the comments being global in area. In the case of the MOO group, the comments were overwhelmingly global in nature, with only two local comments being made. The dominance of global questions in this environment is not surprising given the nature of the communication. As opposed to making written comments on their own time, the students were engaged in active communication—either face-to-face or electronically—and, because of this, were focused more on their peers than the papers they reviewed. Common comments in this situation focused on the thesis statement of the paper (i.e., I didn’t really understand what you [sic] thesis statement showed) or on large parts of the body of the paper (i.e., In paragraph number four you don’t really seem to have any point) rather than focusing on smaller details that would have required carefully examining the papers again.

Perhaps the most interesting type of communication seen during this stage of the peer review process did not exist in the earlier stage. As we coded the transcripts of the face-to-face and MOO conversations, we discovered a number of turns that did not seem to fall under any of our types (i.e., evaluation, clarification, suggestion, or alteration). When we examined these utterances as a group, we discovered that they all seemed to fulfill the function of conversation maintenance for the groups. As seen in Figure 6, this type of communication resulted in over 40% of the turns in the face-to-face group and nearly 70% of the turns in the MOO group.

Sometimes this took the form of greetings (i.e., Hey, what’s up), while others were used by students to “take the floor” (i.e, OK, lets talk about me). In the MOO group, these comments sometimes also dealt with issues of confusion or clarification (i.e., who r u talking to?). As demonstrated in Table 5, examining the nature of the turns with the conversation maintenance turns deleted for the data gives a different picture.

Table 5: The nature of the comments with and without conversation maintenance turns

While these maintenance comments do fulfill an important role in communication, they do not relate to the revision process itself, so this data is not included in the discussion of the comments by area and nature. The importance of this type of communication, however, must not be underestimated in terms of its impact on the peer review process. In the case of the MOO group, it is evident that the large number of maintenance turns resulted in a smaller number of the other types of comments overall. As the purpose of the MOO conversation was to provide extra feedback on the papers and/or to reinforce the comments already made, the large amount of conversation maintenance likely diminished the effectiveness of this feedback in comparison to the face-to-face group. As seen above, even when the conversation maintenance turns are deleted for each group, a much larger percentage of the comments made by the traditional group were revision-oriented in comparison to the technology-enhanced group.

While the quantitative data in the above tables reveals that the face-to-face peer review produced more comments in general, and more revision-oriented comments in particular (as seen in Figure 7), this quantified data is not sufficient to capture the essence of the differences in interaction without observing how the students made the comments in group interaction. Our observations indicate that the face-to-face peer review group heavily relied on the peer review sheets and the peer papers while communicating their ideas. Sometimes they referred back to their written comments, read part of it, or reinforced the ideas they had conveyed briefly on the peer review sheets. They tended to skip over the local points (e.g., spelling problems) that were specified on their written sheet. They also seemed to be more structured in their commenting by following a set of questions given by the instructor.

The MOO group, on the other hand, seemed to refer to their previous written comments very little during the in-class interaction. This difference has several explanations. First, this group made their peer review comments on disc by using the Word features discussed previously. This means that some students did not have a hard copy of their comments in front of them to refer to. Therefore, while they were doing the MOO interaction they simply had to rely on their memories to offer comments. Secondly, even those students who did print out a hard copy of their comments or had a copy saved on disc were unable to rely on them heavily during the interaction because they had to focus their attention on the comments scrolling on the screen. Many students had difficulty in reading the comments on the screen and typing in their own comments and simply did not have time to refer back to their own notes if they had them. Many students also had some difficulty keeping up on the improvised conversation due to deficiencies in typing skills. Some students had to look at the keyboard as they typed, which, once again, made it impossible for them to refer to any written notes they might have used otherwise. This group also seemed to have much less structure in their time management and logical topic progression. The following example illustrates some of the problematic issues in the MOO group’s conversation:

This segment, which took place approximately five minutes into the discussion of the paper by one of the students, Vayomi, immediately demonstrates several difficulties experienced in the MOO format. First, Michael, as seen in line 2, had difficulty logging on to the MOO and arrived after the other students had already begun the discussion. This resulted in a disruption of the conversation as they greeted each other. In addition, in this 11-line segment none of the comments are revision-oriented, and only two of them in some way refer to Vayomi’s paper at all, with the rest being conversation maintenance. Of the comments related to the paper, one is a request for clarification from Vayomi, who did not understand an earlier comment (line 3), and the other is a joke by the newly arrived Michael, which may very well have caused confusion and/or discomfort for Vayomi (line 9). This is because it is not immediately clear from the MOO interaction that Michael’s comment is a joke. In face-to-face interaction, on the other hand, intonation, facial expressions, and body language would probably have made this very clear. The MOO interaction, then, often lacks the contextual clues present in face-to-face interaction potentially resulting in communication breakdowns (Kasper & Kellerman, 1997).

Obviously, the findings of peer review interactions are not consistent with those discussed earlier in this paper from the written peer review data. When making comments by using the Word editing feature, the technology-enhanced group made more comments overall, and more revision-oriented comments, but their follow-up MOO interaction did not seem to reinforce their written comments, and the much reduced amount of revision-oriented comments in the MOO interaction tended to be less effective than traditional face-to-face interaction because of the several difficulties identified above. The inconsistency between written comments and the MOO interaction in the technology-enhanced group could also be explained by the novelty factor. On the other hand, the traditional group seemed to greatly take advantage of the face-to-face interaction by clarifying and reinforcing a lot of the comments they made earlier in their peer review sheets.

Research Question Three: To what extent are the peer comments made (in both area and type) in the different commenting and interaction modes taken into consideration by the students in their revisions?

As discussed above, the peer review process—as performed in most writing classes—consists of a combination of written feedback (as discussed in research question one) and later conversation (as discussed in research question 2) about the paper in question. When examining actual revisions made on second drafts, then, it is important to take both of these steps of the peer review process into account. Therefore, research question three combines the comments made during the written and conversational portions of the peer review process, as shown in Figure 8:

As demonstrated in Figure 8, the technology-enhanced group made altogether 316 comments during the two stages of the peer review process (Word versus Paper and MOO versus face-to-face) while the traditional group made only 180 comments in both areas across the four types. Moreover, the technology-enhanced group had a larger number and percentage of revision-oriented comments than the traditional group (92% versus 76%). Even though the percentage of overall revisions in the technology-enhanced group is lower than that in the traditional group (27% versus 41%), the overall number of revisions in the technology-enhanced group is still larger. This indicates that the technology-enhanced peer review format is advantageous to producing more comments overall and a greater percentage of revision-oriented comments.