Nasim Ghanbari

Among different factors affecting writing assessment, rater and rating scale are two influential variables which determine the outcome of assessment. Taking this into account, this study attempted to identify and classify the raters’ behaviors in the Iranian EFL context when using analytic and holistic rating scales. For this aim, a body of nine expert raters were asked to verbalize their thoughts when rating student essays. They were also asked to do their rating using the analytic scale of ESL Composition Profile and IELTS holistic scale. Upon the qualitative analysis of think-aloud protocols (TAPs), two themes emerged which showed the raters’ behaviors when applying the rating scales. The findings further showed that when using the holistic scale, the raters read the text first to get an overall impression. Then they assessed the text based on their own criteria. Next, they referred to the scale for scoring and in the last stage they provided evidence for their scoring. On the other hand, when applying analytic rating scales, the raters first scanned the text for surface features. Then they read the text for their initial impression. Next, they read each scale component and its descriptor for scoring and finally, they attempted to provide evidence for their scoring. In addition to identifying the raters’ behaviors, the raters’ behaviors were classified. The findings imply that the diagnosis of the rater-rating scale interactions can unveil the strengths and weaknesses of the EFL rating process. This, in turn, can provide more quality training for the raters.