Nasim Ghanbari

It is widely believed that human rating performance is influenced by an array of different factors. Among these, rater-related variables such as experience, language background, and rater’s perceptions and attitudes have been mentioned. As one of the important rater-related factors is the way the raters interact with the rating scales. In particular, how raters perceive the components of the scales to further plan their scoring seems important. For this aim, the present study investigated the raters’ perceptions of the rating scales and their subsequent rating behaviors for two analytic and holistic scales. Hence, nine highly experienced raters were trained and later asked to verbalize their thoughts while rating student essays by applying IELTS holistic scale and the analytic scale of Jacobs et al. ESL Composition Profile (1981). Upon analyzing the think-aloud protocols, four themes emerged. The findings showed that, in the validation theme, the raters merely referred to the holistic scale components to validate their ratings while in the dominancy theme, the raters had a pre-evaluation scale reading to have more reliable rating. In terms of the analytic rating, the raters adopted the dominancy theme as well. Additionally, the results of a T-test indicate that the raters assigned significantly different scores to the texts using the holistic and analytic rating scales. On the whole, the results showed that the raters’ explicit interpretations of the scale components will help them make better quality evaluation of the strengths and weaknesses of the students’ texts. The study provides several implications for rater training programs and EFL writing assessment.