Do experience and text quality matter for raters' decision-making behaviors?

Basit Öğe Kaydı
dc.authorid0000-0002-6948-0423en_US
dc.contributor.authorŞahan, Özgür
dc.contributor.authorRazi, Salim
dc.date.accessioned2021-03-20T20:12:20Z
dc.date.available2021-03-20T20:12:20Z
dc.date.issued2020
dc.departmentBTÜ, Yabancı Diller Yüksekokuluen_US
dc.description.abstractThis study examines the decision-making behaviors of raters with varying levels of experience while assessing EFL essays of distinct qualities. The data were collected from 28 raters with varying levels of rating experience and working at the English language departments of different universities in Turkey. Using a 10-point analytic rubric, each rater voice-recorded their thoughts through think-aloud protocols (TAPs) while scoring 16 essays of distinct text qualities and provided brief score explanations. Data collected from TAPs were analyzed by using a coding scheme adapted from Cumming, Kantor, and Powers (2002). The results revealed that text quality has a larger effect than rating experience on raters' decision-making behaviors. In addition, raters prioritized aspects of style, grammar, and mechanics when rating low-quality essays, but emphasized rhetoric and their general impressions of the text for high-quality essays. Furthermore, low-experienced raters differed more in their behaviors while assessing scripts of distinct qualities than did the medium- and high-experienced groups. The findings suggest that raters' scoring behaviors might evolve with practice, resulting in less variation in their decisions. As such, this research provides implications for developing strategy-based rater training programs, which might help to increase consistency across raters of different experience levels.en_US
dc.description.sponsorshipInternational Research Foundation for English Language Education (TIRF)en_US
dc.description.sponsorshipThe authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The first author received financial support for the research from The International Research Foundation for English Language Education (TIRF; tirfonline.org) as a doctoral dissertation grant (DDG).en_US
dc.identifier.doi10.1177/0265532219900228en_US
dc.identifier.endpage332en_US
dc.identifier.issn0265-5322
dc.identifier.issn1477-0946
dc.identifier.issue3en_US
dc.identifier.scopusqualityQ1en_US
dc.identifier.startpage311en_US
dc.identifier.urihttp://doi.org/10.1177/0265532219900228
dc.identifier.urihttps://hdl.handle.net/20.500.12885/490
dc.identifier.volume37en_US
dc.identifier.wosWOS:000509648000001en_US
dc.identifier.wosqualityQ1en_US
dc.indekslendigikaynakWeb of Scienceen_US
dc.indekslendigikaynakScopusen_US
dc.institutionauthorŞahan, Özgür
dc.language.isoenen_US
dc.publisherSage Publications Ltden_US
dc.relation.ispartofLanguage Testingen_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/closedAccessen_US
dc.subjectEFL writing assessmenten_US
dc.subjectraters' decision-making behaviorsen_US
dc.subjectrating experienceen_US
dc.subjecttext qualityen_US
dc.subjectthink-aloud protocolsen_US
dc.titleDo experience and text quality matter for raters' decision-making behaviors?en_US
dc.typeArticleen_US

Dosyalar

Koleksiyon

WoS İndeksli Yayınlar Koleksiyonu
Scopus İndeksli Yayınlar Koleksiyonu
Yabancı Diller Yüksekokulu Yayın Koleksiyonu