Multi-stage development and validation of an L2 prosody scoring rubric for Chinese learners of English
Export to calendarAbstract
Prosodic features, such as lexical stress, intonation, and rhythm, have been found to be significant predictors of second language (L2) English speech comprehensibility and communicative success. Assessing prosody using some global criteria such as nativelikeness and comprehensibility can be inherently biased and too vague to provide detailed feedback to language learners. To address these limitations, this talk will present a multi-stage development of an L2 English prosody rubric with specific criteria which can provide feedback to language learners to act upon. The study triangulated the criteria by using different methods, including meta-analysis, retrospective verbal protocols, and psychometric analysis.
First, meta-analysis was conducted to identify prosodic features that were significantly correlated with overall L2 speech quality. Effect sizes (k = 494 from 101 primary studies) were pooled using multi-level meta-analysis models for each prosodic feature. Seven features were identified and selected as the criteria in the draft rubric, in reference to the autosegmental-metrical theory of intonational phonology. Then, the draft rubric was piloted by nine expert raters to assess L2 English prosody in 83 speech samples. Qualitative findings from the raters’ verbal protocols informed further revisions to the draft rubric. In the final stage, eleven raters applied the revised rubric to assess 332 speech samples, and their ratings were analysed by the many-facets Rasch modelling. Results from the psychometric analysis suggested that the revised criteria were conceptually relevant and psychometrically fit.
This study aims to develop an L2 English prosody scoring rubric with specific criteria to address limitations of commonly used global pronunciation assessment standards and have implications for training automated pronunciation systems. The talk will conclude with some philosophical discussions on the metaphysical possibility and necessity for the co-existence of language standards and phonetic variations in actual use of prosody.
Teams link: AL lunchtime seminar: Yuanyue Hao | Join meeting in Teams | Microsoft Teams
Bio
Yuanyue Hao is currently a final-year doctoral student in the Department of Education, University of Oxford (just passed viva earlier April). His doctoral research was fully funded by the Swire Scholarship and recognised by Duolingo English Tests’ Doctoral Dissertation Awards. He has presented his research at AAAL, BAAL, LTRC, LTF, etc. With initial research focus on human-mediated pronunciation assessment, he has gradually developed his interests in automated pronunciation assessment, algorithmic fairness, and auditing algorithms. He is also interested in rigorous application of advanced quantitative methods in applied linguistic research, such as machine learning and causal inference. Moreover, he is interested in bringing philosophical discussions into language assessment and applied linguistic research.