Multi-stage development and validation of an L2 prosody scoring rubric for Chinese learners of English

Prosodic features, such as lexical stress, intonation, and rhythm, have been found to be significant predictors of second language (L2) English speech comprehensibility and communicative success. Assessing prosody using some global criteria such as nativelikeness and comprehensibility can be inherently biased and too vague to provide detailed feedback to language learners. To address these limitations, this talk will present a multi-stage development of an L2 English prosody rubric with specific criteria which can provide feedback to language learners to act upon. The study triangulated the criteria by using different methods, including meta-analysis, retrospective verbal protocols, and psychometric analysis.

First, meta-analysis was conducted to identify prosodic features that were significantly correlated with overall L2 speech quality. Effect sizes (k = 494 from 101 primary studies) were pooled using multi-level meta-analysis models for each prosodic feature. Seven features were identified and selected as the criteria in the draft rubric, in reference to the autosegmental-metrical theory of intonational phonology. Then, the draft rubric was piloted by nine expert raters to assess L2 English prosody in 83 speech samples. Qualitative findings from the raters’ verbal protocols informed further revisions to the draft rubric. In the final stage, eleven raters applied the revised rubric to assess 332 speech samples, and their ratings were analysed by the many-facets Rasch modelling. Results from the psychometric analysis suggested that the revised criteria were conceptually relevant and psychometrically fit.

This study aims to develop an L2 English prosody scoring rubric with specific criteria to address limitations of commonly used global pronunciation assessment standards and have implications for training automated pronunciation systems. The talk will conclude with some philosophical discussions on the metaphysical possibility and necessity for the co-existence of language standards and phonetic variations in actual use of prosody.