Automated scoring

23rd May 2023 : 15:00 - 16:30

When humans score responses to open-ended test items, the process tends to be slow and expensive and potentially introduces measurement error due to the subjective nature of decision making.

Automated scoring is an application of computer technologies developed to address these challenges by predicting human scores based on item structure algorithms or response features. Unfortunately, there is no guarantee that a particular attempt to develop an automated scoring model will be successful because features of the assessment design may reduce the automated “scorability” of examinee responses to test items. Our presentation begins with an overview of automated scoring and scorability. We then describe two applications of automated scoring: Pearson’s Intelligent Essay Assessor (IEA) and Math Reasoning Engine (MRE). We continue by illustrating the concept of automated scorability, identifying features of prompts and scoring rubrics that we have found to either improve or reduce the chances of being able to model human scores on test items. Finally, we provide guidelines for item developers and rubric writers that facilitate the automated scorability of examinee responses.

Speakers

Lisa Eggers-Robertson is responsible for product planning and execution of the automated math open-ended response scoring service. She has served as program manager, supporting the College Board’s Accuplacer and SAT programs and other state program as well as a product manager for an interim classroom assessment product. She has spent the last 9 years of her career within the automated scoring team. She is certified as a Project Management Professional (PMP) by the Program Management Institute (PMI) and holds a BBA in Management of Information Systems from the University of Iowa.

Gregory M. Jacobs serves as technical lead and senior member of Pearson’s automated scoring team that is responsible for product planning and execution of the Intelligent Essay Assessor™ scoring service. He works with scoring staff and assessment program teams to provide leadership and execution of the automated scoring process and has over 3 years of industry experience building machine learning models, with an emphasis on Natural Language Processing. He also holds a Juris Doctor from Catholic University and has over a decade of legal experience as a commercial litigator handling complex contractual language disputes.

Edward W. Wolfe is responsible for development and delivery of Pearson’s automated writing and math scoring services, including the Intelligent Essay Assessor™, Continuous Flow, and Math Reasoning Engine. He works with scoring staff and assessment program teams to provide leadership and oversight of the automated scoring process on all programs. Dr. Wolfe holds a PhD in educational psychology from the University of California, Berkeley, and has authored nearly 100 peer-reviewed journal articles and book chapters regarding human and automated scoring and applied psychometrics