Investigation of Rater Effects using Graph Theory and Exponential Random Graph Models: examples from high-stakes examination data

18th November 2024 : 12:45 - 14:00

Quantitative Hub Seminar

This presentation investigates rater effects in high-stakes examinations using Graph Theory (GT) and Exponential Random Graph Models (ERGM). Rater effects, such as biases and inconsistencies, are common challenges in the scoring process, impacting the fairness and validity of assessments. GT and ERGM offer powerful tools to visualize complex rater interactions and reveal hidden patterns that traditional reliability indices, such as Krippendorff’s alpha or Fleiss’s kappa, may not capture.

Using real-world examination data, we demonstrate how graph-based models provide nuanced insights into rater behavior, allowing for the identification of outlier raters and systematic biases. This approach is not intended to replace established methods (e.g., Rasch models) but rather to complement them.

Our empirical analysis shows that GT indices and Rasch model estimates are strongly correlated, highlighting their complementarity. Additionally, certain GT indices can be mathematically linked to Rasch model estimates through the pairwise estimation method, further strengthening their theoretical connection.

Speaker

Iasonas Lamprianou,