Now that you have watched the video, read the relevant sections of Hoferichter & Jentsch (2024) and Kisida et al. (2020), refer to the EEF’s guidance notes, and make a judgement about the designs of these experiments. Record your judgement in the security rating template, and note down any supporting information.
Open the accordion below to compare your judgement with that of an experienced rater.
The relevant information can be found on page 2445:
“The intervention was delivered as a seminar, which students were free to choose from a variety of classes offered at the Department of Educational Science on a regular basis (e.g., pedagogy, educational psychology, research methodology). There were no particular inclusion criteria to participate in the intervention. Fifty-four students participated in the intervention, and students from the same year that took any other class served as control group (n=131).” Hoferichter & Jentsch (2024, p. 2445).
From this we can see that participants self-selected into the intervention and control groups, suggesting a high risk of allocation bias, and that no measures were taken by the researchers to ensure comparability between those groups on known or unknown factors prior to allocation. Therefore, this study is allocated 1 padlock for Design.
The relevant information can be found on pages 4 and 5.
“Applicant groups were randomly assigned such that half of the groups were able to see the program Digging Up Arkansas during the current school year (the treatment group) while the other half were guaranteed a spot the following year (the control group).” (Kisida et al., 2020, p.4-5).
The researchers randomly allocated clusters to intervention and waitlist control groups.
Therefore, this study is allocated 5 padlocks for Design.
