Now that you have watched the video, read the relevant sections of the papers by Hoferichter & Jentsch (2024) and Kisida et al. (2020) refer to the EEF’s guidance notes, and make a judgement about any attrition in these experiments. Record your judgement in the security rating template, and note down any supporting information.
Open the accordion below to compare your judgement with that of an experienced rater.
The relevant information can be found on page 2447:
“As is common in longitudinal designs, we observed a high dropout in our study (51%). Missingness was less likely in the intervention group (OR=0.2, SE=0.37, p < 0.001)…” Hoferichter & Jentsch (2024, p. 2447).
Attrition is greater than 50% in this study, and there is evidence of differential attrition (i.e. more participants in the control group dropped out than in the intervention group).
Therefore, this study is allocated 0 padlocks for Attrition.
The relevant information can be found on page 6.
“Data were successfully collected from all but one of the school groups that agreed to participate […] To maintain internal validity, this school group and its matched control group were dropped from the study.” (Kisida et al. 2020, p.6).
Per the EEF guidance, attrition should be measured at the pupil level regardless of the level of randomisation/allocation. The report does not state the exact numbers of students who were recruited nor how many were in the school group that did not return the assessments, nor the number in its matched control. However, we can do a bit of detective work to get an estimate of the number who dropped out. Table 1, on p. 6, tells us that the total number of students analysed was 1,892 (treatment group n = 895, control group n = 997). Assuming an average class size is approximately 27 children*, then we can estimate the original sample to be 1,946 (analysed sample + treatment class who dropped out + matched control, or 1892+27+27). This would put attrition at about 3%. However, there may have been more than one class in each of these schools as the reports says that some schools sent whole year groups. So, if we assume two classes from each of the treatment and control dropped out this would give an original sample of 2,000 and an overall attrition rate of about 5%. Even if there were three classes per year group, the overall attrition would be about 8%. In all cases, the attrition rates is lower than 11%, well within the lowest range cited in the EEF guidelines.
Therefore, this study is allocated 5 padlocks for Attrition.
* Average class size for primary schools in Arkansas is between 20.4 and 26.9 (National Center for Education Statistics, 2018), so we have taken the larger figure. https://nces.ed.gov/surveys/ntps/tables/ntps1718_fltable06_t1s.asp
