Now that you have watched the video, read the relevant sections of the papers by Hoferichter & Jentsch (2024) and Kisida et al. (2020) refer to the EEF’s guidance notes, and make a judgement about the level of threat posed by confounding in these experiments. Record your judgement in the security rating template, and note down any supporting information.
Open the accordion below to compare your judgement with that of an experienced rater.
The relevant information can be found on pages 2445 and 2449.
We have already observed that the way in which comparison groups were created suggests a high risk of bias because they were self-selecting. When we consider this in terms of possible confounders, the most obvious factor relates to the motivation of the participants. That is, students who chose to do this course rather than one of the others on offer were plausibly more motivated to learn about their own emotional regulation than participants who did not.
In the text on page 2445 and in Table 2 on page 2449, we see that the authors provide information about the proportion of male and female participants, the proportion on high school and elementary tracks, the distribution of age, and the average scores on the three outcome measures at baseline in each group, and that there are some small imbalances between groups on some of these measures but not others.
Therefore, this study is considered as having a moderate risk of bias for Confounding.
The relevant information can be found on pages 4, 5 and 6.
“To ensure baseline equivalence between the treatment and control groups, the 28 eligible applicant groups were stratified into pairs prior to randomization. Twelve school group applicants (six pairs) were stratified based on pairing adjacent grades within the same schools. An additional 14 applicant groups were stratified based on being from the same district and same grade, but in different schools. We then used school percent eligible for free or reduced-price lunch as an initial screen and school percent minority as a second screen to create these seven pairs. Finally, two school groups in different districts within the same grade with similar demographics were used to make one additional treatment-control pairing.” (Kisida et al., 2020, p. 4-5).
“The random assignment of classrooms to treatment and control conditions achieved comparable balance across observable characteristics for the 1,892 Grade 3 to 5 students in our sample (Table 1). Students were balanced across measures of gender, ethnicity, grade level, the number of previous cultural activities they had participated in outside of school (e.g., music lessons, dance lessons), and school percent eligible for free or reduced-priced lunch. Table 1, on page 6, provides descriptive statistics for known characteristics of the participants in each group. These data reassure us that groups were very similar on these measures.” (Kisida et al. 2020, p. 6).
The researchers stratified the randomisation procedure to minimise biased distribution of known characteristics such as geographical location, grades, socio-economic status, and minority representation, and demonstrated baseline equivalence of groups on observable characteristics thought to be important. However, we don’t know whether the randomisation sequence was concealed because they do not report how allocation was conducted.
Therefore, this study is considered as having a moderate risk of bias for Confounding.
