Now that you have watched the video, read the relevant sections of the papers by Hoferichter & Jentsch (2024) and Kisida et al. (2020) refer to the EEF’s guidance notes, and make a judgement about the level of threat posed by experimental effects and contamination in these experiments. Record your judgement in the security rating template, and note down any supporting information.
Open the accordion below to compare your judgement with that of an experienced rater.
The relevant information can be found on page 2445.
No information is provided in the report to explore the possibility of contamination or other experimental effects. However, the fact that participants in the experimental group self-selected to take part in a seminar series designed to help them with resilience, self-efficacy, and emotion regulation means that we cannot rule out the possibility that they were differently invested in the nature of the intervention compared to participants who chose other seminars and therefore may have engaged in ways that reflect this investment.
Therefore, this study is considered as having a high risk of bias for Confounding.
The relevant information can be found on page 8.
“We cannot be sure if the program itself is entirely responsible for the changes we observe. It could also be the case that the experience had positive spillover effects that motivated teachers to spend more time covering the content in their regular classroom setting.” (Kisida, et al. 2020, p. 8).
The use of waitlist controls is also understood to introduce bias associated with the behaviour of control group while it waits (see, for example, Patterson et al. 2016). Knowing that it will get the intervention later, the control group might delay engagement with relevant interventions that, in the absence of its participation in an experiment, it might otherwise engage in.
Both these points are speculative, but in any case no information was collected and/or reported about what control groups did and whether the treatment group modified usual practice as a result on participation in the treatment.
Therefore, this study is considered as having a high risk of bias for Experimental Effects and Contamination.
