13. Calculating the Final Rating

Home > Oxford Education Deanery > Online Learning > Assessing Risk of Bias in Education Research > 13. Calculating the Final Rating

The formula for calculating the final padlock score is to take the initial rating then adjust for any further threats to internal validity. Scores can only be adjusted down, and only by a maximum of two padlocks. Adjustments are made on the basis of the following:

No adjustment made: Up to two threats are classified as Moderate Risk and the direction of the likely biases is unknown or operates in opposite directions.

Drop one padlock: Up to four threats are classified as Moderate Risk but the directions of biases are unknown; OR Up to two threats are classified as Moderate Risk with the same likely direction of bias; OR Up to one threat is classified as High Risk with all others deemed as Low Risk.

Drop two padlocks: One threat is classified as High Risk and two threats are classified as Moderate Risk OR; Two or more threats are classified as High Risk.

Now that you have watched the video and read the formula for calculating the final padlock score, review your judgements for Hoferichter & Jentsch (2024) and Kisida et al. (2020) and calculate the final padlock score for these experiments. Record your judgement in the security rating template.

Open the accordion below to compare your score with that of an experienced rater.

Answers

Hoferichter & Jentsch (2024)

Download a completed security rating template for Hoferichter & Jentsch (2024).

In the additional threats section of the template, you will see that four threats have been judged as presenting a high risk of bias, and two threats have been judged as presenting a moderate risk of bias. The formula states that when two or more threats are classified as high risk, two padlocks should be dropped from the interim rating. As the interim rating for Hoferichter & Jentsch (2024) was already 0, we cannot drop any further padlocks. Therefore, this experiment receives a final rating of 0 padlocks.

Kisida et al. (2020)

Download a completed security rating template for Kisida et al. (2020).

In the additional threats section of the template, you will see that two threats have been judged as presenting a high risk of bias, and one threat has been judged as presenting a moderate risk of bias. The remaining threats are all considered low. The formula states that when two or more threats are classified as high risk, two padlocks should be dropped from the interim rating. The interim rating for Kisida et al. (2020) was 5 padlocks, after dropping two padlocks we arrived at the final rating of 3 padlocks.

Next: Wrap Up