PLEASE ALSO SEE THE SELF RESEARCH GROUP'S WEBSITE AT www.self.ox.ac.uk
Currently Funded Research Grants
ESRC Professorial Fellowship: 2008-2011. Value Adding In Diverse Educational Contexts: Substantive-Methodological Synergies That Address Complex Issues With Sophisticated Methodology (Marsh). (click here for more information)
ESRC Grant (RES-000-22-1904): 2006-2008. Effectiveness of Self-concept Intervention Studies and a Comparison of Traditional and Multilevel Methods of Meta-Analysis (Marsh & O’Mara). (click here for a description of the proposed research, or here for the Executive Summary of the completed project - NEW!)
ESRC Researcher Development Initiative (RES-035-25-0045): 2008-2009. A Multilevel Approach to Meta-Analysis for the Social Sciences (Marsh, O’Mara & Malmberg). (click here for more information)
UK Higher Educational Academy & Higher Education Funding Council for England: 2007-2008. National Student Survey of UK Universities: Benchmarking Universities (Marsh & Cheng). (click here for more information)
Research Councils UK (RCUK): 2007-2012. Academic Fellowship £125,000 partial funding of fellowship, leading to permanent post if 5-year period successful. (Malmberg).
ESRC Grant: 2008-2009. Children's psychological adjustment and fathers' residence, parenting and traits (co-application by Malmberg & Flouri from the IOE, University of London).
ESRC CASE Studentship: 2008-2012. New Value-Added Models in Education: Using Latent-variable Multilevel Models to Solve Longstanding Biases (Marsh and Pearson Research and Assessment). (click here for more information)
ESRC grant (RES-000-22-2960): 2008-2009. Relations Between Specific and Global Domains of Self-concept: A Substantive-Methodological Synergy (Marsh & Scalas). (click here for more information)
Research Grants
Relations Between Specific and Global Domains of Self-concept: A Substantive-Methodological Synergy
ESRC small grant (RES-000-22-2960): 2008-2009
This project examines, in two studies, the relations between specific (physical self-concept, academic self-concept, spiritual self-concept) and global domains (self-esteem) of self-concept using a new latent-variable approach.
This issue is critical in that enhancing self-concept is a major goal in most social science disciplines, and specifically in education, since low self-concepts are related to depression, poor school performance, educational aspirations, and a range of social problems. Positive self-concepts are both relevant outcomes and mediating variables that facilitate a range of desirable outcomes.
One of the most central questions in self-concept research is how effects of multidimensional, specific components of self (e.g., physical, social, academic) on global self-esteem depend on framing factors such as ideal standards and the importance of specific components. Two mechanisms have been hypothesized to affect global self-esteem:
1) the difference or discrepancy between the perception of actual self and ideal standards in specific domains,
2) and the interaction between the actual perception and the importance that particular components of self have for an individual.
However, rigorous empirical research has provided weak support for both of these models. For this reason we propose advanced methodologies to study this problem.
The research has the potential to positively contribute to: a) self-concept theory, by providing answers to central but still unresolved questions that have plagued social science researchers for more than a century; b) methodology, by developing and applying new latent-variable approaches to models based on frame of reference factors that have broad applicability to the social sciences; and c) policy/practice, by providing background information to develop self-concept enhancement programmes that could benefit from information about gender and group differences, but also programmes oriented to prevent disorders and unhealthy behaviours connected with unrealistic ideals or concerns, such as eating disorders.
A Comparison of Traditional and Multilevel Methods of Meta-Analysis Through Simulated and Real Data
End of Award EXECUTIVE SUMMARY
Research reported was supported by a grant from the United Kingdom Economic and Social Research Council (ESRC RES-000-22-1904).
Meta-analysis is a valuable statistical technique for reviewing the available research literature on a particular topic, owing to its comprehensive and relatively unbiased approach to analysis. The purpose of meta-analysis is to synthesise results from multiple studies to observe effect sizes across those studies, with the aim of gaining a greater understanding of related research reports. It involves a synergistic combination of quantitative (representing effect sizes from each study with a standardised metric) and qualitative (classification of study characteristics as moderator variables) methods. Meta-analysis has broad appeal across the social sciences, is becoming the method of choice for academic review articles, and is increasingly the basis of policy in education, welfare, health and other professions.
Methodological approaches to meta-analysis are being developed faster than they can be incorporated into practice. There are at least three main families of meta-analytic models in current use: fixed effects, random effects, and multilevel models. However, meta-analysts frequently use methods that may not be appropriate for the data that they are considering, either because they are not aware of the other models, are unclear about their relative advantages, or are not comfortable with using the techniques. Using inappropriate models can lead to incorrect results, thereby drawing into question the conclusions of such research – as illustrated in our reanalyses of previous meta-analyses.
The present research is a methodological-substantive synergy. From a methodological perspective it has two overarching goals: (1) to test the apparent advantages of the latest meta-analytic model—a multilevel modelling approach, and (2) to provide meta-analysts with some guidance in selecting meta-analytic models. These two overarching goals can best be achieved through comparative analyses of both simulated data and real data. From a substantive perspective, we demonstrate the application of new multilevel approaches to meta-analysis in the two substantive applications, providing new insights and correcting misleading interpretations based on previous meta-analysis research.
Our research project can conveniently be divided into three studies. Study 1 has a methodological focus in which the new method of multilevel model meta-analysis is tested under different statistical conditions using simulated data with multilevel, fixed effects, and random effects models. Study is was more substantive in nature, applying techniques described in Study 1 to analyse the self-concept intervention literature. Study 3 is substantive-methodological, a reanalysis of the original Bornmann, Mutz, and Daniel (2007) meta-analysis showing problems in their original study and systematically comparing fixed, random, and multilevel approaches to meta-analysis that led to the resolution of these problems. The three studies combined represent an innovative, synergistic blend of methodological and substantive research interests.
In Study 1, the multilevel meta-analysis approach was compared with the traditional meta-analytical approaches, known as fixed effects and random effects models, for dealing with data with multiple outcomes. The results of the simulation study suggest that the multilevel approach is in general superior to the fixed-effects and random effects approaches, as it provides good estimates of the parameters (i.e., overall mean effect size and their standard errors). Interestingly, the inclusion of the covariance between the two outcomes (the cornerstone of the multivariate multilevel approach; Kalaian & Raudenbush, 1996) did not seem to improve the estimation of the parameters above the estimation of the multilevel model without the inclusion of the covariance value. If these correlations are available, researchers should include them in the analysis. However, these correlations typically are not available. In such cases, based on results of our simulation, we recommend that researchers fix the correlation at a conservative value and conduct a sensitivity analysis to establish that the estimated effect sizes and differences between effect sizes do not vary as a function of the size of the correlation. For this situation, our simulation establishes that for the conditions we considered the results are robust and unlikely to vary by a substantively important amount when researchers do not have access to these correlations. Thus, the multilevel approach without including the covariance term seems to be sufficient for calculating accurate estimates. In general, all models performed better when the number of studies included in the meta-analysis was larger (100 included studies compared to 20 included studies).
In Study 2, the substantive emphasis was on exploring what factors contribute to the effectiveness of self-concept interventions, as well as issues in evaluating self-concept interventions. In particular, the multidimensionality of the self-concept construct was examined through a construct validity approach. Due to statistical limitations in traditional approaches to meta-analysis, two previous meta-analyses of self-concept interventions (Haney & Durlak, 1998; Hattie, 1992) have included only a single effect size for each intervention study; typically a global measure of self-esteem or an average of different components of self-concept. This approach is consistent with a unidimensional approach to self-concept, but is antithetical to our multidimensional approach, which states that self-evaluations can differ from context to context. We hypothesize that previous meta-analyses underestimated the effectiveness of interventions and, perhaps, did not provide accurate advice on how best to improve self-concept in particular areas relevant to the aims of the intervention. As predicted, we found that interventions targeting specific self-concept domains and then measuring those domains with a relevant instrument (e.g., a mathematics intervention evaluated in relation to mathematics self-concept) yielded the larger effect sizes than interventions evaluated in relation to a total self-concept score (global self-esteem or an average of specific components that confounded the effects of more and less relevant domains) or domains not relevant to the intervention. We also found that interventions are most successful if they include some form of praise or positive feedback, and that the effects of the intervention were still present at follow-up testing. This research has ramifications for improving self-concept and self-esteem through organised programmes, as well as for self-concept researchers designing evaluation studies.
Study 3 is a meta-analysis of gender differences in peer reviews of research grant applications by external funding agencies. Peer review is highly valued in academia, but is widely criticized in terms of potential biases—particularly in relation to gender. In Study 3 we evaluate gender differences in peer reviews of grant applications based on an extension of the Bornmann, Mutz, and Daniel (2007) meta-analysis concluding that women are disadvantaged. We begin by contrasting their results with apparently the most comprehensive single primary study (Marsh, Jayasinghe, & Bond, 2008) showing that applicant gender has no effect on peer reviews of grant applications. In contrast to Bornmann et al.’s original interpretation, application of our new multilevel approach to meta-analysis to their data showed that gender differences varied significantly in relation to country, discipline and especially the type of application, but not publication year. Of particular importance, consistent with a priori predictions based on theory and the Marsh et al. primary study, there were no gender differences for the 40 (of 66) effect sizes based on grant proposals. This lack of gender effect for grant applications was very robust, generalizing over country, discipline, and publication year. In Study 3 we juxtapose the strengths and weaknesses of different meta-analysis models and the complementarity of meta-analyses with comprehensive primary studies.
The methodological focus of our research programme was on the most appropriate ways to evaluate meta-analysis data. In pursuing this goal we juxtaposed classic approaches to meta-analysis (fixed- and random-effect models) with new and evolving multilevel approaches. Clearly there are important advantages in the application of multilevel models and many instances in which the classic approaches are inappropriate in ways that might seriously distort interpretations of the meta-analysis results. Nevertheless, because different multilevel algorithms apparently behave differently and can even lead to different conclusions, we found it substantively (and pedagogically) useful to compare and contrast the results of different approaches to meta-analysis. We concluded by reviewing a variety of methodological issues in the conduct of meta-analysis for which more research is needed to guide applied meta-analysts, relating these issues to our substantive results and appropriate application of multilevel models.
In summary, the juxtaposition of the three studies demonstrates the value of substantive-methodological synergies, bringing together important new methodological advances in the application of meta-analysis (Study 1) to address substantively important issues (Studies 2 & 3).
Further Information
Detailed report of the simulation study: O’Mara, A. J., & Marsh, H. W., (2008). Incorporating within-study correlations in multivariate meta-analysis: Multilevel versus traditional models. Unpublished manuscript.
Course materials on conducting the multivariate multilevel model approach to meta-analysis: http://www.education.ox.ac.uk/research/resgroup/self/training.php.
Presentation of the results of the self-concept intervention meta-analysis: O'Mara, A. J., & Marsh, H. W. (2007, April). An application of multilevel modelling to meta-analysis and comparison with traditional approaches. Presented at the 6th International Amsterdam Multilevel Conference, University of Utrecht, Amsterdam, The Netherlands.
Detailed report of the gender differences in peer review meta-analysis: Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H-D., & O’Mara, A. J. (2008). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis. Unpublished manuscript submitted for publication.
References
Bornmann, L., & Mutz, R. & Daniel, H. D. (2007). Gender differences in grant peer review: A meta-analysis, Journal of Informetrics, 1(3) 226-238.
Haney, P., & Durlak, J.A. (1998). Changing self-esteem in children and adolescents: A meta-analytic review. Journal of Clinical Child Psychology, 27, 423-433.
Hattie, J. (1992). Self-concept. Hillsdale, NJ: Erlbaum.
Kalaian, H. A. & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1, 227-235.
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Peer reviews of grant applications: Effects of applicant and assessor gender. Unpublished manuscript submitted for publication.
This document is available for download at the ESRC Society Today
Effectiveness of Self-concept Intervention Studies and a Comparison of Traditional and Multilevel Methods of Meta-Analysis (Marsh & O’Mara)
(RES-000-22-1904): 2006-2008
Are self-concept enhancement interventions effective? Are traditional meta-analysis approaches to reviewing the self-concept intervention literature appropriate for addressing this question? To answer these questions, we proposed a new multilevel approach to meta-analysis to test a multidimensional theoretical model of self-concept intervention studies. The proposed research represents an innovative, synergistic blend of methodological and substantive research interests. The substantive emphasis is on exploring what factors contribute to the effectiveness of self-concept interventions, as well as issues in evaluating self-concept interventions. In particular, the multidimensionality of the self-concept construct will be examined through a construct validity approach. The methodological emphasis is on differences between traditional meta-analytic models and a new multilevel approach developed as part of the proposed research. These different approaches will be compared for both real (self-concept intervention) and artificial (simulated) data to evaluate implications of violations of statistical assumptions underlying meta-analysis that are particularly relevant for multidimensional constructs like self-concept. It will have important implications for policymakers and practitioners—those with a particular interest in self-concept and self-esteem, as well as potential consumers of meta-analyses, which are increasingly being used as a basis for public policy development.
Due to statistical limitations in traditional approaches to meta-analysis, two previous meta-analyses of self-concept interventions have included only a single effect size for each intervention study; typically a global measure of self-esteem or an average of different components of self-concept. This approach is consistent with a unidimensional approach to self-concept, but is antithetical to our multidimensional approach. We hypothesize that these meta-analyses underestimated the effectiveness of interventions and, perhaps, did not provide accurate advice on how best to improve self-concept in particular areas relevant to the aims of the intervention.
This ESRC grant is a substantive-methodological synergy. The substantive issues include evaluating the effectiveness of self-concept interventions, identifying the best strategies for enhancing self-concept, and elucidating the multidimensional structure of self-concept using a construct validity approach. Methodologically, we are developing and extending meta-analytic methodology through the innovative comparison of traditional (random and fixed effects) and multilevel methods of meta-analysis, using both simulated and real (self-concept intervention) data.
Return to top
A Multilevel Approach to Meta-Analysis for the Social Sciences (Marsh, O’Mara & Malmberg)
ESRC Research Development Initiative (RES-035-25-0045): 2008-2009
Meta-analysis is a quantitative approach to systematic review. It has broad appeal across the social sciences, is becoming the method of choice for academic review articles, and is increasingly the basis of policy in education, welfare, health, and other professions. It involves a synergistic combination of quantitative (representing effect sizes from each study with a standardised metric) and qualitative (classification of study characteristics as moderator variables) methods. Because this is a rapidly evolving area, there is limited training in the UK, particularly using the latest multivariate multilevel modelling (MLM) approach. To meet this training gap, we proposed a programme to be delivered in England (Oxford) and Scotland (Stirling).
The Course
What is meta-analysis? A brief (non-technical) introduction to meta-analysis will be presented at the 2008 NCRM Research Methodology Festival (RMF).
1-day introductory seminar (Oxford). A brief, highly accessible introduction to meta-analysis for social researchers and students who wish to engage with meta-analytic research, but have limited experience with it. Aims are to provide an introduction to the topic, conceptual understanding of different methods/procedures, instructional materials for research methodology teachers to incorporate into their courses, and prerequisite preparation for the subsequent workshop. An abridged version of this seminar will be presented as a workshop for the 2008 NCRM Research Methodology Festival (RMF).
3-day workshop (Oxford & repeated in Stirling). Designed to develop basic and advanced meta-analysis skills. It is heavily practical (computer-based), using SPSS and MLwiN with social science based examples. The materials (Powerpoint presentations, examples, references) also allow research methods instructors to incorporate meta-analysis into existing courses (train-the- trainer). Specific topics include calculating effect sizes, homogeneity testing, determining predictors and moderators, different approaches (fixed vs. random effects, multilevel models), interpreting the results, and special issues (e.g., publication bias).
Advanced seminar (Oxford). This will be conducted by an international expert on advanced multilevel meta-analysis procedures, Prof Joop Hox (author of Multilevel Analysis: Techniques and Applications). It is aimed at experienced meta-analysis researchers, and those who have attended the 3-day workshop.
Anticipated Benefits
Researcher support will include a depository of meta-analysis learning materials (e.g., slides, texts, presentations, annotated reading lists, selected publications, and software and data analysis examples). An emphasis will be placed on developing materials that attendees can use to train others in the use of the method, and that can be integrated into existing courses. An electronic mailing list will also be established, for continued collegial support.
Synergy. Our proposal brings together a unique synergy between various ESRC supported activities. The training programme itself is based on our current ESRC grant to evaluate our multivariate MLM approach to meta-analysis with MLwiN. Collaboration between our team and the Centre for Multilevel Modelling builds on existing strengths and ESRC's commitment to make MLwiN widely available (e.g., we will develop a multilevel meta-analysis module for their Centre website, thus ensuring worldwide exposure). Also, through collaboration with the EPPI Centre for systematic review, we ensure that our materials complement related materials developed by them, and provide an appropriate extension of their work.
National Student Survey of UK Universities: Benchmarking Universities (Marsh & Cheng)
UK Higher Educational Academy & Higher Education Funding Council for England: 2007-2008
Universities throughout the world are undertaking benchmarking exercises in which they compare themselves to other universities on appropriate indices in order to establish their current levels of performance and to initiate continuous self-improvement. In order to pursue benchmarking exercises, there is a need for a comprehensive set of benchmark indicators that focus on outcomes; measure functional effectiveness rather than criteria that are easily countable; are systematically developed so as to have good content (and “face”) validity; and differentiate between universities (or academic units within universities) so as to provide appropriate standards as a basis of ascertaining excellence and continuous improvement.
Graduating university students in the UK evaluate their undergraduate programme on the multiple components of the National Student Survey (NSS). We (in collaboration with HEA and HEFCE) are currently conducting preliminary research to evaluate psychometric properties of the first year of NSS responses. Although the NSS is being completed by 200,000 students from some 150 UK universities each year, there has not been a proper CFA to evaluate its factor structure at the student level (L1), nor multilevel analyses to assess the quality of measures at the levels of the university (L3) and discipline within the university (L2) for benchmarking results published in the Times Good University Guide. This study will extend these analyses to include data for three successive years, evaluating better estimates of the reliability of university rankings, whether different NSS components of university experience can be differentiated at the university level, stability of rankings over time, and potential biases. This study will also relate NSS ratings to Research Assessment Exercises (RAEs) of departments and universities, rankings from quality assurance exercises, audits by professional organizations, student/staff ratios, infrastructure spending, graduate destinations, and completion rates.
This study will provide a critical new multilevel (at the level of university and department) perspective on our controversial research showing that the relation between teaching effectiveness (at the level of the individual academic) and research productivity is almost exactly zero (also see UK Government White Paper on the Future of Higher Education at http://www.dfes.gov.uk), and test whether this result holds in multilevel analyses at the department and university levels.
Value Adding In Diverse Educational Contexts: Substantive-Methodological Synergies That Address Complex Issues With Sophisticated Methodology
ESRC Professorial Fellowship
Complex substantive issues require sophisticated methodologies. Bringing the two together creates a powerful synergy that will make important contributions in understanding critical issues in education today. In a special issue of Contemporary Educational Psychology, Marsh and Hau (2007) drew attention to the dearth of quantitative skills in educational research and argued that substantive-methodological synergies are needed for two main reasons. First, they bring together research in important complex issues and sophisticated, methodology appropriate to the task. Second, they provide models of good research, set standards for the discipline, demonstrate the potential of this approach and thus, motivate researchers to develop the requisite skills to use, or at least be conversant with, advanced quantitative methods.
Educational research is inherently multilevel, in which students are nested in classrooms, classrooms and teachers in schools, schools within districts, and districts within countries. Traditional single level statistics are therefore inherently biased and ignore contextual effects associated with group-level attributes (e.g., the motivational climate of the class). Using a new technique (multilevel latent variable approach) developed as part of this proposal, the Fellowship will provide insights to critical, unresolved educational issues such as:
* Systematic biases, in favour of schools with initially more able students, in value-added models of school and teacher effectiveness used to construct league tables;
* Benchmarking UK universities and departments within universities based on national student satisfaction surveys (published in the Times Good University Guide), and the relation between these teaching programme indicators and research assessment criteria;
* Cross-national comparisons of student engagement in science and educational achievement based on international OECD/PISA data; and
* Classroom/school climate / characteristics of the school, classroom, or teacher that contribute to quality of education and student performance that cannot be explained by individual student characteristics.
These studies not only make important advances in both substantive and quantitative aspects of educational research, but also provide a model for UK educational researchers of appropriate substantive-methodological synergies. Hence, the overarching agenda is to use this research programme to advocate substantive-methodological synergies, demonstrate their power to address diverse and complex issues of practical significance, provide "best practice" models of this approach, and thus assure researchers (particularly doctoral students and early career researchers) of the need to develop appropriate methodological skills. This is to be accomplished, in part, through a diverse set of capacity building strategies (e.g., academic publications; workshops; special sections of national/international conferences, monographs, and journals; research training/supervision; website training modules).
New Value-Added Models in Education: Using Latent-variable Multilevel Models to Solve Longstanding Biases
ESRC CASE Studentship
Statistical “value added” models are widely used to assess school/teacher effectiveness and to construct league tables after controlling for pre-existing student characteristics, but are highly contentious. The proposal demonstrates important biases in traditional value-added models (overestimating the effectiveness of schools with initially more able students) and tests proposed solutions, extending a recent integration of new developments in multilevel modelling, latent variable models, and contextual analysis (Ludtke, Marsh, et al., 2007). The proposal is thus a substantive-methodological synergy, applying and extending “state of the art” methodological developments to solve important substantive issues in educational research and policy.
Aims/Research Hypotheses
Study 1:
1. Demonstrate the Phantom Effect (unreliability in student test scores result in biased estimates in traditional value added models of school-level effectiveness in favour of schools with initially more able students).
2. Test the research hypothesis that the new multilevel latent-variable (MLLV) contextual models will eliminate the Phantom Effect.
Study 2:
1. Demonstrate the Matthew Effect (Traditional value-added models control pretest levels of student achievement but not growth trajectories. Initially better students achieve at a higher level, but are also on a steeper growth trajectory. Traditional value-added models that do not control for trajectories, are systematically biased in favour of schools with more able intakes).
2. Test the research hypothesis that the new MLLV contextual models will eliminate the Matthew Effect.