journals systematic literature review

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

• 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
• 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
• 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
• PMID: 30089228
• DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

PubMed Disclaimer

Similar articles

• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Aromataris E, et al. Int J Evid Based Healthc. 2015 Sep;13(3):132-40. doi: 10.1097/XEB.0000000000000055. Int J Evid Based Healthc. 2015. PMID: 26360830
• RAMESES publication standards: meta-narrative reviews. Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. Wong G, et al. BMC Med. 2013 Jan 29;11:20. doi: 10.1186/1741-7015-11-20. BMC Med. 2013. PMID: 23360661 Free PMC article.
• A Primer on Systematic Reviews and Meta-Analyses. Nguyen NH, Singh S. Nguyen NH, et al. Semin Liver Dis. 2018 May;38(2):103-111. doi: 10.1055/s-0038-1655776. Epub 2018 Jun 5. Semin Liver Dis. 2018. PMID: 29871017 Review.
• Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals. Hedin RJ, Umberham BA, Detweiler BN, Kollmorgen L, Vassar M. Hedin RJ, et al. Anesth Analg. 2016 Oct;123(4):1018-25. doi: 10.1213/ANE.0000000000001452. Anesth Analg. 2016. PMID: 27537925 Review.
• Bridging disciplines-key to success when implementing planetary health in medical training curricula. Malmqvist E, Oudin A. Malmqvist E, et al. Front Public Health. 2024 Aug 6;12:1454729. doi: 10.3389/fpubh.2024.1454729. eCollection 2024. Front Public Health. 2024. PMID: 39165783 Free PMC article. Review.
• Strength of evidence for five happiness strategies. Puterman E, Zieff G, Stoner L. Puterman E, et al. Nat Hum Behav. 2024 Aug 12. doi: 10.1038/s41562-024-01954-0. Online ahead of print. Nat Hum Behav. 2024. PMID: 39134738 No abstract available.
• Nursing Education During the SARS-COVID-19 Pandemic: The Implementation of Information and Communication Technologies (ICT). Soto-Luffi O, Villegas C, Viscardi S, Ulloa-Inostroza EM. Soto-Luffi O, et al. Med Sci Educ. 2024 May 9;34(4):949-959. doi: 10.1007/s40670-024-02056-2. eCollection 2024 Aug. Med Sci Educ. 2024. PMID: 39099870 Review.
• Surveillance of Occupational Exposure to Volatile Organic Compounds at Gas Stations: A Scoping Review Protocol. Mendes TMC, Soares JP, Salvador PTCO, Castro JL. Mendes TMC, et al. Int J Environ Res Public Health. 2024 Apr 23;21(5):518. doi: 10.3390/ijerph21050518. Int J Environ Res Public Health. 2024. PMID: 38791733 Free PMC article. Review.
• Search in MeSH

LinkOut - more resources

Full text sources.

• Ingenta plc
• Ovid Technologies, Inc.

Other Literature Sources

• scite Smart Citations

Miscellaneous

• NCI CPTAC Assay Portal
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Systematic Reviews

Aims and scope.

The journal publishes high quality systematic review products including systematic review protocols, systematic reviews related to a very broad definition of human health, (animal studies relevant for human health), rapid reviews, updates of already completed systematic reviews, and methods research related to the science of systematic reviews, such as decision modelling. At this time Systematic Reviews does not accept reviews of in vitro studies. The journal also aims to ensure that the results of all well-conducted systematic reviews are published, regardless of their outcome. (You can now also prospectively register these reviews on PROSPERO.)

Trending articles

Click here to view which articles have been shared the most in the last month!

• Most accessed

Examining the effectiveness of food literacy interventions in improving food literacy behavior and healthy eating among adults belonging to different socioeconomic groups- a systematic scoping review

Authors: Arijita Manna, Helen Vidgen and Danielle Gallegos

Knowledge and practices of youth awareness on death and dying in school settings: a systematic scoping review protocol

Authors: Emilie Allard, Clémence Coupat, Sabrina Lessard, Noémie Therrien, Claire Godard-Sebillotte, Dimitri Létourneau, Olivia Nguyen, Andréanne Côté, Gabrielle Fortin, Serge Daneault, Maryse Soulières, Josiane Le Gall and Sylvie Fortin

Evaluating the effectiveness of large language models in abstract screening: a comparative analysis

Authors: Michael Li, Jianping Sun and Xianming Tan

Prevalence of human filovirus infections in sub-Saharan Africa: A systematic review and meta-analysis protocol

Authors: Christopher S. Semancik, Christopher L. Cooper, Thomas S. Postler, Matt Price, Heejin Yun, Marija Zaric, Monica Kuteesa, Nina Malkevich, Andrew Kilianski, Swati B. Gupta and Suzanna C. Francis

Patients’ experiences of mechanical ventilation in intensive care units in low- and lower-middle-income countries: protocol of a systematic review

Authors: Mayank Gupta, Priyanka Gupta, Preeti Devi, Utkarsh, Damini Butola and Savita Butola

Most recent articles RSS

View all articles

Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement

Authors: David Moher, Larissa Shamseer, Mike Clarke, Davina Ghersi, Alessandro Liberati, Mark Petticrew, Paul Shekelle and Lesley A Stewart

The PRISMA 2020 statement: an updated guideline for reporting systematic reviews

Authors: Matthew J. Page, Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li…

The Editorial to this article has been published in Systematic Reviews 2021 10 :117

Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study

Authors: Wichor M. Bramer, Melissa L. Rethlefsen, Jos Kleijnen and Oscar H. Franco

PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews

Authors: Melissa L. Rethlefsen, Shona Kirtley, Siw Waffenschmidt, Ana Patricia Ayala, David Moher, Matthew J. Page and Jonathan B. Koffel

Drug discontinuation before contrast procedures and the effect on acute kidney injury and other clinical outcomes: a systematic review protocol

Authors: Swapnil Hiremath, Jeanne Françoise Kayibanda, Benjamin J. W. Chow, Dean Fergusson, Greg A. Knoll, Wael Shabana, Brianna Lahey, Olivia McBride, Alexandra Davis and Ayub Akbari

Most accessed articles RSS

Sign up to receive article alerts

Systematic Reviews is published continuously online-only. We encourage you to sign up to receive free email alerts to keep up to date with all of the latest articles by registering here .

Peer review mentoring

The Editors endorse peer review mentoring of early career researchers. Find out more here

Article collections

Thematic series The role of systematic reviews in evidence-based research Edited by Professor Dawid Pieper and Professor Hans Lund 

Thematic series Canadian Task Force on Preventive Health Care Evidence Reviews Edited by Assoc Prof Craig Lockwood

Thematic series Automation in the systematic review process Edited by Prof Joseph Lau

Thematic series Five years of Systematic Reviews

View all article collections

Latest Tweets

Your browser needs to have JavaScript enabled to view this timeline

• Editorial Board
• Manuscript editing services
• Instructions for Editors
• Sign up for article alerts and news from this journal
• Follow us on Twitter

Annual Journal Metrics

Citation Impact 2023 Journal Impact Factor: 6.3 5-year Journal Impact Factor: 4.5 Source Normalized Impact per Paper (SNIP): 1.919 SCImago Journal Rank (SJR): 1.620

Speed 2023 Submission to first editorial decision (median days): 92 Submission to acceptance (median days): 296

Usage 2023 Downloads: 3,531,065 Altmetric mentions: 3,533

• More about our metrics

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Research Process
• Manuscript Preparation
• Manuscript Review
• Publication Process
• Publication Recognition
• Language Editing Services
• Translation Services

Systematic Literature Review or Literature Review?

• 3 minute read
• 58.2K views

Table of Contents

As a researcher, you may be required to conduct a literature review. But what kind of review do you need to complete? Is it a systematic literature review or a standard literature review? In this article, we’ll outline the purpose of a systematic literature review, the difference between literature review and systematic review, and other important aspects of systematic literature reviews.

What is a Systematic Literature Review?

The purpose of systematic literature reviews is simple. Essentially, it is to provide a high-level of a particular research question. This question, in and of itself, is highly focused to match the review of the literature related to the topic at hand. For example, a focused question related to medical or clinical outcomes.

The components of a systematic literature review are quite different from the standard literature review research theses that most of us are used to (more on this below). And because of the specificity of the research question, typically a systematic literature review involves more than one primary author. There’s more work related to a systematic literature review, so it makes sense to divide the work among two or three (or even more) researchers.

Your systematic literature review will follow very clear and defined protocols that are decided on prior to any review. This involves extensive planning, and a deliberately designed search strategy that is in tune with the specific research question. Every aspect of a systematic literature review, including the research protocols, which databases are used, and dates of each search, must be transparent so that other researchers can be assured that the systematic literature review is comprehensive and focused.

Most systematic literature reviews originated in the world of medicine science. Now, they also include any evidence-based research questions. In addition to the focus and transparency of these types of reviews, additional aspects of a quality systematic literature review includes:

• Clear and concise review and summary
• Comprehensive coverage of the topic
• Accessibility and equality of the research reviewed

Systematic Review vs Literature Review

The difference between literature review and systematic review comes back to the initial research question. Whereas the systematic review is very specific and focused, the standard literature review is much more general. The components of a literature review, for example, are similar to any other research paper. That is, it includes an introduction, description of the methods used, a discussion and conclusion, as well as a reference list or bibliography.

A systematic review, however, includes entirely different components that reflect the specificity of its research question, and the requirement for transparency and inclusion. For instance, the systematic review will include:

• Eligibility criteria for included research
• A description of the systematic research search strategy
• An assessment of the validity of reviewed research
• Interpretations of the results of research included in the review

As you can see, contrary to the general overview or summary of a topic, the systematic literature review includes much more detail and work to compile than a standard literature review. Indeed, it can take years to conduct and write a systematic literature review. But the information that practitioners and other researchers can glean from a systematic literature review is, by its very nature, exceptionally valuable.

This is not to diminish the value of the standard literature review. The importance of literature reviews in research writing is discussed in this article . It’s just that the two types of research reviews answer different questions, and, therefore, have different purposes and roles in the world of research and evidence-based writing.

Systematic Literature Review vs Meta Analysis

It would be understandable to think that a systematic literature review is similar to a meta analysis. But, whereas a systematic review can include several research studies to answer a specific question, typically a meta analysis includes a comparison of different studies to suss out any inconsistencies or discrepancies. For more about this topic, check out Systematic Review VS Meta-Analysis article.

Language Editing Plus

With Elsevier’s Language Editing Plus services , you can relax with our complete language review of your systematic literature review or literature review, or any other type of manuscript or scientific presentation. Our editors are PhD or PhD candidates, who are native-English speakers. Language Editing Plus includes checking the logic and flow of your manuscript, reference checks, formatting in accordance to your chosen journal and even a custom cover letter. Our most comprehensive editing package, Language Editing Plus also includes any English-editing needs for up to 180 days.

How to Make a PowerPoint Presentation of Your Research Paper

Step-by-Step Guide: How to Craft a Strong Research Hypothesis

You may also like.

Descriptive Research Design and Its Myriad Uses

Five Common Mistakes to Avoid When Writing a Biomedical Research Paper

Making Technical Writing in Environmental Engineering Accessible

To Err is Not Human: The Dangers of AI-assisted Academic Writing

When Data Speak, Listen: Importance of Data Collection and Analysis Methods

Choosing the Right Research Methodology: A Guide for Researchers

Why is data validation important in research?

Writing a good review article

Input your search keywords and press Enter.

Systematic Literature Review and Meta-Analysis Journal

The Systematic Literature Review and Meta-Analysis Journal is a multidisciplinary journal focused on the research articles, reviews and empirical research that has used Systematic Literature Review and Meta-Analysis (SLR-M) methods in their research. The journal aimed to facilitate the research in all fields of life until the SLR-M methods have been applied. 

Announcements

Chatgpt for systematic literature review: live session.

Date: 7th May 2023 Time: 6:00 PM Malaysia time Resource Person: Dr. Imran Qureshi Link:  https://bit.ly/chatgpt_slrm   YouTube Channel Link:  https://bit.ly/connectingasiatv  

Current Issue

Vol. 5 No. 2 (2024): June

Published: 2024-08-28

Efficacy and safety of treatments used in differentiated thyroid cancer (DTC): A systematic review

From profitability to sustainability: a comprehensive analysis of tractor services in nigeria's agricultural landscape.

• Download PDF
• CME & MOC
• Share X Facebook Email LinkedIn
• Permissions

Cost and Cost-Effectiveness of the Management Strategies of Chronic Urticaria : A Systematic Review

• 1 Department of Pharmaceutical Care, Faculty of Pharmacy, Chiang Mai University, Chiang Mai, Thailand
• 2 Pharmacoepidemiology and Statistics Research Center (PESRC), Faculty of Pharmacy, Chiang Mai University, Chiang Mai, Thailand
• 3 Division of Dermatology, Department of Internal Medicine, Faculty of Medicine, Chiang Mai University, Chiang Mai, Thailand
• 4 Division of Rheumatology, Allergy and Immunology, Department of Internal Medicine, College of Medicine, University of Cincinnati, Cincinnati, Ohio
• 5 Ottawa Hospital Research Institute, Ottawa Hospital, Ottawa, Ontario, Canada
• 6 Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
• 7 School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

Question   What are the costs and cost-effectiveness of chronic urticaria (CU) management strategies?

Findings   In this systematic review including 17 unique studies, based on 2023 US dollars, median annual cost estimations for CU ranged from $5621 to $6933. Biologic omalizumab, 300 mg, for H 1 antihistamine–refractory chronic spontaneous urticaria (CSU) appears to be cost-effective from a third-party payer’s perspective.

Meaning   In this review, biologic omalizumab, 300 mg, for H 1 antihistamine–refractory CSU offered good value for money in US health care services; price negotiations and rebates of high-cost drugs may help promote sustainability and more effective health care allocation for CU.

Importance   Although treatment for chronic urticaria (CU) has improved over the past decades, evidence regarding costs and net benefits associated with these treatment strategies have yet to be comprehensively characterized and synthesized.

Objective   To summarize the cost and cost-effectiveness of CU management strategies.

Evidence Review   An extensive systematic literature search of 6 databases (MEDLINE, Embase, PubMed Cochrane, Scopus, and CINAHL) and gray literature sources, without language restriction, was conducted and updated to March 23, 2024. Articles that performed cost analysis or full economic evaluation among patients with CU were included. Two reviewers independently extracted data, such as annual costs of health care services or incremental cost-effectiveness ratio (ICER) per quality-adjusted life-year (QALY). All monetary values were converted and inflated to 2023 US dollars. Evidence-based synthesis for health benefit was judged using the Evidence Rating Matrix by the Institute for Clinical and Economic Review.

Findings   Seventeen unique studies (11 cost analysis studies and 6 full economic evaluations) were included. With the wide variation in health care resources, services that included biologic omalizumab utilization had higher annual health care cost estimations for CU management than services that did not include omalizumab prescription (median [IQR] cost, $6933 [$5988-$8717] vs $5621 [$2488-$8754]). The biologic omalizumab, 300 mg, for H 1 antihistamine–refractory chronic spontaneous urticaria (CSU) (3 studies) was found to have a median (IQR) ICER of $89 005 ($36 058-$145 694) per QALY (evidence rating as incremental or better; moderate certainty with substantial net health benefit). Routine laboratory testing among patients with CSU with otherwise normal histories and physical examination findings (1 study) had ICERs ranging from $1 427 928 to $1 950 524 per QALY (evidence rating as comparable or inferior; moderate certainty that the net health benefit is inferior).

Conclusions and Relevance   With limited evidence of cost-effectiveness, biologic omalizumab, 300 mg, for H 1 antihistamine–refractory CSU was found to be cost-effective in US health care services at the willingness to pay threshold of $150 000 per QALY. Meanwhile, routine laboratory testing among patients with CSU without compelling indication was not cost-effective. Future studies in more diverse CU populations and resource settings are needed to fill evidence gaps.

Nochaiwong S , Chuamanochan M , Ruengorn C , Awiphan R , Bernstein JA , Thavorn K. Cost and Cost-Effectiveness of the Management Strategies of Chronic Urticaria : A Systematic Review . JAMA Dermatol. Published online August 28, 2024. doi:10.1001/jamadermatol.2024.2863

Manage citations:

© 2024

Artificial Intelligence Resource Center

Dermatology in JAMA : Read the Latest

Browse and subscribe to JAMA Network podcasts!

Others Also Liked

Select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing
• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PMC10248995

Guidance to best tools and practices for systematic reviews

Kat kolaski.

1 Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC USA

Lynne Romeiser Logan

2 Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY USA

John P. A. Ioannidis

3 Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA USA

Associated Data

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.

A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.

Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13643-023-02255-9.

Part 1. The state of evidence synthesis

Evidence syntheses are commonly regarded as the foundation of evidence-based medicine (EBM). They are widely accredited for providing reliable evidence and, as such, they have significantly influenced medical research and clinical practice. Despite their uptake throughout health care and ubiquity in contemporary medical literature, some important aspects of evidence syntheses are generally overlooked or not well recognized. Evidence syntheses are mostly retrospective exercises, they often depend on weak or irreparably flawed data, and they may use tools that have acknowledged or yet unrecognized limitations. They are complicated and time-consuming undertakings prone to bias and errors. Production of a good evidence synthesis requires careful preparation and high levels of organization in order to limit potential pitfalls [ 1 ]. Many authors do not recognize the complexity of such an endeavor and the many methodological challenges they may encounter. Failure to do so is likely to result in research and resource waste.

Given their potential impact on people’s lives, it is crucial for evidence syntheses to correctly report on the current knowledge base. In order to be perceived as trustworthy, reliable demonstration of the accuracy of evidence syntheses is equally imperative [ 2 ]. Concerns about the trustworthiness of evidence syntheses are not recent developments. From the early years when EBM first began to gain traction until recent times when thousands of systematic reviews are published monthly [ 3 ] the rigor of evidence syntheses has always varied. Many systematic reviews and meta-analyses had obvious deficiencies because original methods and processes had gaps, lacked precision, and/or were not widely known. The situation has improved with empirical research concerning which methods to use and standardization of appraisal tools. However, given the geometrical increase in the number of evidence syntheses being published, a relatively larger pool of unreliable evidence syntheses is being published today.

Publication of methodological studies that critically appraise the methods used in evidence syntheses is increasing at a fast pace. This reflects the availability of tools specifically developed for this purpose [ 4 – 6 ]. Yet many clinical specialties report that alarming numbers of evidence syntheses fail on these assessments. The syntheses identified report on a broad range of common conditions including, but not limited to, cancer, [ 7 ] chronic obstructive pulmonary disease, [ 8 ] osteoporosis, [ 9 ] stroke, [ 10 ] cerebral palsy, [ 11 ] chronic low back pain, [ 12 ] refractive error, [ 13 ] major depression, [ 14 ] pain, [ 15 ] and obesity [ 16 , 17 ]. The situation is even more concerning with regard to evidence syntheses included in clinical practice guidelines (CPGs) [ 18 – 20 ]. Astonishingly, in a sample of CPGs published in 2017–18, more than half did not apply even basic systematic methods in the evidence syntheses used to inform their recommendations [ 21 ].

These reports, while not widely acknowledged, suggest there are pervasive problems not limited to evidence syntheses that evaluate specific kinds of interventions or include primary research of a particular study design (eg, randomized versus non-randomized) [ 22 ]. Similar concerns about the reliability of evidence syntheses have been expressed by proponents of EBM in highly circulated medical journals [ 23 – 26 ]. These publications have also raised awareness about redundancy, inadequate input of statistical expertise, and deficient reporting. These issues plague primary research as well; however, there is heightened concern for the impact of these deficiencies given the critical role of evidence syntheses in policy and clinical decision-making.

Methods and guidance to produce a reliable evidence synthesis

Several international consortiums of EBM experts and national health care organizations currently provide detailed guidance (Table ​ (Table1). 1 ). They draw criteria from the reporting and methodological standards of currently recommended appraisal tools, and regularly review and update their methods to reflect new information and changing needs. In addition, they endorse the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system for rating the overall quality of a body of evidence [ 27 ]. These groups typically certify or commission systematic reviews that are published in exclusive databases (eg, Cochrane, JBI) or are used to develop government or agency sponsored guidelines or health technology assessments (eg, National Institute for Health and Care Excellence [NICE], Scottish Intercollegiate Guidelines Network [SIGN], Agency for Healthcare Research and Quality [AHRQ]). They offer developers of evidence syntheses various levels of methodological advice, technical and administrative support, and editorial assistance. Use of specific protocols and checklists are required for development teams within these groups, but their online methodological resources are accessible to any potential author.

Guidance for development of evidence syntheses

Notably, Cochrane is the largest single producer of evidence syntheses in biomedical research; however, these only account for 15% of the total [ 28 ]. The World Health Organization requires Cochrane standards be used to develop evidence syntheses that inform their CPGs [ 29 ]. Authors investigating questions of intervention effectiveness in syntheses developed for Cochrane follow the Methodological Expectations of Cochrane Intervention Reviews [ 30 ] and undergo multi-tiered peer review [ 31 , 32 ]. Several empirical evaluations have shown that Cochrane systematic reviews are of higher methodological quality compared with non-Cochrane reviews [ 4 , 7 , 9 , 11 , 14 , 32 – 35 ]. However, some of these assessments have biases: they may be conducted by Cochrane-affiliated authors, and they sometimes use scales and tools developed and used in the Cochrane environment and by its partners. In addition, evidence syntheses published in the Cochrane database are not subject to space or word restrictions, while non-Cochrane syntheses are often limited. As a result, information that may be relevant to the critical appraisal of non-Cochrane reviews is often removed or is relegated to online-only supplements that may not be readily or fully accessible [ 28 ].

Influences on the state of evidence synthesis

Many authors are familiar with the evidence syntheses produced by the leading EBM organizations but can be intimidated by the time and effort necessary to apply their standards. Instead of following their guidance, authors may employ methods that are discouraged or outdated 28]. Suboptimal methods described in in the literature may then be taken up by others. For example, the Newcastle–Ottawa Scale (NOS) is a commonly used tool for appraising non-randomized studies [ 36 ]. Many authors justify their selection of this tool with reference to a publication that describes the unreliability of the NOS and recommends against its use [ 37 ]. Obviously, the authors who cite this report for that purpose have not read it. Authors and peer reviewers have a responsibility to use reliable and accurate methods and not copycat previous citations or substandard work [ 38 , 39 ]. Similar cautions may potentially extend to automation tools. These have concentrated on evidence searching [ 40 ] and selection given how demanding it is for humans to maintain truly up-to-date evidence [ 2 , 41 ]. Cochrane has deployed machine learning to identify randomized controlled trials (RCTs) and studies related to COVID-19, [ 2 , 42 ] but such tools are not yet commonly used [ 43 ]. The routine integration of automation tools in the development of future evidence syntheses should not displace the interpretive part of the process.

Editorials about unreliable or misleading systematic reviews highlight several of the intertwining factors that may contribute to continued publication of unreliable evidence syntheses: shortcomings and inconsistencies of the peer review process, lack of endorsement of current standards on the part of journal editors, the incentive structure of academia, industry influences, publication bias, and the lure of “predatory” journals [ 44 – 48 ]. At this juncture, clarification of the extent to which each of these factors contribute remains speculative, but their impact is likely to be synergistic.

Over time, the generalized acceptance of the conclusions of systematic reviews as incontrovertible has affected trends in the dissemination and uptake of evidence. Reporting of the results of evidence syntheses and recommendations of CPGs has shifted beyond medical journals to press releases and news headlines and, more recently, to the realm of social media and influencers. The lay public and policy makers may depend on these outlets for interpreting evidence syntheses and CPGs. Unfortunately, communication to the general public often reflects intentional or non-intentional misrepresentation or “spin” of the research findings [ 49 – 52 ] News and social media outlets also tend to reduce conclusions on a body of evidence and recommendations for treatment to binary choices (eg, “do it” versus “don’t do it”) that may be assigned an actionable symbol (eg, red/green traffic lights, smiley/frowning face emoji).

Strategies for improvement

Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis [ 46 , 53 ]. Informing them about research methodology could increase the likelihood they will apply rigorous methods [ 25 , 33 , 45 ]. We tackle this challenge, from both a theoretical and a practical perspective, by offering guidance applicable to any specialty. It is based on recent methodological research that is extensively referenced to promote self-study. However, the information presented is not intended to be substitute for committed training in evidence synthesis methodology; instead, we hope to inspire our target audience to seek such training. We also hope to inform a broader audience of clinicians and guideline developers influenced by evidence syntheses. Notably, these communities often include the same members who serve in different capacities.

In the following sections, we highlight methodological concepts and practices that may be unfamiliar, problematic, confusing, or controversial. In Part 2, we consider various types of evidence syntheses and the types of research evidence summarized by them. In Part 3, we examine some widely used (and misused) tools for the critical appraisal of systematic reviews and reporting guidelines for evidence syntheses. In Part 4, we discuss how to meet methodological conduct standards applicable to key components of systematic reviews. In Part 5, we describe the merits and caveats of rating the overall certainty of a body of evidence. Finally, in Part 6, we summarize suggested terminology, methods, and tools for development and evaluation of evidence syntheses that reflect current best practices.

Part 2. Types of syntheses and research evidence

A good foundation for the development of evidence syntheses requires an appreciation of their various methodologies and the ability to correctly identify the types of research potentially available for inclusion in the synthesis.

Types of evidence syntheses

Systematic reviews have historically focused on the benefits and harms of interventions; over time, various types of systematic reviews have emerged to address the diverse information needs of clinicians, patients, and policy makers [ 54 ] Systematic reviews with traditional components have become defined by the different topics they assess (Table 2.1 ). In addition, other distinctive types of evidence syntheses have evolved, including overviews or umbrella reviews, scoping reviews, rapid reviews, and living reviews. The popularity of these has been increasing in recent years [ 55 – 58 ]. A summary of the development, methods, available guidance, and indications for these unique types of evidence syntheses is available in Additional File 2 A.

Types of traditional systematic reviews

Both Cochrane [ 30 , 59 ] and JBI [ 60 ] provide methodologies for many types of evidence syntheses; they describe these with different terminology, but there is obvious overlap (Table 2.2 ). The majority of evidence syntheses published by Cochrane (96%) and JBI (62%) are categorized as intervention reviews. This reflects the earlier development and dissemination of their intervention review methodologies; these remain well-established [ 30 , 59 , 61 ] as both organizations continue to focus on topics related to treatment efficacy and harms. In contrast, intervention reviews represent only about half of the total published in the general medical literature, and several non-intervention review types contribute to a significant proportion of the other half.

Evidence syntheses published by Cochrane and JBI

a Data from https://www.cochranelibrary.com/cdsr/reviews . Accessed 17 Sep 2022

b Data obtained via personal email communication on 18 Sep 2022 with Emilie Francis, editorial assistant, JBI Evidence Synthesis

c Includes the following categories: prevalence, scoping, mixed methods, and realist reviews

d This methodology is not supported in the current version of the JBI Manual for Evidence Synthesis

Types of research evidence

There is consensus on the importance of using multiple study designs in evidence syntheses; at the same time, there is a lack of agreement on methods to identify included study designs. Authors of evidence syntheses may use various taxonomies and associated algorithms to guide selection and/or classification of study designs. These tools differentiate categories of research and apply labels to individual study designs (eg, RCT, cross-sectional). A familiar example is the Design Tree endorsed by the Centre for Evidence-Based Medicine [ 70 ]. Such tools may not be helpful to authors of evidence syntheses for multiple reasons.

Suboptimal levels of agreement and accuracy even among trained methodologists reflect challenges with the application of such tools [ 71 , 72 ]. Problematic distinctions or decision points (eg, experimental or observational, controlled or uncontrolled, prospective or retrospective) and design labels (eg, cohort, case control, uncontrolled trial) have been reported [ 71 ]. The variable application of ambiguous study design labels to non-randomized studies is common, making them especially prone to misclassification [ 73 ]. In addition, study labels do not denote the unique design features that make different types of non-randomized studies susceptible to different biases, including those related to how the data are obtained (eg, clinical trials, disease registries, wearable devices). Given this limitation, it is important to be aware that design labels preclude the accurate assignment of non-randomized studies to a “level of evidence” in traditional hierarchies [ 74 ].

These concerns suggest that available tools and nomenclature used to distinguish types of research evidence may not uniformly apply to biomedical research and non-health fields that utilize evidence syntheses (eg, education, economics) [ 75 , 76 ]. Moreover, primary research reports often do not describe study design or do so incompletely or inaccurately; thus, indexing in PubMed and other databases does not address the potential for misclassification [ 77 ]. Yet proper identification of research evidence has implications for several key components of evidence syntheses. For example, search strategies limited by index terms using design labels or study selection based on labels applied by the authors of primary studies may cause inconsistent or unjustified study inclusions and/or exclusions [ 77 ]. In addition, because risk of bias (RoB) tools consider attributes specific to certain types of studies and study design features, results of these assessments may be invalidated if an inappropriate tool is used. Appropriate classification of studies is also relevant for the selection of a suitable method of synthesis and interpretation of those results.

An alternative to these tools and nomenclature involves application of a few fundamental distinctions that encompass a wide range of research designs and contexts. While these distinctions are not novel, we integrate them into a practical scheme (see Fig. ​ Fig.1) 1 ) designed to guide authors of evidence syntheses in the basic identification of research evidence. The initial distinction is between primary and secondary studies. Primary studies are then further distinguished by: 1) the type of data reported (qualitative or quantitative); and 2) two defining design features (group or single-case and randomized or non-randomized). The different types of studies and study designs represented in the scheme are described in detail in Additional File 2 B. It is important to conceptualize their methods as complementary as opposed to contrasting or hierarchical [ 78 ]; each offers advantages and disadvantages that determine their appropriateness for answering different kinds of research questions in an evidence synthesis.

Distinguishing types of research evidence

Application of these basic distinctions may avoid some of the potential difficulties associated with study design labels and taxonomies. Nevertheless, debatable methodological issues are raised when certain types of research identified in this scheme are included in an evidence synthesis. We briefly highlight those associated with inclusion of non-randomized studies, case reports and series, and a combination of primary and secondary studies.

Non-randomized studies

When investigating an intervention’s effectiveness, it is important for authors to recognize the uncertainty of observed effects reported by studies with high RoB. Results of statistical analyses that include such studies need to be interpreted with caution in order to avoid misleading conclusions [ 74 ]. Review authors may consider excluding randomized studies with high RoB from meta-analyses. Non-randomized studies of intervention (NRSI) are affected by a greater potential range of biases and thus vary more than RCTs in their ability to estimate a causal effect [ 79 ]. If data from NRSI are synthesized in meta-analyses, it is helpful to separately report their summary estimates [ 6 , 74 ].

Nonetheless, certain design features of NRSI (eg, which parts of the study were prospectively designed) may help to distinguish stronger from weaker ones. Cochrane recommends that authors of a review including NRSI focus on relevant study design features when determining eligibility criteria instead of relying on non-informative study design labels [ 79 , 80 ] This process is facilitated by a study design feature checklist; guidance on using the checklist is included with developers’ description of the tool [ 73 , 74 ]. Authors collect information about these design features during data extraction and then consider it when making final study selection decisions and when performing RoB assessments of the included NRSI.

Case reports and case series

Correctly identified case reports and case series can contribute evidence not well captured by other designs [ 81 ]; in addition, some topics may be limited to a body of evidence that consists primarily of uncontrolled clinical observations. Murad and colleagues offer a framework for how to include case reports and series in an evidence synthesis [ 82 ]. Distinguishing between cohort studies and case series in these syntheses is important, especially for those that rely on evidence from NRSI. Additional data obtained from studies misclassified as case series can potentially increase the confidence in effect estimates. Mathes and Pieper provide authors of evidence syntheses with specific guidance on distinguishing between cohort studies and case series, but emphasize the increased workload involved [ 77 ].

Primary and secondary studies

Synthesis of combined evidence from primary and secondary studies may provide a broad perspective on the entirety of available literature on a topic. This is, in fact, the recommended strategy for scoping reviews that may include a variety of sources of evidence (eg, CPGs, popular media). However, except for scoping reviews, the synthesis of data from primary and secondary studies is discouraged unless there are strong reasons to justify doing so.

Combining primary and secondary sources of evidence is challenging for authors of other types of evidence syntheses for several reasons [ 83 ]. Assessments of RoB for primary and secondary studies are derived from conceptually different tools, thus obfuscating the ability to make an overall RoB assessment of a combination of these study types. In addition, authors who include primary and secondary studies must devise non-standardized methods for synthesis. Note this contrasts with well-established methods available for updating existing evidence syntheses with additional data from new primary studies [ 84 – 86 ]. However, a new review that synthesizes data from primary and secondary studies raises questions of validity and may unintentionally support a biased conclusion because no existing methodological guidance is currently available [ 87 ].

Recommendations

We suggest that journal editors require authors to identify which type of evidence synthesis they are submitting and reference the specific methodology used for its development. This will clarify the research question and methods for peer reviewers and potentially simplify the editorial process. Editors should announce this practice and include it in the instructions to authors. To decrease bias and apply correct methods, authors must also accurately identify the types of research evidence included in their syntheses.

Part 3. Conduct and reporting

The need to develop criteria to assess the rigor of systematic reviews was recognized soon after the EBM movement began to gain international traction [ 88 , 89 ]. Systematic reviews rapidly became popular, but many were very poorly conceived, conducted, and reported. These problems remain highly prevalent [ 23 ] despite development of guidelines and tools to standardize and improve the performance and reporting of evidence syntheses [ 22 , 28 ]. Table 3.1  provides some historical perspective on the evolution of tools developed specifically for the evaluation of systematic reviews, with or without meta-analysis.

Tools specifying standards for systematic reviews with and without meta-analysis

a Currently recommended

b Validated tool for systematic reviews of interventions developed for use by authors of overviews or umbrella reviews

These tools are often interchangeably invoked when referring to the “quality” of an evidence synthesis. However, quality is a vague term that is frequently misused and misunderstood; more precisely, these tools specify different standards for evidence syntheses. Methodological standards address how well a systematic review was designed and performed [ 5 ]. RoB assessments refer to systematic flaws or limitations in the design, conduct, or analysis of research that distort the findings of the review [ 4 ]. Reporting standards help systematic review authors describe the methodology they used and the results of their synthesis in sufficient detail [ 92 ]. It is essential to distinguish between these evaluations: a systematic review may be biased, it may fail to report sufficient information on essential features, or it may exhibit both problems; a thoroughly reported systematic evidence synthesis review may still be biased and flawed while an otherwise unbiased one may suffer from deficient documentation.

We direct attention to the currently recommended tools listed in Table 3.1  but concentrate on AMSTAR-2 (update of AMSTAR [A Measurement Tool to Assess Systematic Reviews]) and ROBIS (Risk of Bias in Systematic Reviews), which evaluate methodological quality and RoB, respectively. For comparison and completeness, we include PRISMA 2020 (update of the 2009 Preferred Reporting Items for Systematic Reviews of Meta-Analyses statement), which offers guidance on reporting standards. The exclusive focus on these three tools is by design; it addresses concerns related to the considerable variability in tools used for the evaluation of systematic reviews [ 28 , 88 , 96 , 97 ]. We highlight the underlying constructs these tools were designed to assess, then describe their components and applications. Their known (or potential) uptake and impact and limitations are also discussed.

Evaluation of conduct

Development.

AMSTAR [ 5 ] was in use for a decade prior to the 2017 publication of AMSTAR-2; both provide a broad evaluation of methodological quality of intervention systematic reviews, including flaws arising through poor conduct of the review [ 6 ]. ROBIS, published in 2016, was developed to specifically assess RoB introduced by the conduct of the review; it is applicable to systematic reviews of interventions and several other types of reviews [ 4 ]. Both tools reflect a shift to a domain-based approach as opposed to generic quality checklists. There are a few items unique to each tool; however, similarities between items have been demonstrated [ 98 , 99 ]. AMSTAR-2 and ROBIS are recommended for use by: 1) authors of overviews or umbrella reviews and CPGs to evaluate systematic reviews considered as evidence; 2) authors of methodological research studies to appraise included systematic reviews; and 3) peer reviewers for appraisal of submitted systematic review manuscripts. For authors, these tools may function as teaching aids and inform conduct of their review during its development.

Description

Systematic reviews that include randomized and/or non-randomized studies as evidence can be appraised with AMSTAR-2 and ROBIS. Other characteristics of AMSTAR-2 and ROBIS are summarized in Table 3.2 . Both tools define categories for an overall rating; however, neither tool is intended to generate a total score by simply calculating the number of responses satisfying criteria for individual items [ 4 , 6 ]. AMSTAR-2 focuses on the rigor of a review’s methods irrespective of the specific subject matter. ROBIS places emphasis on a review’s results section— this suggests it may be optimally applied by appraisers with some knowledge of the review’s topic as they may be better equipped to determine if certain procedures (or lack thereof) would impact the validity of a review’s findings [ 98 , 100 ]. Reliability studies show AMSTAR-2 overall confidence ratings strongly correlate with the overall RoB ratings in ROBIS [ 100 , 101 ].

Comparison of AMSTAR-2 and ROBIS

a ROBIS includes an optional first phase to assess the applicability of the review to the research question of interest. The tool may be applicable to other review types in addition to the four specified, although modification of this initial phase will be needed (Personal Communication via email, Penny Whiting, 28 Jan 2022)

b AMSTAR-2 item #9 and #11 require separate responses for RCTs and NRSI

Interrater reliability has been shown to be acceptable for AMSTAR-2 [ 6 , 11 , 102 ] and ROBIS [ 4 , 98 , 103 ] but neither tool has been shown to be superior in this regard [ 100 , 101 , 104 , 105 ]. Overall, variability in reliability for both tools has been reported across items, between pairs of raters, and between centers [ 6 , 100 , 101 , 104 ]. The effects of appraiser experience on the results of AMSTAR-2 and ROBIS require further evaluation [ 101 , 105 ]. Updates to both tools should address items shown to be prone to individual appraisers’ subjective biases and opinions [ 11 , 100 ]; this may involve modifications of the current domains and signaling questions as well as incorporation of methods to make an appraiser’s judgments more explicit. Future revisions of these tools may also consider the addition of standards for aspects of systematic review development currently lacking (eg, rating overall certainty of evidence, [ 99 ] methods for synthesis without meta-analysis [ 105 ]) and removal of items that assess aspects of reporting that are thoroughly evaluated by PRISMA 2020.

Application

A good understanding of what is required to satisfy the standards of AMSTAR-2 and ROBIS involves study of the accompanying guidance documents written by the tools’ developers; these contain detailed descriptions of each item’s standards. In addition, accurate appraisal of a systematic review with either tool requires training. Most experts recommend independent assessment by at least two appraisers with a process for resolving discrepancies as well as procedures to establish interrater reliability, such as pilot testing, a calibration phase or exercise, and development of predefined decision rules [ 35 , 99 – 101 , 103 , 104 , 106 ]. These methods may, to some extent, address the challenges associated with the diversity in methodological training, subject matter expertise, and experience using the tools that are likely to exist among appraisers.

The standards of AMSTAR, AMSTAR-2, and ROBIS have been used in many methodological studies and epidemiological investigations. However, the increased publication of overviews or umbrella reviews and CPGs has likely been a greater influence on the widening acceptance of these tools. Critical appraisal of the secondary studies considered evidence is essential to the trustworthiness of both the recommendations of CPGs and the conclusions of overviews. Currently both Cochrane [ 55 ] and JBI [ 107 ] recommend AMSTAR-2 and ROBIS in their guidance for authors of overviews or umbrella reviews. However, ROBIS and AMSTAR-2 were released in 2016 and 2017, respectively; thus, to date, limited data have been reported about the uptake of these tools or which of the two may be preferred [ 21 , 106 ]. Currently, in relation to CPGs, AMSTAR-2 appears to be overwhelmingly popular compared to ROBIS. A Google Scholar search of this topic (search terms “AMSTAR 2 AND clinical practice guidelines,” “ROBIS AND clinical practice guidelines” 13 May 2022) found 12,700 hits for AMSTAR-2 and 1,280 for ROBIS. The apparent greater appeal of AMSTAR-2 may relate to its longer track record given the original version of the tool was in use for 10 years prior to its update in 2017.

Barriers to the uptake of AMSTAR-2 and ROBIS include the real or perceived time and resources necessary to complete the items they include and appraisers’ confidence in their own ratings [ 104 ]. Reports from comparative studies available to date indicate that appraisers find AMSTAR-2 questions, responses, and guidance to be clearer and simpler compared with ROBIS [ 11 , 101 , 104 , 105 ]. This suggests that for appraisal of intervention systematic reviews, AMSTAR-2 may be a more practical tool than ROBIS, especially for novice appraisers [ 101 , 103 – 105 ]. The unique characteristics of each tool, as well as their potential advantages and disadvantages, should be taken into consideration when deciding which tool should be used for an appraisal of a systematic review. In addition, the choice of one or the other may depend on how the results of an appraisal will be used; for example, a peer reviewer’s appraisal of a single manuscript versus an appraisal of multiple systematic reviews in an overview or umbrella review, CPG, or systematic methodological study.

Authors of overviews and CPGs report results of AMSTAR-2 and ROBIS appraisals for each of the systematic reviews they include as evidence. Ideally, an independent judgment of their appraisals can be made by the end users of overviews and CPGs; however, most stakeholders, including clinicians, are unlikely to have a sophisticated understanding of these tools. Nevertheless, they should at least be aware that AMSTAR-2 and ROBIS ratings reported in overviews and CPGs may be inaccurate because the tools are not applied as intended by their developers. This can result from inadequate training of the overview or CPG authors who perform the appraisals, or to modifications of the appraisal tools imposed by them. The potential variability in overall confidence and RoB ratings highlights why appraisers applying these tools need to support their judgments with explicit documentation; this allows readers to judge for themselves whether they agree with the criteria used by appraisers [ 4 , 108 ]. When these judgments are explicit, the underlying rationale used when applying these tools can be assessed [ 109 ].

Theoretically, we would expect an association of AMSTAR-2 with improved methodological rigor and an association of ROBIS with lower RoB in recent systematic reviews compared to those published before 2017. To our knowledge, this has not yet been demonstrated; however, like reports about the actual uptake of these tools, time will tell. Additional data on user experience is also needed to further elucidate the practical challenges and methodological nuances encountered with the application of these tools. This information could potentially inform the creation of unifying criteria to guide and standardize the appraisal of evidence syntheses [ 109 ].

Evaluation of reporting

Complete reporting is essential for users to establish the trustworthiness and applicability of a systematic review’s findings. Efforts to standardize and improve the reporting of systematic reviews resulted in the 2009 publication of the PRISMA statement [ 92 ] with its accompanying explanation and elaboration document [ 110 ]. This guideline was designed to help authors prepare a complete and transparent report of their systematic review. In addition, adherence to PRISMA is often used to evaluate the thoroughness of reporting of published systematic reviews [ 111 ]. The updated version, PRISMA 2020 [ 93 ], and its guidance document [ 112 ] were published in 2021. Items on the original and updated versions of PRISMA are organized by the six basic review components they address (title, abstract, introduction, methods, results, discussion). The PRISMA 2020 update is a considerably expanded version of the original; it includes standards and examples for the 27 original and 13 additional reporting items that capture methodological advances and may enhance the replicability of reviews [ 113 ].

The original PRISMA statement fostered the development of various PRISMA extensions (Table 3.3 ). These include reporting guidance for scoping reviews and reviews of diagnostic test accuracy and for intervention reviews that report on the following: harms outcomes, equity issues, the effects of acupuncture, the results of network meta-analyses and analyses of individual participant data. Detailed reporting guidance for specific systematic review components (abstracts, protocols, literature searches) is also available.

PRISMA extensions

PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses

a Note the abstract reporting checklist is now incorporated into PRISMA 2020 [ 93 ]

Uptake and impact

The 2009 PRISMA standards [ 92 ] for reporting have been widely endorsed by authors, journals, and EBM-related organizations. We anticipate the same for PRISMA 2020 [ 93 ] given its co-publication in multiple high-impact journals. However, to date, there is a lack of strong evidence for an association between improved systematic review reporting and endorsement of PRISMA 2009 standards [ 43 , 111 ]. Most journals require a PRISMA checklist accompany submissions of systematic review manuscripts. However, the accuracy of information presented on these self-reported checklists is not necessarily verified. It remains unclear which strategies (eg, authors’ self-report of checklists, peer reviewer checks) might improve adherence to the PRISMA reporting standards; in addition, the feasibility of any potentially effective strategies must be taken into consideration given the structure and limitations of current research and publication practices [ 124 ].

Pitfalls and limitations of PRISMA, AMSTAR-2, and ROBIS

Misunderstanding of the roles of these tools and their misapplication may be widespread problems. PRISMA 2020 is a reporting guideline that is most beneficial if consulted when developing a review as opposed to merely completing a checklist when submitting to a journal; at that point, the review is finished, with good or bad methodological choices. However, PRISMA checklists evaluate how completely an element of review conduct was reported, but do not evaluate the caliber of conduct or performance of a review. Thus, review authors and readers should not think that a rigorous systematic review can be produced by simply following the PRISMA 2020 guidelines. Similarly, it is important to recognize that AMSTAR-2 and ROBIS are tools to evaluate the conduct of a review but do not substitute for conceptual methodological guidance. In addition, they are not intended to be simple checklists. In fact, they have the potential for misuse or abuse if applied as such; for example, by calculating a total score to make a judgment about a review’s overall confidence or RoB. Proper selection of a response for the individual items on AMSTAR-2 and ROBIS requires training or at least reference to their accompanying guidance documents.

Not surprisingly, it has been shown that compliance with the PRISMA checklist is not necessarily associated with satisfying the standards of ROBIS [ 125 ]. AMSTAR-2 and ROBIS were not available when PRISMA 2009 was developed; however, they were considered in the development of PRISMA 2020 [ 113 ]. Therefore, future studies may show a positive relationship between fulfillment of PRISMA 2020 standards for reporting and meeting the standards of tools evaluating methodological quality and RoB.

Choice of an appropriate tool for the evaluation of a systematic review first involves identification of the underlying construct to be assessed. For systematic reviews of interventions, recommended tools include AMSTAR-2 and ROBIS for appraisal of conduct and PRISMA 2020 for completeness of reporting. All three tools were developed rigorously and provide easily accessible and detailed user guidance, which is necessary for their proper application and interpretation. When considering a manuscript for publication, training in these tools can sensitize peer reviewers and editors to major issues that may affect the review’s trustworthiness and completeness of reporting. Judgment of the overall certainty of a body of evidence and formulation of recommendations rely, in part, on AMSTAR-2 or ROBIS appraisals of systematic reviews. Therefore, training on the application of these tools is essential for authors of overviews and developers of CPGs. Peer reviewers and editors considering an overview or CPG for publication must hold their authors to a high standard of transparency regarding both the conduct and reporting of these appraisals.

Part 4. Meeting conduct standards

Many authors, peer reviewers, and editors erroneously equate fulfillment of the items on the PRISMA checklist with superior methodological rigor. For direction on methodology, we refer them to available resources that provide comprehensive conceptual guidance [ 59 , 60 ] as well as primers with basic step-by-step instructions [ 1 , 126 , 127 ]. This section is intended to complement study of such resources by facilitating use of AMSTAR-2 and ROBIS, tools specifically developed to evaluate methodological rigor of systematic reviews. These tools are widely accepted by methodologists; however, in the general medical literature, they are not uniformly selected for the critical appraisal of systematic reviews [ 88 , 96 ].

To enable their uptake, Table 4.1  links review components to the corresponding appraisal tool items. Expectations of AMSTAR-2 and ROBIS are concisely stated, and reasoning provided.

Systematic review components linked to appraisal with AMSTAR-2 and ROBIS a

CoI conflict of interest, MA meta-analysis, NA not addressed, PICO participant, intervention, comparison, outcome, PRISMA-P Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols, RoB risk of bias

a Components shown in bold are chosen for elaboration in Part 4 for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors; and/or 2) the component is evaluated by standards of an AMSTAR-2 “critical” domain

b Critical domains of AMSTAR-2 are indicated by *

Issues involved in meeting the standards for seven review components (identified in bold in Table 4.1 ) are addressed in detail. These were chosen for elaboration for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors based on consistent reports of their frequent AMSTAR-2 or ROBIS deficiencies [ 9 , 11 , 15 , 88 , 128 , 129 ]; and/or 2) the review component is judged by standards of an AMSTAR-2 “critical” domain. These have the greatest implications for how a systematic review will be appraised: if standards for any one of these critical domains are not met, the review is rated as having “critically low confidence.”

Research question

Specific and unambiguous research questions may have more value for reviews that deal with hypothesis testing. Mnemonics for the various elements of research questions are suggested by JBI and Cochrane (Table 2.1 ). These prompt authors to consider the specialized methods involved for developing different types of systematic reviews; however, while inclusion of the suggested elements makes a review compliant with a particular review’s methods, it does not necessarily make a research question appropriate. Table 4.2  lists acronyms that may aid in developing the research question. They include overlapping concepts of importance in this time of proliferating reviews of uncertain value [ 130 ]. If these issues are not prospectively contemplated, systematic review authors may establish an overly broad scope, or develop runaway scope allowing them to stray from predefined choices relating to key comparisons and outcomes.

Research question development

a Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Hulley SB, Cummings SR, Browner WS, editors. Designing clinical research: an epidemiological approach; 4th edn. Lippincott Williams & Wilkins; 2007. p. 14–22

b Doran, GT. There’s a S.M.A.R.T. way to write management’s goals and objectives. Manage Rev. 1981;70:35-6.

c Johnson BT, Hennessy EA. Systematic reviews and meta-analyses in the health sciences: best practice methods for research syntheses. Soc Sci Med. 2019;233:237–51

Once a research question is established, searching on registry sites and databases for existing systematic reviews addressing the same or a similar topic is necessary in order to avoid contributing to research waste [ 131 ]. Repeating an existing systematic review must be justified, for example, if previous reviews are out of date or methodologically flawed. A full discussion on replication of intervention systematic reviews, including a consensus checklist, can be found in the work of Tugwell and colleagues [ 84 ].

Protocol development is considered a core component of systematic reviews [ 125 , 126 , 132 ]. Review protocols may allow researchers to plan and anticipate potential issues, assess validity of methods, prevent arbitrary decision-making, and minimize bias that can be introduced by the conduct of the review. Registration of a protocol that allows public access promotes transparency of the systematic review’s methods and processes and reduces the potential for duplication [ 132 ]. Thinking early and carefully about all the steps of a systematic review is pragmatic and logical and may mitigate the influence of the authors’ prior knowledge of the evidence [ 133 ]. In addition, the protocol stage is when the scope of the review can be carefully considered by authors, reviewers, and editors; this may help to avoid production of overly ambitious reviews that include excessive numbers of comparisons and outcomes or are undisciplined in their study selection.

An association with attainment of AMSTAR standards in systematic reviews with published prospective protocols has been reported [ 134 ]. However, completeness of reporting does not seem to be different in reviews with a protocol compared to those without one [ 135 ]. PRISMA-P [ 116 ] and its accompanying elaboration and explanation document [ 136 ] can be used to guide and assess the reporting of protocols. A final version of the review should fully describe any protocol deviations. Peer reviewers may compare the submitted manuscript with any available pre-registered protocol; this is required if AMSTAR-2 or ROBIS are used for critical appraisal.

There are multiple options for the recording of protocols (Table 4.3 ). Some journals will peer review and publish protocols. In addition, many online sites offer date-stamped and publicly accessible protocol registration. Some of these are exclusively for protocols of evidence syntheses; others are less restrictive and offer researchers the capacity for data storage, sharing, and other workflow features. These sites document protocol details to varying extents and have different requirements [ 137 ]. The most popular site for systematic reviews, the International Prospective Register of Systematic Reviews (PROSPERO), for example, only registers reviews that report on an outcome with direct relevance to human health. The PROSPERO record documents protocols for all types of reviews except literature and scoping reviews. Of note, PROSPERO requires authors register their review protocols prior to any data extraction [ 133 , 138 ]. The electronic records of most of these registry sites allow authors to update their protocols and facilitate transparent tracking of protocol changes, which are not unexpected during the progress of the review [ 139 ].

Options for protocol registration of evidence syntheses

a Authors are advised to contact their target journal regarding submission of systematic review protocols

b Registration is restricted to approved review projects

c The JBI registry lists review projects currently underway by JBI-affiliated entities. These records include a review’s title, primary author, research question, and PICO elements. JBI recommends that authors register eligible protocols with PROSPERO

d See Pieper and Rombey [ 137 ] for detailed characteristics of these five registries

e See Pieper and Rombey [ 137 ] for other systematic review data repository options

Study design inclusion

For most systematic reviews, broad inclusion of study designs is recommended [ 126 ]. This may allow comparison of results between contrasting study design types [ 126 ]. Certain study designs may be considered preferable depending on the type of review and nature of the research question. However, prevailing stereotypes about what each study design does best may not be accurate. For example, in systematic reviews of interventions, randomized designs are typically thought to answer highly specific questions while non-randomized designs often are expected to reveal greater information about harms or real-word evidence [ 126 , 140 , 141 ]. This may be a false distinction; randomized trials may be pragmatic [ 142 ], they may offer important (and more unbiased) information on harms [ 143 ], and data from non-randomized trials may not necessarily be more real-world-oriented [ 144 ].

Moreover, there may not be any available evidence reported by RCTs for certain research questions; in some cases, there may not be any RCTs or NRSI. When the available evidence is limited to case reports and case series, it is not possible to test hypotheses nor provide descriptive estimates or associations; however, a systematic review of these studies can still offer important insights [ 81 , 145 ]. When authors anticipate that limited evidence of any kind may be available to inform their research questions, a scoping review can be considered. Alternatively, decisions regarding inclusion of indirect as opposed to direct evidence can be addressed during protocol development [ 146 ]. Including indirect evidence at an early stage of intervention systematic review development allows authors to decide if such studies offer any additional and/or different understanding of treatment effects for their population or comparison of interest. Issues of indirectness of included studies are accounted for later in the process, during determination of the overall certainty of evidence (see Part 5 for details).

Evidence search

Both AMSTAR-2 and ROBIS require systematic and comprehensive searches for evidence. This is essential for any systematic review. Both tools discourage search restrictions based on language and publication source. Given increasing globalism in health care, the practice of including English-only literature should be avoided [ 126 ]. There are many examples in which language bias (different results in studies published in different languages) has been documented [ 147 , 148 ]. This does not mean that all literature, in all languages, is equally trustworthy [ 148 ]; however, the only way to formally probe for the potential of such biases is to consider all languages in the initial search. The gray literature and a search of trials may also reveal important details about topics that would otherwise be missed [ 149 – 151 ]. Again, inclusiveness will allow review authors to investigate whether results differ in gray literature and trials [ 41 , 151 – 153 ].

Authors should make every attempt to complete their review within one year as that is the likely viable life of a search. (1) If that is not possible, the search should be updated close to the time of completion [ 154 ]. Different research topics may warrant less of a delay, for example, in rapidly changing fields (as in the case of the COVID-19 pandemic), even one month may radically change the available evidence.

Excluded studies

AMSTAR-2 requires authors to provide references for any studies excluded at the full text phase of study selection along with reasons for exclusion; this allows readers to feel confident that all relevant literature has been considered for inclusion and that exclusions are defensible.

Risk of bias assessment of included studies

The design of the studies included in a systematic review (eg, RCT, cohort, case series) should not be equated with appraisal of its RoB. To meet AMSTAR-2 and ROBIS standards, systematic review authors must examine RoB issues specific to the design of each primary study they include as evidence. It is unlikely that a single RoB appraisal tool will be suitable for all research designs. In addition to tools for randomized and non-randomized studies, specific tools are available for evaluation of RoB in case reports and case series [ 82 ] and single-case experimental designs [ 155 , 156 ]. Note the RoB tools selected must meet the standards of the appraisal tool used to judge the conduct of the review. For example, AMSTAR-2 identifies four sources of bias specific to RCTs and NRSI that must be addressed by the RoB tool(s) chosen by the review authors. The Cochrane RoB-2 [ 157 ] tool for RCTs and ROBINS-I [ 158 ] for NRSI for RoB assessment meet the AMSTAR-2 standards. Appraisers on the review team should not modify any RoB tool without complete transparency and acknowledgment that they have invalidated the interpretation of the tool as intended by its developers [ 159 ]. Conduct of RoB assessments is not addressed AMSTAR-2; to meet ROBIS standards, two independent reviewers should complete RoB assessments of included primary studies.

Implications of the RoB assessments must be explicitly discussed and considered in the conclusions of the review. Discussion of the overall RoB of included studies may consider the weight of the studies at high RoB, the importance of the sources of bias in the studies being summarized, and if their importance differs in relationship to the outcomes reported. If a meta-analysis is performed, serious concerns for RoB of individual studies should be accounted for in these results as well. If the results of the meta-analysis for a specific outcome change when studies at high RoB are excluded, readers will have a more accurate understanding of this body of evidence. However, while investigating the potential impact of specific biases is a useful exercise, it is important to avoid over-interpretation, especially when there are sparse data.

Synthesis methods for quantitative data

Syntheses of quantitative data reported by primary studies are broadly categorized as one of two types: meta-analysis, and synthesis without meta-analysis (Table 4.4 ). Before deciding on one of these methods, authors should seek methodological advice about whether reported data can be transformed or used in other ways to provide a consistent effect measure across studies [ 160 , 161 ].

Common methods for quantitative synthesis

CI confidence interval (or credible interval, if analysis is done in Bayesian framework)

a See text for descriptions of the types of data combined in each of these approaches

b See Additional File 4  for guidance on the structure and presentation of forest plots

c General approach is similar to aggregate data meta-analysis but there are substantial differences relating to data collection and checking and analysis [ 162 ]. This approach to syntheses is applicable to intervention, diagnostic, and prognostic systematic reviews [ 163 ]

d Examples include meta-regression, hierarchical and multivariate approaches [ 164 ]

e In-depth guidance and illustrations of these methods are provided in Chapter 12 of the Cochrane Handbook [ 160 ]

Meta-analysis

Systematic reviews that employ meta-analysis should not be referred to simply as “meta-analyses.” The term meta-analysis strictly refers to a specific statistical technique used when study effect estimates and their variances are available, yielding a quantitative summary of results. In general, methods for meta-analysis involve use of a weighted average of effect estimates from two or more studies. If considered carefully, meta-analysis increases the precision of the estimated magnitude of effect and can offer useful insights about heterogeneity and estimates of effects. We refer to standard references for a thorough introduction and formal training [ 165 – 167 ].

There are three common approaches to meta-analysis in current health care–related systematic reviews (Table 4.4 ). Aggregate meta-analyses is the most familiar to authors of evidence syntheses and their end users. This standard meta-analysis combines data on effect estimates reported by studies that investigate similar research questions involving direct comparisons of an intervention and comparator. Results of these analyses provide a single summary intervention effect estimate. If the included studies in a systematic review measure an outcome differently, their reported results may be transformed to make them comparable [ 161 ]. Forest plots visually present essential information about the individual studies and the overall pooled analysis (see Additional File 4  for details).

Less familiar and more challenging meta-analytical approaches used in secondary research include individual participant data (IPD) and network meta-analyses (NMA); PRISMA extensions provide reporting guidelines for both [ 117 , 118 ]. In IPD, the raw data on each participant from each eligible study are re-analyzed as opposed to the study-level data analyzed in aggregate data meta-analyses [ 168 ]. This may offer advantages, including the potential for limiting concerns about bias and allowing more robust analyses [ 163 ]. As suggested by the description in Table 4.4 , NMA is a complex statistical approach. It combines aggregate data [ 169 ] or IPD [ 170 ] for effect estimates from direct and indirect comparisons reported in two or more studies of three or more interventions. This makes it a potentially powerful statistical tool; while multiple interventions are typically available to treat a condition, few have been evaluated in head-to-head trials [ 171 ]. Both IPD and NMA facilitate a broader scope, and potentially provide more reliable and/or detailed results; however, compared with standard aggregate data meta-analyses, their methods are more complicated, time-consuming, and resource-intensive, and they have their own biases, so one needs sufficient funding, technical expertise, and preparation to employ them successfully [ 41 , 172 , 173 ].

Several items in AMSTAR-2 and ROBIS address meta-analysis; thus, understanding the strengths, weaknesses, assumptions, and limitations of methods for meta-analyses is important. According to the standards of both tools, plans for a meta-analysis must be addressed in the review protocol, including reasoning, description of the type of quantitative data to be synthesized, and the methods planned for combining the data. This should not consist of stock statements describing conventional meta-analysis techniques; rather, authors are expected to anticipate issues specific to their research questions. Concern for the lack of training in meta-analysis methods among systematic review authors cannot be overstated. For those with training, the use of popular software (eg, RevMan [ 174 ], MetaXL [ 175 ], JBI SUMARI [ 176 ]) may facilitate exploration of these methods; however, such programs cannot substitute for the accurate interpretation of the results of meta-analyses, especially for more complex meta-analytical approaches.

Synthesis without meta-analysis

There are varied reasons a meta-analysis may not be appropriate or desirable [ 160 , 161 ]. Syntheses that informally use statistical methods other than meta-analysis are variably referred to as descriptive, narrative, or qualitative syntheses or summaries; these terms are also applied to syntheses that make no attempt to statistically combine data from individual studies. However, use of such imprecise terminology is discouraged; in order to fully explore the results of any type of synthesis, some narration or description is needed to supplement the data visually presented in tabular or graphic forms [ 63 , 177 ]. In addition, the term “qualitative synthesis” is easily confused with a synthesis of qualitative data in a qualitative or mixed methods review. “Synthesis without meta-analysis” is currently the preferred description of other ways to combine quantitative data from two or more studies. Use of this specific terminology when referring to these types of syntheses also implies the application of formal methods (Table 4.4 ).

Methods for syntheses without meta-analysis involve structured presentations of the data in any tables and plots. In comparison to narrative descriptions of each study, these are designed to more effectively and transparently show patterns and convey detailed information about the data; they also allow informal exploration of heterogeneity [ 178 ]. In addition, acceptable quantitative statistical methods (Table 4.4 ) are formally applied; however, it is important to recognize these methods have significant limitations for the interpretation of the effectiveness of an intervention [ 160 ]. Nevertheless, when meta-analysis is not possible, the application of these methods is less prone to bias compared with an unstructured narrative description of included studies [ 178 , 179 ].

Vote counting is commonly used in systematic reviews and involves a tally of studies reporting results that meet some threshold of importance applied by review authors. Until recently, it has not typically been identified as a method for synthesis without meta-analysis. Guidance on an acceptable vote counting method based on direction of effect is currently available [ 160 ] and should be used instead of narrative descriptions of such results (eg, “more than half the studies showed improvement”; “only a few studies reported adverse effects”; “7 out of 10 studies favored the intervention”). Unacceptable methods include vote counting by statistical significance or magnitude of effect or some subjective rule applied by the authors.

AMSTAR-2 and ROBIS standards do not explicitly address conduct of syntheses without meta-analysis, although AMSTAR-2 items 13 and 14 might be considered relevant. Guidance for the complete reporting of syntheses without meta-analysis for systematic reviews of interventions is available in the Synthesis without Meta-analysis (SWiM) guideline [ 180 ] and methodological guidance is available in the Cochrane Handbook [ 160 , 181 ].

Familiarity with AMSTAR-2 and ROBIS makes sense for authors of systematic reviews as these appraisal tools will be used to judge their work; however, training is necessary for authors to truly appreciate and apply methodological rigor. Moreover, judgment of the potential contribution of a systematic review to the current knowledge base goes beyond meeting the standards of AMSTAR-2 and ROBIS. These tools do not explicitly address some crucial concepts involved in the development of a systematic review; this further emphasizes the need for author training.

We recommend that systematic review authors incorporate specific practices or exercises when formulating a research question at the protocol stage, These should be designed to raise the review team’s awareness of how to prevent research and resource waste [ 84 , 130 ] and to stimulate careful contemplation of the scope of the review [ 30 ]. Authors’ training should also focus on justifiably choosing a formal method for the synthesis of quantitative and/or qualitative data from primary research; both types of data require specific expertise. For typical reviews that involve syntheses of quantitative data, statistical expertise is necessary, initially for decisions about appropriate methods, [ 160 , 161 ] and then to inform any meta-analyses [ 167 ] or other statistical methods applied [ 160 ].

Part 5. Rating overall certainty of evidence

Report of an overall certainty of evidence assessment in a systematic review is an important new reporting standard of the updated PRISMA 2020 guidelines [ 93 ]. Systematic review authors are well acquainted with assessing RoB in individual primary studies, but much less familiar with assessment of overall certainty across an entire body of evidence. Yet a reliable way to evaluate this broader concept is now recognized as a vital part of interpreting the evidence.

Historical systems for rating evidence are based on study design and usually involve hierarchical levels or classes of evidence that use numbers and/or letters to designate the level/class. These systems were endorsed by various EBM-related organizations. Professional societies and regulatory groups then widely adopted them, often with modifications for application to the available primary research base in specific clinical areas. In 2002, a report issued by the AHRQ identified 40 systems to rate quality of a body of evidence [ 182 ]. A critical appraisal of systems used by prominent health care organizations published in 2004 revealed limitations in sensibility, reproducibility, applicability to different questions, and usability to different end users [ 183 ]. Persistent use of hierarchical rating schemes to describe overall quality continues to complicate the interpretation of evidence. This is indicated by recent reports of poor interpretability of systematic review results by readers [ 184 – 186 ] and misleading interpretations of the evidence related to the “spin” systematic review authors may put on their conclusions [ 50 , 187 ].

Recognition of the shortcomings of hierarchical rating systems raised concerns that misleading clinical recommendations could result even if based on a rigorous systematic review. In addition, the number and variability of these systems were considered obstacles to quick and accurate interpretations of the evidence by clinicians, patients, and policymakers [ 183 ]. These issues contributed to the development of the GRADE approach. An international working group, that continues to actively evaluate and refine it, first introduced GRADE in 2004 [ 188 ]. Currently more than 110 organizations from 19 countries around the world have endorsed or are using GRADE [ 189 ].

GRADE approach to rating overall certainty

GRADE offers a consistent and sensible approach for two separate processes: rating the overall certainty of a body of evidence and the strength of recommendations. The former is the expected conclusion of a systematic review, while the latter is pertinent to the development of CPGs. As such, GRADE provides a mechanism to bridge the gap from evidence synthesis to application of the evidence for informed clinical decision-making [ 27 , 190 ]. We briefly examine the GRADE approach but only as it applies to rating overall certainty of evidence in systematic reviews.

In GRADE, use of “certainty” of a body of evidence is preferred over the term “quality.” [ 191 ] Certainty refers to the level of confidence systematic review authors have that, for each outcome, an effect estimate represents the true effect. The GRADE approach to rating confidence in estimates begins with identifying the study type (RCT or NRSI) and then systematically considers criteria to rate the certainty of evidence up or down (Table 5.1 ).

GRADE criteria for rating certainty of evidence

a Applies to randomized studies

b Applies to non-randomized studies

This process results in assignment of one of the four GRADE certainty ratings to each outcome; these are clearly conveyed with the use of basic interpretation symbols (Table 5.2 ) [ 192 ]. Notably, when multiple outcomes are reported in a systematic review, each outcome is assigned a unique certainty rating; thus different levels of certainty may exist in the body of evidence being examined.

GRADE certainty ratings and their interpretation symbols a

a From the GRADE Handbook [ 192 ]

GRADE’s developers acknowledge some subjectivity is involved in this process [ 193 ]. In addition, they emphasize that both the criteria for rating evidence up and down (Table 5.1 ) as well as the four overall certainty ratings (Table 5.2 ) reflect a continuum as opposed to discrete categories [ 194 ]. Consequently, deciding whether a study falls above or below the threshold for rating up or down may not be straightforward, and preliminary overall certainty ratings may be intermediate (eg, between low and moderate). Thus, the proper application of GRADE requires systematic review authors to take an overall view of the body of evidence and explicitly describe the rationale for their final ratings.

Advantages of GRADE

Outcomes important to the individuals who experience the problem of interest maintain a prominent role throughout the GRADE process [ 191 ]. These outcomes must inform the research questions (eg, PICO [population, intervention, comparator, outcome]) that are specified a priori in a systematic review protocol. Evidence for these outcomes is then investigated and each critical or important outcome is ultimately assigned a certainty of evidence as the end point of the review. Notably, limitations of the included studies have an impact at the outcome level. Ultimately, the certainty ratings for each outcome reported in a systematic review are considered by guideline panels. They use a different process to formulate recommendations that involves assessment of the evidence across outcomes [ 201 ]. It is beyond our scope to describe the GRADE process for formulating recommendations; however, it is critical to understand how these two outcome-centric concepts of certainty of evidence in the GRADE framework are related and distinguished. An in-depth illustration using examples from recently published evidence syntheses and CPGs is provided in Additional File 5 A (Table AF5A-1).

The GRADE approach is applicable irrespective of whether the certainty of the primary research evidence is high or very low; in some circumstances, indirect evidence of higher certainty may be considered if direct evidence is unavailable or of low certainty [ 27 ]. In fact, most interventions and outcomes in medicine have low or very low certainty of evidence based on GRADE and there seems to be no major improvement over time [ 202 , 203 ]. This is still a very important (even if sobering) realization for calibrating our understanding of medical evidence. A major appeal of the GRADE approach is that it offers a common framework that enables authors of evidence syntheses to make complex judgments about evidence certainty and to convey these with unambiguous terminology. This prevents some common mistakes made by review authors, including overstating results (or under-reporting harms) [ 187 ] and making recommendations for treatment. This is illustrated in Table AF5A-2 (Additional File 5 A), which compares the concluding statements made about overall certainty in a systematic review with and without application of the GRADE approach.

Theoretically, application of GRADE should improve consistency of judgments about certainty of evidence, both between authors and across systematic reviews. In one empirical evaluation conducted by the GRADE Working Group, interrater reliability of two individual raters assessing certainty of the evidence for a specific outcome increased from ~ 0.3 without using GRADE to ~ 0.7 by using GRADE [ 204 ]. However, others report variable agreement among those experienced in GRADE assessments of evidence certainty [ 190 ]. Like any other tool, GRADE requires training in order to be properly applied. The intricacies of the GRADE approach and the necessary subjectivity involved suggest that improving agreement may require strict rules for its application; alternatively, use of general guidance and consensus among review authors may result in less consistency but provide important information for the end user [ 190 ].

GRADE caveats

Simply invoking “the GRADE approach” does not automatically ensure GRADE methods were employed by authors of a systematic review (or developers of a CPG). Table 5.3 lists the criteria the GRADE working group has established for this purpose. These criteria highlight the specific terminology and methods that apply to rating the certainty of evidence for outcomes reported in a systematic review [ 191 ], which is different from rating overall certainty across outcomes considered in the formulation of recommendations [ 205 ]. Modifications of standard GRADE methods and terminology are discouraged as these may detract from GRADE’s objectives to minimize conceptual confusion and maximize clear communication [ 206 ].

Criteria for using GRADE in a systematic review a

a Adapted from the GRADE working group [ 206 ]; this list does not contain the additional criteria that apply to the development of a clinical practice guideline

Nevertheless, GRADE is prone to misapplications [ 207 , 208 ], which can distort a systematic review’s conclusions about the certainty of evidence. Systematic review authors without proper GRADE training are likely to misinterpret the terms “quality” and “grade” and to misunderstand the constructs assessed by GRADE versus other appraisal tools. For example, review authors may reference the standard GRADE certainty ratings (Table 5.2 ) to describe evidence for their outcome(s) of interest. However, these ratings are invalidated if authors omit or inadequately perform RoB evaluations of each included primary study. Such deficiencies in RoB assessments are unacceptable but not uncommon, as reported in methodological studies of systematic reviews and overviews [ 104 , 186 , 209 , 210 ]. GRADE ratings are also invalidated if review authors do not formally address and report on the other criteria (Table 5.1 ) necessary for a GRADE certainty rating.

Other caveats pertain to application of a GRADE certainty of evidence rating in various types of evidence syntheses. Current adaptations of GRADE are described in Additional File 5 B and included on Table 6.3 , which is introduced in the next section.

Concise Guide to best practices for evidence syntheses, version 1.0 a

AMSTAR A MeaSurement Tool to Assess Systematic Reviews, CASP Critical Appraisal Skills Programme, CERQual Confidence in the Evidence from Reviews of Qualitative research, ConQual Establishing Confidence in the output of Qualitative research synthesis, COSMIN COnsensus-based Standards for the selection of health Measurement Instruments, DTA diagnostic test accuracy, eMERGe meta-ethnography reporting guidance, ENTREQ enhancing transparency in reporting the synthesis of qualitative research, GRADE Grading of Recommendations Assessment, Development and Evaluation, MA meta-analysis, NRSI non-randomized studies of interventions, P protocol, PRIOR Preferred Reporting Items for Overviews of Reviews, PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PROBAST Prediction model Risk Of Bias ASsessment Tool, QUADAS quality assessment of studies of diagnostic accuracy included in systematic reviews, QUIPS Quality In Prognosis Studies, RCT randomized controlled trial, RoB risk of bias, ROBINS-I Risk Of Bias In Non-randomised Studies of Interventions, ROBIS Risk of Bias in Systematic Reviews, ScR scoping review, SWiM systematic review without meta-analysis

a Superscript numbers represent citations provided in the main reference list. Additional File 6 lists links to available online resources for the methods and tools included in the Concise Guide

b The MECIR manual [ 30 ] provides Cochrane’s specific standards for both reporting and conduct of intervention systematic reviews and protocols

c Editorial and peer reviewers can evaluate completeness of reporting in submitted manuscripts using these tools. Authors may be required to submit a self-reported checklist for the applicable tools

d The decision flowchart described by Flemming and colleagues [ 223 ] is recommended for guidance on how to choose the best approach to reporting for qualitative reviews

e SWiM was developed for intervention studies reporting quantitative data. However, if there is not a more directly relevant reporting guideline, SWiM may prompt reviewers to consider the important details to report. (Personal Communication via email, Mhairi Campbell, 14 Dec 2022)

f JBI recommends their own tools for the critical appraisal of various quantitative primary study designs included in systematic reviews of intervention effectiveness, prevalence and incidence, and etiology and risk as well as for the critical appraisal of systematic reviews included in umbrella reviews. However, except for the JBI Checklists for studies reporting prevalence data and qualitative research, the development, validity, and reliability of these tools are not well documented

g Studies that are not RCTs or NRSI require tools developed specifically to evaluate their design features. Examples include single case experimental design [ 155 , 156 ] and case reports and series [ 82 ]

h The evaluation of methodological quality of studies included in a synthesis of qualitative research is debatable [ 224 ]. Authors may select a tool appropriate for the type of qualitative synthesis methodology employed. The CASP Qualitative Checklist [ 218 ] is an example of a published, commonly used tool that focuses on assessment of the methodological strengths and limitations of qualitative studies. The JBI Critical Appraisal Checklist for Qualitative Research [ 219 ] is recommended for reviews using a meta-aggregative approach

i Consider including risk of bias assessment of included studies if this information is relevant to the research question; however, scoping reviews do not include an assessment of the overall certainty of a body of evidence

j Guidance available from the GRADE working group [ 225 , 226 ]; also recommend consultation with the Cochrane diagnostic methods group

k Guidance available from the GRADE working group [ 227 ]; also recommend consultation with Cochrane prognostic methods group

l Used for syntheses in reviews with a meta-aggregative approach [ 224 ]

m Chapter 5 in the JBI Manual offers guidance on how to adapt GRADE to prevalence and incidence reviews [ 69 ]

n Janiaud and colleagues suggest criteria for evaluating evidence certainty for meta-analyses of non-randomized studies evaluating risk factors [ 228 ]

o The COSMIN user manual provides details on how to apply GRADE in systematic reviews of measurement properties [ 229 ]

The expected culmination of a systematic review should be a rating of overall certainty of a body of evidence for each outcome reported. The GRADE approach is recommended for making these judgments for outcomes reported in systematic reviews of interventions and can be adapted for other types of reviews. This represents the initial step in the process of making recommendations based on evidence syntheses. Peer reviewers should ensure authors meet the minimal criteria for supporting the GRADE approach when reviewing any evidence synthesis that reports certainty ratings derived using GRADE. Authors and peer reviewers of evidence syntheses unfamiliar with GRADE are encouraged to seek formal training and take advantage of the resources available on the GRADE website [ 211 , 212 ].

Part 6. Concise Guide to best practices

Accumulating data in recent years suggest that many evidence syntheses (with or without meta-analysis) are not reliable. This relates in part to the fact that their authors, who are often clinicians, can be overwhelmed by the plethora of ways to evaluate evidence. They tend to resort to familiar but often inadequate, inappropriate, or obsolete methods and tools and, as a result, produce unreliable reviews. These manuscripts may not be recognized as such by peer reviewers and journal editors who may disregard current standards. When such a systematic review is published or included in a CPG, clinicians and stakeholders tend to believe that it is trustworthy. A vicious cycle in which inadequate methodology is rewarded and potentially misleading conclusions are accepted is thus supported. There is no quick or easy way to break this cycle; however, increasing awareness of best practices among all these stakeholder groups, who often have minimal (if any) training in methodology, may begin to mitigate it. This is the rationale for inclusion of Parts 2 through 5 in this guidance document. These sections present core concepts and important methodological developments that inform current standards and recommendations. We conclude by taking a direct and practical approach.

Inconsistent and imprecise terminology used in the context of development and evaluation of evidence syntheses is problematic for authors, peer reviewers and editors, and may lead to the application of inappropriate methods and tools. In response, we endorse use of the basic terms (Table 6.1 ) defined in the PRISMA 2020 statement [ 93 ]. In addition, we have identified several problematic expressions and nomenclature. In Table 6.2 , we compile suggestions for preferred terms less likely to be misinterpreted.

Terms relevant to the reporting of health care–related evidence syntheses a

a Reproduced from Page and colleagues [ 93 ]

Terminology suggestions for health care–related evidence syntheses

a For example, meta-aggregation, meta-ethnography, critical interpretative synthesis, realist synthesis

b This term may best apply to the synthesis in a mixed methods systematic review in which data from different types of evidence (eg, qualitative, quantitative, economic) are summarized [ 64 ]

We also propose a Concise Guide (Table 6.3 ) that summarizes the methods and tools recommended for the development and evaluation of nine types of evidence syntheses. Suggestions for specific tools are based on the rigor of their development as well as the availability of detailed guidance from their developers to ensure their proper application. The formatting of the Concise Guide addresses a well-known source of confusion by clearly distinguishing the underlying methodological constructs that these tools were designed to assess. Important clarifications and explanations follow in the guide’s footnotes; associated websites, if available, are listed in Additional File 6 .

To encourage uptake of best practices, journal editors may consider adopting or adapting the Concise Guide in their instructions to authors and peer reviewers of evidence syntheses. Given the evolving nature of evidence synthesis methodology, the suggested methods and tools are likely to require regular updates. Authors of evidence syntheses should monitor the literature to ensure they are employing current methods and tools. Some types of evidence syntheses (eg, rapid, economic, methodological) are not included in the Concise Guide; for these, authors are advised to obtain recommendations for acceptable methods by consulting with their target journal.

We encourage the appropriate and informed use of the methods and tools discussed throughout this commentary and summarized in the Concise Guide (Table 6.3 ). However, we caution against their application in a perfunctory or superficial fashion. This is a common pitfall among authors of evidence syntheses, especially as the standards of such tools become associated with acceptance of a manuscript by a journal. Consequently, published evidence syntheses may show improved adherence to the requirements of these tools without necessarily making genuine improvements in their performance.

In line with our main objective, the suggested tools in the Concise Guide address the reliability of evidence syntheses; however, we recognize that the utility of systematic reviews is an equally important concern. An unbiased and thoroughly reported evidence synthesis may still not be highly informative if the evidence itself that is summarized is sparse, weak and/or biased [ 24 ]. Many intervention systematic reviews, including those developed by Cochrane [ 203 ] and those applying GRADE [ 202 ], ultimately find no evidence, or find the evidence to be inconclusive (eg, “weak,” “mixed,” or of “low certainty”). This often reflects the primary research base; however, it is important to know what is known (or not known) about a topic when considering an intervention for patients and discussing treatment options with them.

Alternatively, the frequency of “empty” and inconclusive reviews published in the medical literature may relate to limitations of conventional methods that focus on hypothesis testing; these have emphasized the importance of statistical significance in primary research and effect sizes from aggregate meta-analyses [ 183 ]. It is becoming increasingly apparent that this approach may not be appropriate for all topics [ 130 ]. Development of the GRADE approach has facilitated a better understanding of significant factors (beyond effect size) that contribute to the overall certainty of evidence. Other notable responses include the development of integrative synthesis methods for the evaluation of complex interventions [ 230 , 231 ], the incorporation of crowdsourcing and machine learning into systematic review workflows (eg the Cochrane Evidence Pipeline) [ 2 ], the shift in paradigm to living systemic review and NMA platforms [ 232 , 233 ] and the proposal of a new evidence ecosystem that fosters bidirectional collaborations and interactions among a global network of evidence synthesis stakeholders [ 234 ]. These evolutions in data sources and methods may ultimately make evidence syntheses more streamlined, less duplicative, and more importantly, they may be more useful for timely policy and clinical decision-making; however, that will only be the case if they are rigorously reported and conducted.

We look forward to others’ ideas and proposals for the advancement of methods for evidence syntheses. For now, we encourage dissemination and uptake of the currently accepted best tools and practices for their development and evaluation; at the same time, we stress that uptake of appraisal tools, checklists, and software programs cannot substitute for proper education in the methodology of evidence syntheses and meta-analysis. Authors, peer reviewers, and editors must strive to make accurate and reliable contributions to the present evidence knowledge base; online alerts, upcoming technology, and accessible education may make this more feasible than ever before. Our intention is to improve the trustworthiness of evidence syntheses across disciplines, topics, and types of evidence syntheses. All of us must continue to study, teach, and act cooperatively for that to happen.

Acknowledgements

Michelle Oakman Hayes for her assistance with the graphics, Mike Clarke for his willingness to answer our seemingly arbitrary questions, and Bernard Dan for his encouragement of this project.

Authors’ contributions

All authors participated in the development of the ideas, writing, and review of this manuscript. The author(s) read and approved the final manuscript.

The work of John Ioannidis has been supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford University.

Declarations

The authors declare no competing interests.

This article has been published simultaneously in BMC Systematic Reviews, Acta Anaesthesiologica Scandinavica, BMC Infectious Diseases, British Journal of Pharmacology, JBI Evidence Synthesis, the Journal of Bone and Joint Surgery Reviews , and the Journal of Pediatric Rehabilitation Medicine .

Publisher’ s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Open access
• Published: 23 August 2024

The role of emotions in academic performance of undergraduate medical students: a narrative review

• Nora Alshareef 1 , 2 ,
• Ian Fletcher 2 &
• Sabir Giga 2  

BMC Medical Education volume  24 , Article number:  907 ( 2024 ) Cite this article

180 Accesses

Metrics details

This paper is devoted to a narrative review of the literature on emotions and academic performance in medicine. The review aims to examine the role emotions play in the academic performance of undergraduate medical students.

Eight electronic databases were used to search the literature from 2013 to 2023, including Academic Search Ultimate, British Education Index, CINAHL, Education Abstract, ERIC, Medline, APA Psych Articles and APA Psych Info. Using specific keywords and terms in the databases, 3,285,208 articles were found. After applying the predefined exclusion and inclusion criteria to include only medical students and academic performance as an outcome, 45 articles remained, and two reviewers assessed the quality of the retrieved literature; 17 articles were selected for the narrative synthesis.

The findings indicate that depression and anxiety are the most frequently reported variables in the reviewed literature, and they have negative and positive impacts on the academic performance of medical students. The included literature also reported that a high number of medical students experienced test anxiety during their study, which affected their academic performance. Positive emotions lead to positive academic outcomes and vice versa. However, Feelings of shame did not have any effect on the academic performance of medical students.

The review suggests a significant relationship between emotions and academic performance among undergraduate medical students. While the evidence may not establish causation, it underscores the importance of considering emotional factors in understanding student performance. However, reliance on cross-sectional studies and self-reported data may introduce recall bias. Future research should concentrate on developing anxiety reduction strategies and enhancing mental well-being to improve academic performance.

Peer Review reports

Introduction

Studying medicine is a multi-dimensional process involving acquiring medical knowledge, clinical skills, and professional attitudes. Previous research has found that emotions play a significant role in this process [ 1 , 2 ]. Different types of emotions are important in an academic context, influencing performance on assessments and evaluations, reception of feedback, exam scores, and overall satisfaction with the learning experience [ 3 ]. In particular, medical students experience a wide range of emotions due to many emotionally challenging situations, such as experiencing a heavy academic workload, being in the highly competitive field of medicine, retaining a large amount of information, keeping track of a busy schedule, taking difficult exams, and dealing with a fear of failure [ 4 , 5 , 6 ].Especially during their clinical years, medical students may experience anxiety when interacting with patients who are suffering, ill, or dying, and they must work with other healthcare professionals. Therefore, it is necessary to understand the impact of emotions on medical students to improve their academic outcomes [ 7 ].

To distinguish the emotions frequently experienced by medical students, it is essential to define them. Depression is defined by enduring emotions of sadness, despair, and a diminished capacity for enjoyment or engagement in almost all activities [ 4 ]. Negative emotions encompass unpleasant feelings such as anger, fear, sadness, and anxiety, and they frequently cause distress [ 8 ]. Anxiety is a general term that refers to a state of heightened nervousness or worry, which can be triggered by various factors. Test anxiety, on the other hand, is a specific type of anxiety that arises in the context of taking exams or assessments. Test anxiety is characterised by physiological arousal, negative self-perception, and a fear of failure, which can significantly impair a student’s ability to perform well academically [ 9 , 10 ]. Shame is a self-conscious emotion that arises from the perception of having failed to meet personal or societal standards. It can lead to feelings of worthlessness and inadequacy, severely impacting a student’s motivation and academic performance [ 11 , 12 ]. In contrast, positive emotions indicate a state of enjoyable involvement with the surroundings, encompassing feelings of happiness, appreciation, satisfaction, and love [ 8 ].

Academic performance generally refers to the outcomes of a student’s learning activities, often measured through grades, scores, and other formal assessments. Academic achievement encompasses a broader range of accomplishments, including mastery of skills, attainment of knowledge, and the application of learning in practical contexts. While academic performance is often quantifiable, academic achievement includes qualitative aspects of a student’s educational journey [ 13 ].

According to the literature, 11–40% of medical students suffer from stress, depression, and anxiety due to the intensity of medical school, and these negative emotions impact their academic achievement [ 14 , 15 ]. Severe anxiety may impair memory function, decrease concentration, lead to a state of hypervigilance, and interfere with judgment and cognitive function, further affecting academic performance [ 16 ]. However, some studies have suggested that experiencing some level of anxiety has a positive effect and serves as motivation that can improve academic performance [ 16 , 17 ].

Despite the importance of medical students’ emotions and their relation to academic performance, few studies have been conducted in this area. Most of these studies have focused on the prevalence of specific emotions without correlating with medical students’ academic performance. Few systematic reviews have addressed the emotional challenges medical students face. However, there is a lack of comprehensive reviews that discuss the role of emotions and academic outcomes. Therefore, this review aims to fill this gap by exploring the relationship between emotions and the academic performance of medical students.

Aim of the study

This review aims to examine the role emotions play in the academic performance of undergraduate medical students.

A systematic literature search examined the role of emotions in medical students’ academic performance. The search adhered to the concepts of a systematic review, following the criteria of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 18 ]. Then, narrative synthesise was done to analyse the retrieved literature and synthesise the results. A systematic literature search and narrative review provide complete coverage and flexibility to explore and understand findings. Systematic search assures rigour and reduces bias, while narrative synthesis allows for flexible integration and interpretation. This balance improves review quality and utility.

Eligibility criteria

Inclusion criteria.

The study’s scope was confined to January 2013 to December 2023, focusing exclusively on undergraduate medical students. The research encompassed articles originating within medical schools worldwide, accepting content from all countries. The criteria included only full-text articles in English published in peer-reviewed journals. Primary research was considered, embracing quantitative and mixed-method research. The selected studies had to explicitly reference academic performance, test results, or GPA as key outcomes to address the research question.

Exclusion criteria

The study excluded individuals beyond the undergraduate medical student demographic, such as students in other health fields and junior doctors. There was no imposed age limit for the student participants. The research specifically focused on articles within medical schools, excluding those from alternative settings. It solely considered full-text articles in English-language peer-reviewed journals. Letters or commentary articles were excluded, and the study did not limit itself to a particular type of research. Qualitative studies were excluded from the review because they did not have the quantitative measures required to answer the review’s aim. This review excluded articles on factors impacting academic performance, those analysing nursing students, and gender differences. The reasons and numbers for excluding articles are shown in Table  1 .

Information sources

Eight electronic databases were used to search the literature. These were the following: Academic Search Ultimate, British Education Index, CINAHL, Education Abstract, ERIC, Medline, APA Psych Articles and APA Psych Info. The databases were chosen from several fields based on relevant topics, including education, academic evaluation and assessment, medical education, psychology, mental health, and medical research. Initially, with the help of a subject librarian, the researcher used all the above databases; the databases were searched with specific keywords and terms, and the terms were divided into the following concepts emotions, academic performance and medical students. Google Scholar, EBSCOhost, and the reference list of the retrieved articles were also used to identify other relevant articles.

Search strategy

This review started with a search of the databases. Eight electronic databases were used to search the literature from 2013 to 2023. Specific keywords and terms were used to search the databases, resulting in 3,285,208 articles. After removing duplicates, letters and commentary, this number was reduced to 1,637 articles. Exclusion and inclusion criteria were then applied, resulting in 45 articles. After two assessors assessed the literature, 17 articles were selected for the review. The search terms are as follows:

Keywords: Emotion, anxiety, stress, empathy, test anxiety, exam anxiety, test stress, exam stress, depression, emotional regulation, test scores, academic performance, grades, GPA, academic achievement, academic success, test result, assessment, undergraduate medical students and undergraduate medical education.

Emotions: TI (Emotion* OR Anxiety OR Stress OR empathy) OR emotion* OR (test anxiety or exam anxiety or test stress or exam stress) OR (depression) OR AB ((Emotion* OR Anxiety OR Stress OR empathy) OR emotion* OR (test anxiety or exam anxiety or test stress or exam stress)) (MH “Emotions”) OR (MH “Emotional Regulation”) DE “EMOTIONS”.

Academic performance: TI (test scores or academic performance or grades or GPA) OR (academic achievement or academic performance or academic success) OR (test result* OR assessment*) OR AB (test scores or academic performance or grades or GPA) OR (academic achievement or academic performance or academic success) OR test result* OR assessment*.

Medical Students: TI (undergraduate medical students OR undergraduate medical education) OR AB (undergraduate medical students OR undergraduate medical education), TI “medical students” OR AB “medical students” DE “Medical Students”.

Selection process

This literature review attempts to gather only peer-reviewed journal articles published in English on undergraduate medical students’ negative and positive emotions and academic performance from January 2013 to December 2023. Their emotions, including depression, anxiety, physiological distress, shame, happiness, joy, and all emotions related to academic performance, were examined in quantitative research and mixed methods.

Moreover, to focus the search, the author specified and defined each keyword using advanced search tools, such as subject headings in the case of the Medline database. The author used ‘MeSH 2023’ as the subject heading, then entered the term ‘Emotion’ and chose all the relevant meanings. This method was applied to most of the keywords.

Studies were included based on predefined criteria related to study design, participants, exposure, outcomes, and study types. Two independent reviewers screened each record, and the report was retrieved. In the screening process, reviewers independently assessed each article against the inclusion criteria, and discrepancies were resolved through consensus during regular team meetings. In cases of persistent disagreement, a third reviewer was consulted. Endnote library program was used for the initial screening phase. This tool was used to identify duplicates, facilitated the independent screening of titles and abstracts and helped to retrieve the full-text articles. The reasons for excluding the articles are presented in Table  1 .

Data collection process

Two independent reviewers extracted data from the eligible studies, with any discrepancies resolved through discussion and consensus. If the two primary reviewers could not agree, a third reviewer served as an arbitrator. For each included study, the following information was extracted and recorded in a standardised database: first author name, publication year, study design, sample characteristics, details of the emotions exposed, outcome measures, and results.

Academic performance as an outcome for medical students was defined to include the following: Exam scores (e.g., midterm, final exams), Clinical assessments (e.g., practical exams, clinical rotations), Overall grade point average (GPA) or any other relevant indicators of academic achievement.

Data were sought for all outcomes, including all measures, time points, and analyses within each outcome domain. In cases where studies reported multiple measures or time points, all relevant data were extracted to provide a comprehensive overview of academic performance. If a study reported outcomes beyond the predefined domains, inclusion criteria were established to determine whether these additional outcomes would be included in the review. This involved assessing relevance to the primary research question and alignment with the predefined outcome domains.

Quality assessment

The quality and risk of bias in included studies were assessed using the National Institute of Health’s (NIH) critical appraisal tool. The tool evaluates studies based on the following domains: selection bias, performance bias, detection bias, attrition bias, reporting bias, and other biases. Two independent reviewers assessed the risk of bias in each included study. Reviewers worked collaboratively to reach a consensus on assessments. Discrepancies were resolved through discussion and consensus. In cases of persistent disagreement, a third reviewer was consulted.

To determine the validity of eligible articles, all the included articles were critically appraised, and all reviewers assessed bias. The validity and reliability of the results were assessed by using objective measurement. Each article was scored out of 14, with 14 indicating high-quality research and 1 indicating low-quality research. High-quality research, according to the NIH (2013), includes a clear and focused research question, defines the study population, features a high participation rate, mentions inclusion and exclusion criteria, uses clear and specific measurements, reports results in detail, lists the confounding factors and lists the implications for the local community. Therefore, an article was scored 14 if it met all criteria of the critical appraisal tool. Based on scoring, each study was classified into one of three quality categories: good, fair or poor. The poorly rated articles mean their findings were unreliable, and they will not be considered, including two articles [ 16 , 19 ]. Seventeen articles were chosen after critical appraisal using the NIH appraisal tool, as shown in Table  2 .

Effect measures

For each outcome examined in the included studies, various effect measures were utilised to quantify the relationship between emotions and academic performance among undergraduate medical students. The effect measures commonly reported across the studies included prevalence rat, correlation coefficients, and mean differences. The reviewer calculated the effect size for the studies that did not report the effect. The choice of effect measure depended on the nature of the outcome variable and the statistical analysis conducted in each study. These measures were used to assess the strength and direction of the association between emotional factors and academic performance.

The synthesis method

The findings of individual studies were summarised to highlight crucial characteristics. Due to the predicted heterogeneity, the synthesis involved pooling effect estimates and using a narrative method. A narrative synthesis approach was employed in the synthesis of this review to assess and interpret the findings from the included studies qualitatively. The narrative synthesis involved a qualitative examination of the content of each study, focusing on identifying common themes. This synthesis was employed to categorise and interpret data, allowing for a nuanced understanding of the synthesis. Themes related to emotions were identified and extracted for synthesis. Control-value theory [ 20 ] was used as an overarching theory, providing a qualitative synthesis of the evidence and contributing to a deeper understanding of the research question. If the retrieved articles include populations other than medical, such as dental students or non-medical students, the synthesis will distinguish between them and summarise the findings of the medical students only, highlighting any differences or similarities.

The Control-Value Theory, formulated by Pekrun (2006), is a conceptual framework that illustrates the relationship between emotions and academic achievement through two fundamental assessments: control and value. Control pertains to the perceived ability of a learner to exert influence over their learning activities and the results they achieve. Value relates to a student’s significance to these actions and results. The theory suggests that students are prone to experiencing good feelings, such as satisfaction and pride when they possess a strong sense of control and importance towards their academic assignments. On the other hand, individuals are prone to encountering adverse emotions (such as fear and embarrassment) when they perceive a lack of control or worth in these particular occupations. These emotions subsequently impact students’ motivation, learning strategies, and, eventually, their academic achievement. The relevance of control-value theory in reviewing medical student emotions and their influence on academic performance is evident for various reasons. This theory offers a complete framework that facilitates comprehending the intricate connection between emotions and academic achievement. It considers positive and negative emotions, providing a comprehensive viewpoint on how emotions might influence learning and performance. The relevance of control and value notions is particularly significant for medical students due to their frequent exposure to high-stakes tests and difficult courses. Gaining insight into the students’ perception of their power over academic assignments and the importance they attach to their medical education might aid in identifying emotional stimuli and devising remedies. Multiple research has confirmed the theory’s assertions, showing the critical influence of control and value evaluations on students’ emotional experiences and academic achievements [ 21 , 22 ].

Data extraction

For this step, a data extraction sheet was developed using the data extraction template provided by the Cochrane Handbook. To ensure the review is evidence-based and bias-free, the Cochrane Handbook strongly suggests that more than one reviewer review the data. Therefore, the main researcher extracted the data from the included studies, and another reviewer checked the included, excluded and extracted data. Any disagreements were resolved via discussion by a third reviewer. The data extraction Table  2 identified all study features, including the author’s name, the year of publication, the method used the aim of the study, the number and description of participants, data collection tools, and study findings.

Finalisation of references and study characteristics

Prisma sheet and the summary of final studies that have been used for the review.

When the keywords and search terms related to emotions, as mentioned above, in the eight databases listed, 3,285,208 articles were retrieved. After using advanced search and subject headings, the number of articles increased to 3,352,371. Similarly, searching for the second keyword, ‘academic performance,’ using all the advanced search tools yielded 8,119,908 articles. Searching for the third keyword, ‘medical students’, yielded 145,757 articles. All terms were searched in article titles and abstracts. After that, the author combined all search terms by using ‘AND’ and applied the time limit from 2013 to 2023; the search narrowed to 2,570 articles. After duplicates, letters and commentary were excluded, the number was reduced to 1,637 articles. After reading the title and abstract to determine relevance to the topic and applying the exclusion and inclusion criteria mentioned above, 45 articles remained; after the quality of the retrieved literature was assessed by two reviewers, 17 articles were selected for the review. The PRISMA flow diagram summarising the same is presented in Fig.  1 . Additionally, One article by Ansari et al. (2018) was selected for the review; it met most inclusion and exclusion criteria except that the outcome measure is cognitive function and not academic performance. Therefore, it was excluded from the review. Figure  1 shows the Prisma flow diagram (2020) of studies identified from the databases.

Prisma flow diagram (2020)

Study characteristics

Table  2 , summarising the characteristics of the included studies, is presented below.

Findings of the study

Country of the study.

Many of the studies were conducted in developing countries, with the majority being conducted in Europe ( n  = 4), followed by Pakistan ( n  = 2), then Saudi Arabia ( n  = 2), and the United States ( n  = 2). The rest of the studies were conducted in South America ( n  = 1), Morocco ( n  = 1), Brazil ( n  = 1), Australia ( n  = 1), Iran ( n  = 1), South Korea ( n  = 1) and Bosnia and Herzegovina ( n  = 1). No included studies were conducted in the United Kingdom.

Study design

Regarding study design, most of the included articles used a quantitative methodology, including 12 cross-sectional studies. There were two randomised controlled trials, one descriptive correlation study, one cohort study, and only one mixed-method study.

Population and study setting

Regarding population and setting, most of the studies focused on all medical students studying in a medical school setting, from first-year medical students to those in their final year. One study compared medical students with non-medical students; another combined medical students with dental students.

The study aims varied across the included studies. Seven studies examined the prevalence of depression and anxiety among medical students and their relation to academic performance. Four studies examined the relationship between test anxiety and academic performance in medical education. Four studies examined the relationship between medical students’ emotions and academic achievements. One study explored the influence of shame on medical students’ learning.

Study quality

The studies were assessed for quality using tools created by the NIH (2013) and then divided into good, fair, and poor based on these results. Nine studies had a high-quality methodology, seven achieved fair ratings, and only three achieved poor ratings. The studies that were assigned the poor rating were mainly cross-sectional studies, and the areas of weakness were due to the study design, low response rate, inadequate reporting of the methodology and statistics, invalid tools, and unclear research goals.

Outcome measures

Most of the outcome measures were heterogenous and self-administered questionnaires; one study used focus groups and observation ward assessment [ 23 ]. All the studies used the medical students’ academic grades.

Results of the study

The prevalence rate of psychological distress in the retrieved articles.

Depression and anxiety are the most common forms of psychological distress examined concerning academic outcomes among medical students. Studies consistently show concerningly high rates, with prevalence estimates ranging from 7.3 to 66.4% for anxiety and 3.7–69% for depression. These findings indicate psychological distress levels characterised as moderate to high based on common cut-off thresholds have a clear detrimental impact on academic achievement [ 16 , 24 , 25 , 26 ].

The studies collectively examine the impact of psychological factors on academic performance in medical education contexts, using a range of effect sizes to quantify their findings. Aboalshamat et al. (2015) identified a small effect size ( η 2 = 0.018) for depression’s impact on academic performance, suggesting a modest influence. Mihailescu (2016) found a significant negative correlation between levels of depression/anxiety (rho=-0.14, rho=-0.19), academic performance and GPA among medical students. Burr and Beck Dallaghan (2019) reported professional efficacy explaining 31.3% of the variance in academic performance, indicating a significant effect size. However, Del-Ben (2013) et al. did not provide the significant impact of affective changes on academic achievement, suggesting trivial effect sizes for these factors.

In conclusion, anxiety and depression, both indicators of psychological discomfort, are common among medical students. There is a link between distress and poor academic performance results, implying that this relationship merits consideration. Table  3 below shows the specific value of depression and anxiety in retrieved articles.

Test anxiety

In this review, four studies examined the relationship between test anxiety and academic performance in medical education [ 27 , 28 , 29 , 30 ]. The studies found high rates of test anxiety among medical students, ranging from 52% [ 27 ] to as high as 81.1% [ 29 ]. Final-year students tend to experience the highest test anxiety [ 29 ].

Test anxiety has a significant negative correlation with academic performance measures and grade point average (GPA) [ 27 , 28 , 29 ]. Green et al. (2016) found that test anxiety was moderately negatively correlated with USMLE score ( r = − 0.24, p  = 0.00); high test anxiety was associated with low USMLE scores in the control group, further suggesting that anxiety can adversely affect performance. The findings that a test-taking strategy course reduced anxiety without improving test scores highlight the complex nature of anxiety’s impact on performance.

Nazir et al. (2021) found that excellent female medical students reported significantly lower test anxiety than those with low academic grades, with an odds ratio of 1.47, indicating that students with higher test anxiety are more likely to have lower academic grades. Kim’s (2016) research shows moderate correlations between test anxiety and negative achievement emotions such as anxiety and boredom, but interestingly, this anxiety does not significantly affect practical exam scores (OSCE) or GPAs. However, one study found that examination stress enhanced academic performance with a large effect size (W = 0.78), with stress levels at 47.4% among their sample, suggesting that a certain stress level before exams may be beneficial [ 30 ].

Three papers explored shame’s effect on medical students’ academic achievement [ 24 , 31 , 32 ]. Hayat et al. (2018) reported that academic feelings, like shame, significantly depend on the academic year. shame was found to have a slight negative and significant correlation with the academic achievement of learners ( r =-0.15). One study found that some medical students felt shame during simulations-based education examinations because they had made incorrect decisions, which decreased their self-esteem and motivation to learn. However, others who felt shame were motivated to study harder to avoid repeating the same mistakes [ 23 ].

Hautz (2017) study examined how shame affects medical students’ learning using a randomised controlled trial where researchers divided the students into two groups: one group performed a breast examination on mannequins and the other group on actual patients. The results showed that students who performed the clinical examination on actual patients experienced significantly higher levels of shame but performed better in examinations than in the mannequin group. In the final assessments on standardised patients, both groups performed equally well. Therefore, shame decreased with more clinical practice, but shame did not have significant statistics related to learning or performance. Similarly, Burr and Dallaghan (2019) reported that the shame level of medical students was (40%) but had no association with academic performance.

Academic performance, emotions and medical students

Three articles discussed medical students’ emotions and academic performance [ 23 , 24 , 32 ]. Burr and Dallaghan (2019) examine the relationship between academic success and emotions in medical students, such as pride, hope, worry, and shame. It emphasises the links between academic accomplishment and professional efficacy, as well as hope, pride, worry, and shame. Professional efficacy was the most significant factor linked to academic performance, explaining 31.3% of the variance. The importance of emotions on understanding, processing of data, recall of memories, and cognitive burden is emphasised throughout the research. To improve academic achievement, efforts should be made to increase student self-efficacy.

Hayat et al. (2018) found that positive emotions and intrinsic motivation are highly connected with academic achievement, although emotions fluctuate between educational levels but not between genders. The correlations between negative emotions and academic achievement, ranging from − 0.15 to -0.24 for different emotions, suggest small but statistically significant adverse effects.

Behren et al.‘s (2019) mixed-method study found that students felt various emotions during the simulation, focusing on positive emotions and moderate anxiety. However, no significant relationships were found between positive emotions and the student’s performance during the simulation [ 23 ].

This review aims to investigate the role of emotions in the academic performance of undergraduate medical students. Meta-analysis cannot be used because of the heterogeneity of the data collection tools and different research designs [ 33 ]. Therefore, narrative synthesis was adopted in this paper. The studies are grouped into four categories as follows: (1) The effect of depression and anxiety on academic performance, (2) Test anxiety and academic achievement, (3) Shame and academic performance, and (4) Academic performance, emotions and medical students. The control-value theory [ 20 ], will be used to interpret the findings.

The effect of depression and anxiety on academic performance

According to the retrieved research, depression and anxiety can have both a negative and a positive impact on the academic performance of medical students. Severe anxiety may impair memory function, decrease concentration, lead to a state of hypervigilance, interfere with judgment and cognitive function, and further affect academic performance [ 4 ]. Most of the good-quality retrieved articles found that anxiety and depression were associated with low academic performance [ 16 , 24 , 25 , 26 ]. Moreira (2018) and Mihailescu (2016) found that higher depression levels were associated with more failed courses and a lower GPA. However, they did not find any association between anxiety level and academic performance.

By contrast, some studies have suggested that experiencing some level of anxiety reinforces students’ motivation to improve their academic performance [ 16 , 34 ]. Zalihic et al. (2017) conducted a study to investigate anxiety sensitivity about academic success and noticed a positive relationship between anxiety level and high academic scores; they justified this because when medical students feel anxious, they tend to prepare and study more, and they desire to achieve better scores and fulfil social expectations. Similarly, another study found anxiety has a negative impact on academic performance when excessive and a positive effect when manageable, in which case it encourages medical students and motivates them to achieve higher scores [ 35 ].

In the broader literature, the impact of anxiety on academic performance has contradictory research findings. While some studies suggest that having some level of anxiety can boost students’ motivation to improve their academic performance, other research has shown that anxiety has a negative impact on their academic success [ 36 , 37 ]. In the cultural context, education and anxiety attitudes differ widely across cultures. High academic pressure and societal expectations might worsen anxiety in many East Asian societies. Education is highly valued in these societies, frequently leading to significant academic stress. This pressure encompasses attaining high academic marks and outperformance in competitive examinations. The academic demands exerted on students can result in heightened levels of anxiety. The apprehension of not meeting expectations can lead to considerable psychological distress and anxiety, which can appear in their physical and mental health and academic achievement [ 38 , 39 ].

Test anxiety and academic achievement

The majority of the studies reviewed confirm that test anxiety negatively affects academic performance [ 27 , 28 , 29 ]. Several studies have found a significant correlation between test anxiety and academic achievement, indicating that higher levels of test anxiety are associated with lower exam scores and lower academic performance [ 40 , 41 ]. For example, Green et al. (2016) RCT study found that test anxiety has a moderately significant negative correlation with the USMLE score. They found that medical students who took the test-taking strategy course had lower levels of test anxiety than the control group, and their test anxiety scores after the exam had improved from the baseline. Although their test anxiety improved after taking the course, there was no significant difference in the exam scores between students who had and had not taken the course. Therefore, the intervention they used was not effective. According to the control-value theory, this intervention can be improved if they design an emotionally effective learning environment, have a straightforward instructional design, foster self-regulation of negative emotions, and teach students emotion-oriented regulation [ 22 ].

Additionally, according to this theory, students who perceive exams as difficult are more likely to experience test anxiety because test anxiety results from a student’s negative appraisal of the task and outcome values, leading to a reduction in their performance. This aligns with Kim’s (2016) study, which found that students who believed that the OSCE was a problematic exam experienced test anxiety more than other students [ 9 , 22 , 42 ].

In the wider literature, a meta-analysis review by von der Embse (2018) found a medium significant negative correlation ( r =-0.24) between test anxiety and test performance in undergraduate educational settings [ 43 ] . Also, they found a small significant negative correlation ( r =-0.17) between test anxiety and GPA. This indicates that higher levels of test anxiety are associated with lower test performance. Moreover, Song et al. (2021) experimental study examined the effects of test anxiety on working memory capacity and found that test anxiety negatively correlated with academic performance [ 44 ]. Therefore, the evidence from Song’s study suggests a small but significant effect of anxiety on working memory capacity. However, another cross-sectional study revealed that test anxiety in medical students had no significant effect on exam performance [ 45 ]. The complexities of this relationship necessitate additional investigation. Since the retrieved articles are from different countries, it is critical to recognise the possible impact of cultural differences on the impact of test anxiety. Cultural factors such as different educational systems, assessment tools and societal expectations may lead to variances in test anxiety experience and expression across diverse communities [ 46 , 47 ]. Culture has a substantial impact on how test anxiety is expressed and evaluated. Research suggests that the degree and manifestations of test anxiety differ among different cultural settings, emphasising the importance of using culturally validated methods to evaluate test anxiety accurately. A study conducted by Lowe (2019) with Canadian and U.S. college students demonstrated cultural variations in the factors contributing to test anxiety. Canadian students exhibited elevated levels of physiological hyperarousal, but U.S. students had more pronounced cognitive interference. These variations indicate that the cultural environment has an influence on how students perceive and respond to test anxiety, resulting in differing effects on their academic performance in different cultures. Furthermore, scholars highlight the significance of carrying out meticulous instruments to assess test anxiety, which are comparable among diverse cultural cohorts. This technique guarantees that the explanations of test scores are reliable and can be compared across different populations. Hence, it is imperative to comprehend and tackle cultural disparities in order to create efficient interventions and assistance for students who encounter test anxiety in diverse cultural environments. Therefore, there is a need for further studies to examine the level of test anxiety and cultural context.

Shame and academic performance

The review examined three studies that discuss the impact of feelings of shame on academic performance [ 23 , 24 , 48 ]. Generally, shame is considered a negative emotion which involves self-reflection and self-evaluation, and it leads to rumination and self-condemnation [ 49 ]. Intimate examinations conducted by medical students can induce feelings of shame, affecting their ability to communicate with patients and their clinical decisions. Shame can increase the avoidance of intimate physical examinations and also encourage clinical practice [ 23 , 24 , 48 ].

One study found that some medical students felt shame during simulations-based education examinations because they had made incorrect decisions, which decreased their self-esteem and motivation to learn. However, others who felt shame were motivated to study harder to avoid repeating the same mistakes [ 23 ]. Shame decreased with more clinical practice, but shame did not affect their learning or performance [ 48 ]. The literature on how shame affects medical students’ learning is inconclusive [ 31 ].

In the broader literature, shame is considered maladaptive, leading to dysfunctional behaviour, encouraging withdrawal and avoidance of events and inhibiting social interaction. However, few studies have been conducted on shame in the medical field. Therefore, more research is needed to investigate the role of shame in medical students’ academic performance [ 49 ]. In the literature, there are several solutions that can be used to tackle the problem of shame in medical education; it is necessary to establish nurturing learning settings that encourage students to openly discuss their problems and mistakes without the worry of facing severe criticism. This can be accomplished by encouraging medical students to participate in reflective practice, facilitating the processing of their emotions, and enabling them to derive valuable insights from their experiences, all while avoiding excessive self-blame [ 50 ]. Offering robust mentorship and support mechanisms can assist students in effectively managing the difficulties associated with intimate examinations. Teaching staff have the ability to demonstrate proper behaviours and provide valuable feedback and effective mentoring [ 51 ]. Training and workshops that specifically target communication skills and the handling of sensitive situations can effectively equip students to handle intimate tests, hence decreasing the chances of them avoiding such examinations due to feelings of shame [ 52 ].

The literature review focused on three studies that examined the relationship between emotions and the academic achievements of medical students [ 23 , 24 , 32 ].

Behren et al. (2019) mixed-method study on the achievement emotions of medical students during simulations found that placing students in challenging clinical cases that they can handle raises positive emotions. Students perceived these challenges as a positive drive for learning and mild anxiety was considered beneficial. However, the study also found non-significant correlations between emotions and performance during the simulation, indicating a complex relationship between emotions and academic performance. The results revealed that feelings of frustration were perceived to reduce students’ interest and motivation for studying, hampered their decision-making process, and negatively affected their self-esteem, which is consistent with the academic achievement emotions literature where negative emotions are associated with poor intrinsic motivation and reduced the ability to learn [ 3 ].

The study also emphasises that mild anxiety can have positive effects, corroborated by Gregor (2005), which posits that moderate degrees of anxiety can improve performance. The author suggests that an ideal state of arousal (which may be experienced as anxiety) enhances performance. Mild anxiety is commonly seen as a type of psychological stimulation that readies the body for upcoming challenges, frequently referred to as a “fight or flight” response. Within the realm of academic performance, this state of heightened arousal can enhance concentration and optimise cognitive functions such as memory, problem-solving skills, and overall performance. However, once the ideal point is surpassed, any additional increase in arousal can result in a decline in performance [ 53 ]. This is additionally supported by Cassady and Johnson (2002), who discovered that a specific level of anxiety can motivate students to engage in more comprehensive preparation, hence enhancing their performance.

The reviewed research reveals a positive correlation between positive emotions and academic performance and a negative correlation between negative emotions and academic performance. These findings align with the control–value theory [ 8 , 22 ], which suggests that positive emotions facilitate learning through mediating factors, including cognitive learning strategies such as strategic thinking, critical thinking and problem-solving and metacognitive learning strategies such as monitoring, regulating, and planning students’ intrinsic and extrinsic motivation. Additionally, several studies found that extrinsic motivation from the educational environment and the application of cognitive and emotional strategies improve students’ ability to learn and, consequently, their academic performance [ 23 , 24 , 32 ]. By contrast, negative emotions negatively affect academic performance. This is because negative emotions reduce students’ motivation, concentration, and ability to process information [ 23 , 24 , 32 ].

Limitations of the study

This review aims to thoroughly investigate the relationship between emotions and academic performance in undergraduate medical students, but it has inherent limitations. Overall, the methodological quality of the retrieved studies is primarily good and fair. Poor-quality research was excluded from the synthesis. The good-quality papers demonstrated strengths in sampling techniques, data analysis, collection and reporting. However, most of the retrieved articles used cross-section studies, and the drawback of this is a need for a more causal relationship, which is a limitation in the design of cross-sectional studies. Furthermore, given the reliance on self-reported data, there were concerns about potential recall bias. These methodological difficulties were noted in most of the examined research. When contemplating the implications for practice and future study, the impact of these limitations on the validity of the data should be acknowledged.

The limitation of the review process and the inclusion criteria restricted the study to articles published from January 2013 to December 2023, potentially overlooking relevant research conducted beyond this timeframe. Additionally, the exclusive focus on undergraduate medical students may constrain the applicability of findings to other health fields or educational levels.

Moreover, excluding articles in non-English language and those not published in peer-reviewed journals introduces potential language and publication biases. Reliance on electronic databases and specific keywords may inadvertently omit studies using different terms or indexing. While the search strategy is meticulous, it might not cover every relevant study due to indexing and database coverage variations. However, the two assessors’ involvement in study screening, selection, data extraction, and quality assessment improved the robustness of the review and ensured that it included all the relevant research.

In conclusion, these limitations highlight the need for careful interpretation of the study’s findings and stress the importance of future research addressing these constraints to offer a more comprehensive understanding of the nuanced relationship between emotions and academic performance in undergraduate medical education.

Conclusion and future research

The review exposes the widespread prevalence of depression, anxiety and test anxiety within the medical student population. The impact on academic performance is intricate, showcasing evidence of adverse and favourable relationships. Addressing the mental health challenges of medical students necessitates tailored interventions for enhancing mental well-being in medical education. Furthermore, it is crucial to create practical strategies considering the complex elements of overcoming test anxiety. Future research should prioritise the advancement of anxiety reduction strategies to enhance academic performance, focusing on the control-value theory’s emphasis on creating an emotionally supportive learning environment. Additionally, Test anxiety is very common among medical students, but the literature has not conclusively determined its actual effect on academic performance. Therefore, there is a clear need for a study that examines the relationship between test anxiety and academic performance. Moreover, the retrieved literature did not provide effective solutions for managing test anxiety. This gap highlights the need for practical solutions informed by Pekrun’s Control-Value Theory. Ideally, a longitudinal study measuring test anxiety and exam scores over time would be the most appropriate approach. it is also necessary to explore cultural differences to develop more effective solutions and support systems tailored to specific cultural contexts.

The impact of shame on academic performance in medical students was inconclusive. Shame is a negative emotion that has an intricate influence on learning outcomes. The inadequacy of current literature emphasises the imperative for additional research to unravel the nuanced role of shame in the academic journeys of medical students.

Overall, emotions play a crucial role in shaping students’ academic performance, and research has attempted to find solutions to improve medical students’ learning experiences; thus, it is recommended that medical schools revise their curricula and consider using simulation-based learning in their instructional designs to enhance learning and improve students’ emotions. Also, studies have suggested using academic coaching to help students achieve their goals, change their learning styles, and apply self-testing and simple rehearsal of the material. Moreover, the study recommended to improve medical students’ critical thinking and autonomy and changing teaching styles to support students better.

Data availability

all included articles are mentioned in the manuscript, The quality assessment of included articles are located in the supplementary materials file no. 1.

Weurlander M, Lonn A, Seeberger A, Hult H, Thornberg R, Wernerson A. Emotional challenges of medical students generate feelings of uncertainty. Med Educ. 2019;53(10):1037–48.

Article   Google Scholar  

Boekaerts M, Pekrun R. Emotions and emotion regulation in academic settings. Handbook of educational psychology: Routledge; 2015. pp. 90–104.

Google Scholar  

Camacho-Morles J, Slemp GR, Pekrun R, Loderer K, Hou H, Oades LG. Activity achievement emotions and academic performance: a meta-analysis. Educational Psychol Rev. 2021;33(3):1051–95.

Aboalshamat K, Hou X-Y, Strodl E. Psychological well-being status among medical and dental students in Makkah, Saudi Arabia: a cross-sectional study. Med Teach. 2015;37(Suppl 1):S75–81.

Mirghni HO, Ahmed Elnour MA. The perceived stress and approach to learning effects on academic performance among Sudanese medical students. Electron Physician. 2017;9(4):4072–6.

Baessler F, Zafar A, Schweizer S, Ciprianidis A, Sander A, Preussler S, et al. Are we preparing future doctors to deal with emotionally challenging situations? Analysis of a medical curriculum. Patient Educ Couns. 2019;102(7):1304–12.

Rowe AD, Fitness J. Understanding the role of negative emotions in Adult Learning and Achievement: a Social Functional Perspective. Behav Sci (Basel). 2018;8(2).

Pekrun R, Frenzel AC, Goetz T, Perry RP. The control-value theory of achievement emotions: An integrative approach to emotions in education. Emotion in education: Elsevier; 2007. pp. 13–36.

Zeidner M. Test anxiety: The state of the art. 1998.

Cassady JC, Johnson RE. Cognitive test anxiety and academic performance. Contemp Educ Psychol. 2002;27(2):270–95.

Tangney JP, Dearing RL. Shame and guilt: Guilford Press; 2003.

Fang J, Brown GT, Hamilton R. Changes in Chinese students’ academic emotions after examinations: pride in success, shame in failure, and self-loathing in comparison. Br J Educ Psychol. 2023;93(1):245–61.

York TT, Gibson C, Rankin S. Defining and measuring academic success. Practical Assess Res Evaluation. 2019;20(1):5.

Abdulghani HM, Irshad M, Al Zunitan MA, Al Sulihem AA, Al Dehaim MA, Al Esefir WA, et al. Prevalence of stress in junior doctors during their internship training: a cross-sectional study of three Saudi medical colleges’ hospitals. Neuropsychiatr Dis Treat. 2014;10:1879–86.

Moreira de Sousa J, Moreira CA, Telles-Correia D, Anxiety. Depression and academic performance: a Study Amongst Portuguese Medical Students Versus non-medical students. Acta Med Port. 2018;31(9):454–62.

Junaid MA, Auf AI, Shaikh K, Khan N, Abdelrahim SA. Correlation between academic performance and anxiety in Medical students of Majmaah University - KSA. JPMA J Pakistan Med Association. 2020;70(5):865–8.

MihĂIlescu AI, Diaconescu LV, Donisan T, Ciobanu AM, THE INFLUENCE OF EMOTIONAL, DISTRESS ON THE ACADEMIC PERFORMANCE IN UNDERGRADUATE MEDICAL STUDENTS. Romanian J Child Adolesc Psychiatry. 2016;4(1/2):27–40.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372.

Hahn H, Kropp P, Kirschstein T, Rücker G, Müller-Hilke B. Test anxiety in medical school is unrelated to academic performance but correlates with an effort/reward imbalance. PLoS ONE. 2017;12(2):1–13.

Pekrun R. The control-value theory of achievement emotions: assumptions, corollaries, and Implications for Educational Research and Practice. Educational Psychol Rev. 2006;18(4):315–41.

Graesser AC. Emotions are the experiential glue of learning environments in the 21st century. Learn Instruction. 2019.

Pekrun R, Perry RP. Control-value theory of achievement emotions. International handbook of emotions in education: Routledge; 2014. pp. 120 – 41.

Behrens CC, Dolmans DH, Gormley GJ, Driessen EW. Exploring undergraduate students achievement emotions during ward round simulation: a mixed-method study. BMC Med Educ. 2019;19(1):316.

Burr J, Beck-Dallaghan GL. The relationship of emotions and Burnout to Medical Students’ academic performance. Teach Learn Med. 2019;31(5):479–86.

Zalihić A, Mešukić S, Sušac B, Knezović K, Martinac M. Anxiety sensitivity as a predictor of academic success of medical students at the University of Mostar. Psychiatria Danubina. 2017;29(Suppl 4):851–4.

Del-Ben CM, Machado VF, Madisson MM, Resende TL, Valério FP, Troncon LEDA. Relationship between academic performance and affective changes during the first year at medical school. Med Teach. 2013;35(5):404–10.

Nazir MA, Izhar F, Talal A, Sohail ZB, Majeed A, Almas K. A quantitative study of test anxiety and its influencing factors among medical and dental students. J Taibah Univ Med Sci. 2021;16(2):253–9.

Green M, Angoff N, Encandela J. Test anxiety and United States Medical Licensing Examination scores. Clin Teacher. 2016;13(2):142–6.

Ben Loubir D, Serhier Z, Diouny S, Battas O, Agoub M, Bennani Othmani M. Prevalence of stress in Casablanca medical students: a cross-sectional study. Pan Afr Med J. 2014;19:149.

Kausar U, Haider SI, Mughal IA, Noor MSA, Stress levels; stress levels of final year mbbs students and its effect on their academic performance. Prof Med J. 2018;25(6):932–6.

Hautz WE, Schröder T, Dannenberg KA, März M, Hölzer H, Ahlers O, et al. Shame in Medical Education: a randomized study of the Acquisition of intimate Examination skills and its effect on subsequent performance. Teach Learn Med. 2017;29(2):196–206.

Hayat AA, Salehi A, Kojuri J. Medical student’s academic performance: the role of academic emotions and motivation. J Adv Med Educ Professionalism. 2018;6(4):168–75.

Deeks JJ, Riley RD, Higgins JP. Combining Results Using Meta-Analysis. Systematic Reviews in Health Research: Meta‐Analysis in Context. 2022:159 – 84.

Aboalshamat K, Hou X-Y, Strodl E. The impact of a self-development coaching programme on medical and dental students’ psychological health and academic performance: a randomised controlled trial. BMC Med Educ. 2015;15:134.

Jamil H, Alakkari M, Al-Mahini MS, Alsayid M, Al Jandali O. The impact of anxiety and depression on academic performance: a cross-sectional study among medical students in Syria. Avicenna J Med. 2022;12(03):111–9.

Mirawdali S, Morrissey H, Ball P. Academic anxiety and its effects on academic performance. 2018.

Al-Qaisy LM. The relation of depression and anxiety in academic achievement among group of university students. Int J Psychol Couns. 2011;3(5):96–100.

Cheng DR, Poon F, Nguyen TT, Woodman RJ, Parker JD. Stigma and perception of psychological distress and depression in Australian-trained medical students: results from an inter-state medical school survey. Psychiatry Res. 2013;209(3):684–90.

Lee M, Larson R. The Korean ‘examination hell’: long hours of studying, distress, and depression. J Youth Adolesc. 2000;29(2):249–71.

Ali SK. 861 – Social phobia among medical students. Eur Psychiatry. 2013;28:1.

Bonna AS, Sarwar M, Md Nasrullah A, Bin Razzak S, Chowdhury KS, Rahman SR. Exam anxiety among medical students in Dhaka City and its Associated Factors-A cross-sectional study. Asian J Med Health. 2022;20(11):20–30.

Kim K-J. Factors associated with medical student test anxiety in objective structured clinical examinations: a preliminary study. Int J Med Educ. 2016;7:424–7.

Von der Embse N, Jester D, Roy D, Post J. Test anxiety effects, predictors, and correlates: a 30-year meta-analytic review. J Affect Disord. 2018;227:483–93.

Song J, Chang L, Zhou R. Test anxiety impairs filtering ability in visual working memory: evidence from event-related potentials. J Affect Disord. 2021;292:700–7.

Theobald M, Breitwieser J, Brod G. Test anxiety does not predict exam performance when knowledge is controlled for: strong evidence against the interference hypothesis of test anxiety. Psychol Sci. 2022;33(12):2073–83.

Lowe PA. Examination of test anxiety in samples of Australian and US Higher Education Students. High Educ Stud. 2019;9(4):33–43.

Kavanagh BE, Ziino SA, Mesagno C. A comparative investigation of test anxiety, coping strategies and perfectionism between Australian and United States students. North Am J Psychol. 2016;18(3).

Mihăilescu AI, Diaconescu LV, Ciobanu AM, Donisan T, Mihailescu C. The impact of anxiety and depression on academic performance in undergraduate medical students. Eur Psychiatry. 2016;33:S341–2.

Terrizzi JA Jr, Shook NJ. On the origin of shame: does shame emerge from an evolved disease-avoidance architecture? Front Behav Neurosci. 2020;14:19.

Epstein RM. Mindful practice. JAMA. 1999;282(9):833–9.

Hauer KE, Teherani A, Dechet A, Aagaard EM. Medical students’ perceptions of mentoring: a focus-group analysis. Med Teach. 2005;27(8):732–4.

Kalet A, Pugnaire MP, Cole-Kelly K, Janicik R, Ferrara E, Schwartz MD, et al. Teaching communication in clinical clerkships: models from the macy initiative in health communications. Acad Med. 2004;79(6):511–20.

Gregor A. Examination anxiety: live with it, control it or make it work for you? School Psychol Int. 2005;26(5):617–35.

Download references

Acknowledgements

I would like to thank Lancaster university library for helping me to search the literature and to find the appropriate databases and thanks to Lancaster university to prove access to several softwares.

No funding.

Author information

Authors and affiliations.

King Abdulaziz University, Jeddah, Saudi Arabia

Nora Alshareef

Lancaster University, Lancaster, UK

Nora Alshareef, Ian Fletcher & Sabir Giga

You can also search for this author in PubMed   Google Scholar

Contributions

NA made substantial contributions throughout the systematic review process and was actively involved in writing and revising the manuscript. NA’s responsible for the design of the study, through the acquisition, analysis, and interpretation of data, to the drafting and substantive revision of the manuscript. NA has approved the submitted version and is personally accountable for her contributions, ensuring the accuracy and integrity of the work. IF was instrumental in screening the literature, extracting data, and conducting the quality assessment of the included studies. Additionally, IF played a crucial role in revising the results and discussion sections of the manuscript, ensuring that the interpretation of data was both accurate and insightful. IF has approved the submitted version and has agreed to be personally accountable for his contributions, particularly in terms of the accuracy and integrity of the parts of the work he was directly involved in. SG contributed significantly to the selection of papers and data extraction, demonstrating critical expertise in resolving disagreements among authors. SG’s involvement was crucial in revising the entire content of the manuscript, enhancing its coherence and alignment with the study’s objectives. SG has also approved the submitted version and is personally accountable for his contributions, committed to upholding the integrity of the entire work.

Corresponding author

Correspondence to Nora Alshareef .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Consent of publication was obtained from the other authors.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Alshareef, N., Fletcher, I. & Giga, S. The role of emotions in academic performance of undergraduate medical students: a narrative review. BMC Med Educ 24 , 907 (2024). https://doi.org/10.1186/s12909-024-05894-1

Download citation

Received : 08 March 2024

Accepted : 12 August 2024

Published : 23 August 2024

DOI : https://doi.org/10.1186/s12909-024-05894-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical students
• Academic performance

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

Olecranon Stress Fracture Treated with Headless Compression Screws and Bone Marrow Aspirate Concentrate Augmentation: A Case Report and Systematic Review of the Literature

• Collin Freking, B.S., Austin Gartner, B.S., Vafa Behzadpour, M.D., William G. Messamore, M.D., Ph.D.

Copyright (c) 2024 Collin Freking, B.S., Austin Gartner, B.S., Vafa Behzadpour, M.D., William G. Messamore, M.D., Ph.D.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License .

All articles in the Kansas Journal of Medicine are licensed under the terms of the Creative Commons Attribution Non-Commercial No Derivatives License (CC-BY-NC-ND 4.0) . 

How to Cite

• Endnote/Zotero/Mendeley (RIS)

Make a Submission

Information.

• For Readers
• For Authors
• For Librarians

Vol. 17 Supplement 2: Abstracts from the 2024 KUMC Student Research Forum

"Lightning Outside Waverly, KS,"   June 2000 Photo courtesy of Michael Kenned y, MD

Online ISSN: 1948-2035

This journal is supported by the University of Kansas Libraries .

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

The effects of the addition of strontium on the biological response to calcium phosphate biomaterials: a systematic review.

1. Introduction

2. material and methods, 2.1. database search strategy, 2.2. selection and eligibility criteria, 2.3. quality assessment of the selected studies and data extraction, 3.1. database search, 3.2. quality assessment, 4. discussion, 4.1. functionalization with strontium, 4.2. strontium release, 4.3. in vitro studies, 4.4. in vivo studies, 4.5. involvement of signaling pathways, 5. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest.

• Liang, W.; Zhou, C.; Bai, J.; Zhang, H.; Jiang, B.; Wang, J.; Fu, L.; Long, H.; Huang, X.; Zhao, J.; et al. Current advancements in therapeutic approaches in orthopedic surgery: A review of recent trends. Front. Bioeng. Biotechnol. 2024 , 12 , 1328997. [ Google Scholar ] [ CrossRef ]
• Fendi, F.; Abdullah, B.; Suryani, S.; Usman, A.N.; Tahir, D. Development and application of hydroxyapatite-based scaffolds for bone tissue regeneration: A systematic literature review. Bone 2024 , 183 , 117075. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ramírez Fernández, M.P.; Gehrke, S.A.; Mazón, P.; Calvo-Guirado, J.L.; De Aza, P.N. Implant Stability of Biological Hydroxyapatites Used in Dentistry. Materials 2017 , 10 , 644. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Diez-Escudero, A.; Espanol, M.; Beats, S.; Ginebra, M.P. In vitro degradation of calcium phosphates: Effect of multiscale porosity, textural properties and composition. Acta Biomater. 2017 , 60 , 81–92. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Dornelas, J.; Dornelas, G.; Rossi, A.; Piattelli, A.; Pietro, N.D.; Romasco, T.; Mourão, C.F.; Alves, G.G. The Incorporation of Zinc into Hydroxyapatite and Its Influence on the Cellular Response to Biomaterials: A Systematic Review. J. Funct. Biomater. 2024 , 15 , 178. [ Google Scholar ] [ CrossRef ]
• De Lima, I.R.; Alves, G.G.; Soriano, C.A.; Campaneli, A.P.; Gasparoto, T.H.; Ramos, E.S.; De Sena, L.Á.; Rossi, A.M.; Granjeiro, J.M. Understanding the impact of divalent cation substitution on hydroxyapatite: An in vitro multiparametric study on biocompatibility. J. Biomed. Mater. Res. Part A 2011 , 98 , 351–358. [ Google Scholar ] [ CrossRef ]
• Kahler, B.; Chugal, N.; Lin, L. Alkaline Materials and Regenerative Endodontics: A Review. Materials 2017 , 10 , 1389. [ Google Scholar ] [ CrossRef ]
• Reginster, J.Y.; Brandi, M.L.; Cannata-Andía, J.; Cooper, C.; Cortet, B.; Feron, J.M.; Genant, H.; Palacios, S.; Ringe, J.D.; Rizzoli, R. The position of strontium ranelate in today’s management of osteoporosis. Osteoporos. Int. 2015 , 26 , 1667–1671. [ Google Scholar ] [ CrossRef ]
• Dahl, S.G.; Allain, P.; Marie, P.J.; Mauras, Y.; Boivin, G.; Ammann, P.; Tsouderos, Y.; Delmas, P.D.; Christiansen, C. Incorporation and distribution of strontium in bone. Bone 2001 , 28 , 446–453. [ Google Scholar ] [ CrossRef ]
• Schneider, K.; Schwarz, M.; Burkholder, I.; Kopp-Schneider, A.; Edler, L.; Kinsner-Ovaskainen, A.; Hartung, T.; Hoffmann, S. "ToxRTool", a new tool to assess the reliability of toxicological data. Toxicol. Lett. 2009 , 189 , 138–144. [ Google Scholar ] [ CrossRef ]
• Aina, V.; Bergandi, L.; Lusvardi, G.; Malavasi, G.; Imrie, F.E.; Gibson, I.R.; Cerrato, G.; Ghigo, D. Sr-containing hydroxyapatite: Morphologies of HA crystals and bioactivity on osteoblast cells. Mater. Sci. Eng. C 2013 , 33 , 1132–1142. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Alkhraisat, M.H.; Moseke, C.; Blanco, L.; Je, B.; Lopez-Carbacos, E.; Gbureck, U.; Hamdan Alkhraisat, M.; Moseke, C.; Blanco, L.; Barralet, J.E.; et al. Strontium modified biocements with zero order release kinetics. Biomaterials 2008 , 29 , 4691–4697. [ Google Scholar ] [ CrossRef ]
• Alkhraisat, M.H.; Mariño, F.T.; Rodríguez, C.R.; Jerez, L.B.; Cabarcos, E.L. Combined effect of strontium and pyrophosphate on the properties of brushite cements. Acta Biomater. 2008 , 4 , 664–670. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Birgani, Z.T.; Malhotra, A.; van Blitterswijk, C.A.; Habibovic, P. Human mesenchymal stromal cells response to biomimetic octacalcium phosphate containing strontium. J. Biomed. Mater. Res. Part A 2016 , 104 , 1946–1960. [ Google Scholar ] [ CrossRef ]
• Boanini, E.; Torricelli, P.; Fini, M.; Sima, F.; Serban, N.; Mihailescu, I.N.; Bigi, A. Magnesium and strontium doped octacalcium phosphate thin films by matrix assisted pulsed laser evaporation. J. Inorg. Biochem. 2012 , 107 , 65–72. [ Google Scholar ] [ CrossRef ]
• Boanini, E.; Torricelli, P.; Sima, F.; Axente, E.; Fini, M.; Mihailescu, I.N.; Bigi, A. Strontium and zoledronate hydroxyapatites graded composite coatings for bone prostheses. J. Colloid. Interface Sci. 2015 , 448 , 1–7. [ Google Scholar ] [ CrossRef ]
• Bracci, B.; Torricelli, P.; Panzavolta, S.; Boanini, E.; Giardino, R.; Bigi, A. Effect of Mg 2+ , Sr 2+ , and Mn 2+ on the chemico-physical and in vitro biological properties of calcium phosphate biomimetic coatings. J. Inorg. Biochem. 2009 , 103 , 1666–1674. [ Google Scholar ] [ CrossRef ]
• Capuccini, C.; Torricelli, P.; Sima, F.; Boanini, E.; Ristoscu, C.; Bracci, B.; Socol, G.; Fini, M.; Mihailescu, I.N.; Bigi, A. Strontium-substituted hydroxyapatite coatings synthesized by pulsed-laser deposition: In vitro osteoblast and osteoclast response. Acta Biomater. 2008 , 4 , 1885–1893. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Yw, C.; Gq, S.; Yl, D.; Xx, Y.; Xh, Z.; Cs, Z.; Cx, W. In vitro study on the influence of strontium-doped calcium polyphosphate on the angiogenesis-related behaviors of HUVECs. J. Mater. Sci. Mater. Med. 2008 , 19 , 2655–2662. [ Google Scholar ]
• Chen, S.; Wang, Y.; Ma, J. A facile way to construct Sr-doped apatite coating on the surface of 3D printed scaffolds to improve osteogenic effect. J. Biomater. Appl. 2022 , 37 , 344–354. [ Google Scholar ] [ CrossRef ]
• Chen, F.; Tian, L.; Pu, X.; Zeng, Q.; Xiao, Y.; Chen, X.; Zhang, X. Enhanced ectopic bone formation by strontium-substituted calcium phosphate ceramics through regulation of osteoclastogenesis and osteoblastogenesis. Biomater. Sci. 2022 , 10 , 5925–5937. [ Google Scholar ] [ CrossRef ]
• Chung, C.J.; Long, H.Y. Systematic strontium substitution in hydroxyapatite coatings on titanium via micro-arc treatment and their osteoblast/osteoclast responses. Acta Biomater. 2011 , 7 , 4081–4087. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gu, Z.; Xie, H.; Li, L.; Zhang, X.; Liu, F.; Yu, X. Application of strontium-doped calcium polyphosphate scaffold on angiogenesis for bone tissue engineering. J. Mater. Sci. Mater. Med. 2013 , 24 , 1251–1260. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gu, Z.; Wang, H.; Li, L.; Wang, Q.; Yu, X. Cell-mediated degradation of strontium-doped calcium polyphosphate scaffold for bone tissue engineering. Biomed. Mater. 2012 , 7 , 065007. [ Google Scholar ] [ CrossRef ]
• Gu, X.; Lin, W.; Li, D.; Guo, H.; Li, P.; Fan, Y. Degradation and biocompatibility of a series of strontium substituted hydroxyapatite coatings on magnesium alloys. RSC Adv. 2019 , 9 , 15013–15021. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Harrison, C.J.; Hatton, P.V.; Gentile, P.; Miller, C.A. Nanoscale Strontium-Substituted Hydroxyapatite Pastes and Gels for Bone Tissue Regeneration. Nanomater 2021 , 11 , 1611. [ Google Scholar ] [ CrossRef ]
• Hernández, L.; Parra, J.; Vázquez, B.; Bravo, A.L.; Collía, F.; Goñi, I.; Gurruchaga, M.; San Román, J.S. Injectable acrylic bone cements for vertebroplasty based on a radiopaque hydroxyapatite. Bioactivity and biocompatibility. J. Biomed. Mater. Res. Part B Appl. Biomater. 2009 , 88 , 103–114. [ Google Scholar ] [ CrossRef ]
• Huang, C.; Li, L.; Yu, X.; Gu, Z.; Zhang, X. The inhibitory effect of strontium-doped calcium polyphosphate particles on cytokines from macrophages and osteoblasts leading to aseptic loosening in vitro. Biomed. Mater. 2014 , 9 , 025010. [ Google Scholar ] [ CrossRef ]
• Huang, M.; Li, T.; Zhao, N.; Yao, Y.; Yang, H.; Du, C.; Wang, Y. Doping strontium in tricalcium phosphate microspheres using yeast-based biotemplate. Mater. Chem. Phys. 2014 , 147 , 540–544. [ Google Scholar ] [ CrossRef ]
• Qh, J.; Gong, X.; Xx, W.; Fm, H. Osteogenesis of rat mesenchymal stem cells and osteoblastic cells on strontium-doped nanohydroxyapatite-coated titanium surfaces. Int. J. Oral Maxillofac. Implants 2015 , 30 , 461–471. [ Google Scholar ]
• Jiang, S.; Wang, X.; Ma, Y.; Zhou, Y.; Liu, L.; Yu, F.; Fang, B.; Lin, K.; Xia, L.; Cai, M. Synergistic Effect of Micro-Nano-Hybrid Surfaces and Sr Doping on the Osteogenic and Angiogenic Capacity of Hydroxyapatite Bioceramics Scaffolds. Int. J. Nanomed. 2022 , 17 , 783–797. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kuang, G.-M.; Yau, W.P.; Lam, W.M.; Wu, J.; Chiu, K.Y.; Lu, W.W.; Pan, H. An effective approach by a chelate reaction in optimizing the setting process of strontium-incorporated calcium phosphate bone cement. J. Biomed. Mater. Res. Part B-Appl. Biomater. 2012 , 100B , 778–787. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Li, J.; Yang, L.; Guo, X.; Cui, W.; Yang, S.; Wang, J.; Qu, Y.; Shao, Z.; Xu, S. Osteogenesis effects of strontium-substituted hydroxyapatite coatings on true bone ceramic surfaces in vitro and in vivo. Biomed. Mater. 2017 , 13 , 015018. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Liang, Y.; Li, H.; Xu, J.; Li, X.; Qi, M.; Hu, M. Morphology, composition, and bioactivity of strontium-doped brushite coatings deposited on titanium implants via electrochemical deposition. Int. J. Mol. Sci. 2014 , 15 , 9952–9962. [ Google Scholar ] [ CrossRef ]
• Liu, F.; Zhang, X.; Yu, X.; Xu, Y.; Feng, T.; Ren, D. In vitro study in stimulating the secretion of angiogenic growth factors of strontium-doped calcium polyphosphate for bone tissue engineering. J. Mater. Sci. Mater. Med. 2011 , 22 , 683–692. [ Google Scholar ] [ CrossRef ]
• Lourenço, A.H.; Torres, A.L.; Vasconcelos, D.P.; Ribeiro-Machado, C.; Barbosa, J.N.; Barbosa, M.A.; Barrias, C.C.; Ribeiro, C.C. Osteogenic, anti-osteoclastogenic and immunomodulatory properties of a strontium-releasing hybrid scaffold for bone repair. Mater. Sci. Eng. C Mater. Biol. Appl. 2019 , 99 , 1289–1303. [ Google Scholar ] [ CrossRef ]
• Ma, P.; Chen, T.; Wu, X.; Hu, Y.; Huang, K.; Wang, Y.; Dai, H. Effects of bioactive strontium-substituted hydroxyapatite on osseointegration of polyethylene terephthalate artificial ligaments. J. Mater. Chem. B 2021 , 9 , 6600–6613. [ Google Scholar ] [ CrossRef ]
• Mohan, B.G.; Suresh Babu, S.; Varma, H.K.; John, A.; Bg, M.; Suresh Babu, S.; Hk, V.; John, A. In vitro evaluation of bioactive strontium-based ceramic with rabbit adipose-derived stem cells for bone tissue regeneration. J. Mater. Sci. Mater. Med. 2013 , 24 , 2831–2844. [ Google Scholar ] [ CrossRef ]
• Nguyen, T.T.; Jang, Y.S.; Lee, M.H.; Bae, T.S. Effect of strontium doping on the biocompatibility of calcium phosphate-coated titanium substrates. J. Appl. Biomater. Funct. Mater. 2019 , 17 , 2280800019826517. [ Google Scholar ] [ CrossRef ]
• Ni, G.X.; Yao, Z.P.; Huang, G.T.; Liu, W.G.; Lu, W.W. The effect of strontium incorporation in hydroxyapatite on osteoblasts in vitro. J. Mater. Sci. Mater. Med. 2011 , 22 , 961–967. [ Google Scholar ] [ CrossRef ]
• Olivier, F.; Rochet, N.; Delpeux-Ouldriane, S.; Chancolon, J.; Sarou-Kanian, V.; Fayon, F.; Bonnamy, S. Strontium incorporation into biomimetic carbonated calcium-deficient hydroxyapatite coated carbon cloth: Biocompatibility with human primary osteoblasts. Mater. Sci. Eng. C Mater. Biol. Appl. 2020 , 116 , 111192. [ Google Scholar ] [ CrossRef ]
• Pal, A.; Nasker, P.; Paul, S.; Chowdhury, A.R.; Sinha, A.; Das, M. Strontium doped hydroxyapatite from Mercenaria clam shells: Synthesis, mechanical and bioactivity study. J. Mech. Behav. Biomed. Mater. 2019 , 90 , 328–336. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ramadas, M.; Ferreira, J.M.F.; Ballamurugan, A.M. Fabrication of three dimensional bioactive Sr 2+ substituted apatite scaffolds by gel-casting technique for hard tissue regeneration. J. Tissue Eng. Regen. Med. 2021 , 15 , 577–585. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Sartoretto, S.C.; Calasans-Maia, M.D.; Atnn, A.; Resende, R.F.B.; da Costa Fernandes, C.J.; de Magalhães Padilha, P.; Rossi, A.M.; Teti, A.; Granjeiro, J.M.; Zambuzzi, W.F. The role of apoptosis associated speck-like protein containing a caspase-1 recruitment domain (ASC) in response to bone substitutes. Mater. Sci. Eng. C Mater. Biol. Appl. 2020 , 112 , 110965. [ Google Scholar ] [ CrossRef ]
• Stipniece, L.; Ramata-Stunda, A.; Vecstaudza, J.; Kreicberga, I.; Livkisa, D.; Rubina, A.; Sceglovs, A.; Salma-Ancane, K. A Comparative Study on Physicochemical Properties and In Vitro Biocompatibility of Sr-Substituted and Sr Ranelate-Loaded Hydroxyapatite Nanoparticles. ACS Appl. Bio Mater. 2023 , 6 , 5264–5281. [ Google Scholar ] [ CrossRef ]
• Sun, L.; Li, T.; Yu, S.; Mao, M.; Guo, D. A Novel Fast-Setting Strontium-Containing Hydroxyapatite Bone Cement With a Simple Binary Powder System. Front. Bioeng. Biotechnol. 2021 , 9 , 643557. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tovani, C.B.; Oliveira, T.M.; Soares, M.P.R.; Nassif, N.; Fukada, S.Y.; Ciancaglini, P.; Gloter, A.; Ramos, A.P. Strontium Calcium Phosphate Nanotubes as Bioinspired Building Blocks for Bone Regeneration. ACS Appl. Mater. Interfaces 2020 , 12 , 43422–43434. [ Google Scholar ] [ CrossRef ]
• Xie, H.; Wang, J.; Li, C.; Gu, Z.; Chen, Q.; Li, L. Application of strontium doped calcium polyphosphate bioceramic as scaffolds for bone tissue engineering. Ceram. Int. 2013 , 39 , 8945–8954. [ Google Scholar ] [ CrossRef ]
• Xie, H.; Gu, Z.; He, Y.; Xu, J.; Xu, C.; Li, L.; Ye, Q. Microenvironment construction of strontium-calcium-based biomaterials for bone tissue regeneration: The equilibrium effect of calcium to strontium. J. Mater. Chem. B 2018 , 6 , 2332–2339. [ Google Scholar ] [ CrossRef ]
• Xing, H.; Li, R.; Wei, Y.; Ying, B.; Li, D.; Qin, Y. Improved Osteogenesis of Selective-Laser-Melted Titanium Alloy by Coating Strontium-Doped Phosphate With High-Efficiency Air-Plasma Treatment. Front. Bioeng. Biotechnol. 2020 , 8 , 367. [ Google Scholar ] [ CrossRef ]
• Yuan, B.; Raucci, M.G.; Fan, Y.; Zhu, X.; Yang, X.; Zhang, X.; Santin, M.; Ambrosio, L. Injectable strontium-doped hydroxyapatite integrated with phosphoserine-tethered poly(epsilon-lysine) dendrons for osteoporotic bone defect repair. J. Mater. Chem. B 2018 , 6 , 7974–7984. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zhang, W.; Shen, Y.; Pan, H.; Lin, K.; Liu, X.; Darvell, B.W.; Lu, W.W.; Chang, J.; Deng, L.; Wang, D.; et al. Effects of strontium in modified biomaterials. Acta Biomater. 2011 , 7 , 800–808. [ Google Scholar ] [ CrossRef ]
• Zhao, Y.; Guo, D.; Hou, S.; Zhong, H.; Yan, J.; Zhang, C.; Zhou, Y. Porous Allograft Bone Scaffolds: Doping with Strontium. PLoS ONE 2013 , 8 , e69339. [ Google Scholar ] [ CrossRef ]
• Zhao, R.; Chen, S.; Zhao, W.; Yang, L.; Yuan, B.; Ioan, V.S.; Iulian, A.V.; Yang, X.; Zhu, X.; Zhang, X. A bioceramic scaffold composed of strontium-doped three-dimensional hydroxyapatite whiskers for enhanced bone regeneration in osteoporotic defects. Theranostics 2020 , 10 , 1572–1589. [ Google Scholar ] [ CrossRef ]
• Zhou, J.; Li, B.; Han, Y.; Zhao, L. The osteogenic capacity of biomimetic hierarchical micropore/nanorod-patterned Sr-HA coatings with different interrod spacings. Nanomed. Nanotechnol. Biol. Med. 2016 , 12 , 1161–1173. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Am, B.; Xia, W.; Palmquist, A.; Lindahl, C.; Emanuelsson, L.; Lausmaa, J.; Engqvist, H.; Thomsen, P.; Ballo, A.M.; Xia, W.; et al. Bone tissue reactions to biomimetic ion-substituted apatite surfaces on titanium implants. J. R. Soc. Interface 2012 , 9 , 1615–1624. [ Google Scholar ] [ CrossRef ]
• Cheng, C.; Alt, V.; Pan, L.; Thormann, U.; Schnettler, R.; Strauss, L.G.; Schumacher, M.; Gelinsky, M.; Dimitrakopoulou-Strauss, A. Preliminary evaluation of different biomaterials for defect healing in an experimental osteoporotic rat model with dynamic PET-CT (dPET-CT) using F-18-sodium fluoride (NaF). Injury 2014 , 45 , 501–505. [ Google Scholar ] [ CrossRef ]
• Elgali, I.; Turri, A.; Xia, W.; Norlindh, B.; Johansson, A.; Dahlin, C.; Thomsen, P.; Omar, O. Guided bone regeneration using resorbable membrane and different bone substitutes: Early histological and molecular events. Acta Biomater. 2016 , 29 , 409–423. [ Google Scholar ] [ CrossRef ]
• Gx, N.; Ww, L.; Tang, B.; Ah, N.; Ky, C.; Km, C.; Zy, L.; Kd, L. Effect of weight-bearing on bone-bonding behavior of strontium-containing hydroxyapatite bone cement. J. Biomed. Mater. Res. A 2007 , 83 , 570–576. [ Google Scholar ]
• Machado, C.P.G.; Sartoretto, S.C.; Alves, A.T.N.N.; Lima, I.B.C.; Rossi, A.M.; Granjeiro, J.M.; Calasans-Maia, M.D. Histomorphometric evaluation of strontium-containing nanostructured hydroxyapatite as bone substitute in sheep. Braz. Oral Res. 2016 , 30 , e45. [ Google Scholar ] [ CrossRef ]
• Yan, J.; Sun, J.-F.; Chu, P.K.; Han, Y.; Zhang, Y.-M. Bone integration capability of a series of strontium-containing hydroxyapatite coatings formed by micro-arc oxidation. J. Biomed. Mater. Res. Part A 2013 , 101 , 2465–2480. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Yang, L.; Perez-Amodio, S.; Fy, B.-d.G.; Everts, V.; van Blitterswijk, C.A.; Habibovic, P. The effects of inorganic additives to calcium phosphate on in vitro behavior of osteoblasts and osteoclasts. Biomaterials 2010 , 31 , 2976–2989. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zarins, J.; Pilmane, M.; Sidhoma, E.; Salma, I.; Locs, J. Immunohistochemical evaluation after Sr-enriched biphasic ceramic implantation in rabbits femoral neck: Comparison of seven different bone conditions. J. Mater. Sci. Mater. Med. 2018 , 29 , 119. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zhang, J.; Liu, L.; Zhao, S.; Wang, H.; Yang, G. Characterization and In Vivo Evaluation of Trace Element-Loaded Implant Surfaces in Ovariectomized Rats. Int. J. Oral Maxillofac. Implant. 2015 , 30 , 1105–1112. [ Google Scholar ] [ CrossRef ]
• Zhao, C.-R.; Wang, R.-Q.; Li, G.; Xue, X.-X.; Sun, C.-J.; Qu, X.-J.; Li, W.-B. Synthesis of indazole based diarylurea derivatives and their antiproliferative activity against tumor cell lines. Bioorganic Med. Chem. Lett. 2013 , 23 , 1989–1992. [ Google Scholar ] [ CrossRef ]
• Martín-Merino, E.; Petersen, I.; Hawley, S.; Álvarez-Gutierrez, A.; Khalid, S.; Llorente-Garcia, A.; Delmestri, A.; Javaid, M.K.; Van Staa, T.P.; Judge, A.; et al. Risk of venous thromboembolism among users of different anti-osteoporosis drugs: A population-based cohort analysis including over 200,000 participants from Spain and the UK. Osteoporos. Int. 2018 , 29 , 467–478. [ Google Scholar ] [ CrossRef ]
• McWilliam, R.H.; Chang, W.; Liu, Z.; Wang, J.; Han, F.; Black, R.A.; Wu, J.; Luo, X.; Li, B.; Shu, W. Three-dimensional biofabrication of nanosecond laser micromachined nanofibre meshes for tissue engineered scaffolds. Biomater. Transl. 2023 , 4 , 104–114. [ Google Scholar ] [ CrossRef ]
• Francis, W.R.; Liu, Z.; Owens, S.E.; Wang, X.; Xue, H.; Lord, A.; Kanamarlapudi, V.; Xia, Z. Role of hypoxia inducible factor 1alpha in cobalt nanoparticle induced cytotoxicity of human THP-1 macrophages. Biomater. Transl. 2021 , 2 , 143–150. [ Google Scholar ] [ CrossRef ]
• Akiko Sakai, A.V.M.O.K.I.; Shigeki, M. Preparation of Sr-containing carbonate apatite as a bone substitute and its properties. Dent. Mater. J. 2012 , 31 , 197–205. [ Google Scholar ] [ CrossRef ]
• Wan, B.; Wang, R.; Sun, Y.; Cao, J.; Wang, H.; Guo, J.; Chen, D. Building Osteogenic Microenvironments With Strontium-Substituted Calcium Phosphate Ceramics. Front. Bioeng. Biotechnol. 2020 , 8 , 591467. [ Google Scholar ] [ CrossRef ]
• Golub, E.E.; Boesze-Battaglia, K. The role of alkaline phosphatase in mineralization. Curr. Opin. Orthop. 2007 , 18 , 444–448. [ Google Scholar ] [ CrossRef ]
• Marie, P.J.; Felsenberg, D.; Brandi, M.L. How strontium ranelate, via opposite effects on bone resorption and formation, prevents osteoporosis. Osteoporos. Int. 2011 , 22 , 1659–1667. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zambuzzi, W.F.; Coelho, P.G.; Alves, G.G.; Granjeiro, J.M. Intracellular signal transduction as a factor in the development of "smart" biomaterials for bone tissue engineering. Biotechnol. Bioeng. 2011 , 108 , 1246–1250. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kozhemyakina, E.; Lassar, A.B.; Zelzer, E. A pathway to bone: Signaling molecules and transcription factors involved in chondrocyte development and maturation. Development 2015 , 142 , 817–831. [ Google Scholar ] [ CrossRef ]
• Sartoretto, S.; Gemini-Piperni, S.; da Silva, R.A.; Calasans, M.D.; Rucci, N.; dos Santos, T.M.; Lima, I.B.C.; Rossi, A.M.; Alves, G.; Granjeiro, J.M.; et al. Apoptosis-associated speck-like protein containing a caspase-1 recruitment domain (ASC) contributes to osteoblast differentiation and osteogenesis. J. Cell. Physiol. 2019 , 234 , 4140–4153. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Brennan, L.M.; Widder, M.W.; Lee, L.E.J.; van der Schalie, W.H. Long-term storage and impedance-based water toxicity testing capabilities of fluidic biochips seeded with RTgill-W1 cells. Toxicol. Vitr. Int. J. Publ. Assoc. BIBRA 2012 , 26 , 736–745. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Alves Côrtes, J.; Dornelas, J.; Duarte, F.; Messora, M.R.; Mourão, C.F.; Alves, G. The Effects of the Addition of Strontium on the Biological Response to Calcium Phosphate Biomaterials: A Systematic Review. Appl. Sci. 2024 , 14 , 7566. https://doi.org/10.3390/app14177566

Alves Côrtes J, Dornelas J, Duarte F, Messora MR, Mourão CF, Alves G. The Effects of the Addition of Strontium on the Biological Response to Calcium Phosphate Biomaterials: A Systematic Review. Applied Sciences . 2024; 14(17):7566. https://doi.org/10.3390/app14177566

Alves Côrtes, Juliana, Jessica Dornelas, Fabiola Duarte, Michel Reis Messora, Carlos Fernando Mourão, and Gutemberg Alves. 2024. "The Effects of the Addition of Strontium on the Biological Response to Calcium Phosphate Biomaterials: A Systematic Review" Applied Sciences 14, no. 17: 7566. https://doi.org/10.3390/app14177566

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals