Skip to main content Skip to page footer

Research Quality Compass

Methodology - Charité Compass on Research Quality

Background

Economic, social, and cultural challenges are increasingly pushing societal demands for the effective translation of scientific knowledge into societal benefits. At the same time, extensive meta-research has exposed substantial and widespread deficits in the scientific rigour, transparency, and reproducibility of research studies, which significantly undermine public trust in science. To enhance the translational potential of biomedical research, the BIH QUEST Centre for Responsible Research has developed a comprehensive implementation program aimed at improving research quality, promoting open science, including patient and stakeholder engagement, and supporting transparent and effective research data management. Complementing the implementation program, the Center is currently establishing the COMPASS monitoring and evaluation system to track its progress and effectively assess the intended outcomes.

Instrument Development

The Charité COMPASS on Research Quality presented here is a modular research assessment tool that translates the intended outcomes of QUEST Center’s implementation program into measurable research practices.

In the scope of the QUEST Center’s monitoring and evaluation activities, the Compass represents an assessment tool that translates program outcomes, in particular research practices enhancing the translational potential of biomedical research, into measurable research practices. The instrument was developed through a rigorous, theory- and practice-informed process. First, QUEST program-related research practices were mapped into clear dimensions, sub-dimensions that were translated into measurable topic-specific indicators, i.e. scales and sub-scales, whereby items populating the scales were drawn from the literature on research methodology, established reporting guidelines, or developed together with experts in the field of biomedical research. In the next step, iterative expert review ensured content validity and relevance, prompting refinements to wording and structure. Further, survey instructions and response formats were defined, followed by a usability pre-test that checked logical flow, respondent friendliness and digital compatibility. The instrument is now in the pilot phase, where empirical studies are assessing its validity and reliability. 

Utilisation

Using either the whole scale or single subscales of the Compass, offers valuable opportunities to diverse target groups of the science system to assess and reflect on high-quality and responsible research practices, namely: 

  • Student
  • Researcher
  • Research Team
  • Reviewer

Data Analysis & Presentation

Given the unidimensional nature of the items, response values are suitable for summation and averaging. However, the median is the preferred measure of central tendency, as it is more robust to potential outliers and skewed distributions. 

To facilitate interpretation of the assessments, the Compass displays results as a radar chart.

Scales & Subscales

The scales and subscales in this instrument primarily reflect research practices relevant to QUEST program activities. 

Ethics of Science 

  • Equitable authorship, Timely reporting 

Research ethics 

  • Privacy protection, Data protection, Transparent Reporting, Animal Welfare (further subdivided into Replacement, Reduction, Refinement) 

Evidence Synthesis & Evaluation in Biomedical Research 

  • Systematic Literature Search, Critical Appraisal, Meta-Analysis, Evidence Synthesis 

Study Design in Biomedical Research 

  • Research Question, Theory Framing, Research Design, Pre-Registration, Data Management Plan 

Research Validity in Biomedical Research 

  • Internal Validity, External Validity, Statistical, Construct Validity 

Research Documentation in Biomedical Research 

  • Accurate, Legible, Original, Contemporaneous, Accessible (*ALCOA)

Presentation Quality in Biomedical Research 

  • Clarity, Coherence, Accuracy, Inclusiveness

Scientific Relevance of Biomedical Research

  • Originality, Theoretical Contribution, Methodological Advancements,

Practical Relevance of Biomedical Research 

  • Applicability, Impact on Stakeholders, Policy Influence 

Patient and Stakeholder Engagement in Biomedical Research 

  • Allocation of Tasks, Interaction, Epistemology

Accessibility & Reuse in Biomedical Research

  • Open-Access Publishing, Open Methods, Open Data, Open Code, Reuse 

Research Data Management in Biomedical Research 

  • Data Description, Documentation, Storage & Backup, Legal and Ethics, Preservation & Sharing, Responsibilities

Literature

Al-Shahi Salman, Rustam, Elaine Beller, Jonathan Kagan, Elina Hemminki, Robert S. Phillips, Julian Savulescu, Malcolm Macleod, and others, ‘Increasing Value and Reducing Waste in Biomedical Research Regulation and Management’, The Lancet, 383.9912 (2014), 176–85 <https://doi.org/10.1016/S0140-6736(13)62297-7>

Al-Shahi Salman, Rustam, Elaine Beller, Jonathan Kagan, Elina Hemminki, Robert S. Phillips, Julian Savulescu, Malcolm R. Macleod, and others, ‘Reducing Waste from Incomplete or Unusable Reports of Biomedical Research’, The Lancet, 383.9912 (2014), 176–85 <https://doi.org/10.1016/S0140-6736(13)62228-X>

Barlösius, Eva, ‘Concepts of Originality in the Natural Science, Medical, and Engineering Disciplines: An Analysis of Research Proposals’, Science Technology and Human Values, 44.6 (2019), 915–37 <https://doi.org/10.1177/0162243918808370>

Chan, An Wen, Fujian Song, Andrew Vickers, Tom Jefferson, Kay Dickersin, Peter C. Gøtzsche, and others, ‘Increasing Value and Reducing Waste: Addressing Inaccessible Research’, The Lancet, 383.9913 (2014), 257–66 <https://doi.org/10.1016/S0140-6736(13)62296-5>

Cobey, Kelly D., Stefanie Haustein, Jamie Brehaut, Ulrich Dirnagl, Delwen L. Franzen, Lars G. Hemkens, and others, ‘Community Consensus on Core Open Science Practices to Monitor in Biomedicine’, PLoS Biology, 21.1 (2023), 1–17 <https://doi.org/10.1371/journal.pbio.3001949>

Correction Chalmers, Iain, Michael B. Bracken, Ben Djulbegovic, Silvio Garattini, Jonathan Grant, A. Metin Gülmezoglu, and others, ‘How to Increase Value and Reduce Waste When Research Priorities Are Set’, The Lancet, 383.9912 (2014), 156–65 <https://doi.org/10.1016/S0140-6736(13)62229-1>

Döring, Nicola, ‘Forschungs- Und Wissenschaftsethik’, in Forschungsmethoden Und Evaluation in Den Sozial- Und Humanwissenschaften (Berlin, Heidelberg: Springer Berlin Heidelberg, 2023), pp. 119–43 <https://doi.org/10.1007/978-3-662-64762-2_4>

———, ‘Qualitätskriterien in Der Empirischen Sozialforschung’, in Forschungsmethoden Und Evaluation in Den Sozial- Und Humanwissenschaften (Berlin, Heidelberg: Springer Berlin Heidelberg, 2023), pp. 79–118 <https://doi.org/10.1007/978-3-662-64762-2_3>

———, ‘Untersuchungsdesign’, in Forschungsmethoden Und Evaluation in Den Sozial- Und Humanwissenschaften (Berlin, Heidelberg: Springer Berlin Heidelberg, 2023), pp. 183–221 <https://doi.org/10.1007/978-3-662-64762-2_7>

Farin-Glattacker, Erik, Silke Kirschning, Thorsten Meyer, and Rolf Buschmann-Steinhage, ‘Partizipation an Der Forschung – Eine Matrix Zur Orientierung’, Ausschuss „Reha-Forschung“ Der Deutschen Vereinigung Für Rehabilitation (DVfR) Und Der Deutschen Gesellschaft Für Rehabilitationswissenschaften (DGRW), 2014, 1–24

Go-fair, ‘FAIR Principles’ <https://www.go-fair.org/fair-principles/>

Hug, Sven E., and Mirjam Aeschbach, ‘Criteria for Assessing Grant Applications: A Systematic Review’, Palgrave Communications, 6.1 (2020), 1–15 <https://doi.org/10.1057/s41599-020-0412-9>

Hug, Sven E., Michael Ochsner, and Hans Dieter Daniel, ‘Criteria for Assessing Research Quality in the Humanities: A Delphi Study among Scholars of English Literature, German Literature and Art History’, Research Evaluation, 22.5 (2013), 369–83 <https://doi.org/10.1093/reseval/rvt008>

Ioannidis, John P.A., Sander Greenland, Mark A. Hlatky, Muin J. Khoury, Malcolm R. Macleod, David Moher, and others, ‘Increasing Value and Reducing Waste in Research Design, Conduct, and Analysis’, The Lancet, 383.9912 (2014), 166–75 <https://doi.org/10.1016/S0140-6736(13)62227-8>

Macleod, Malcolm R, Susan Michie, Ian Roberts, Ulrich Dirnagl, Iain Chalmers, John P A Ioannidis, and others, ‘Biomedical Research: Increasing Value, Reducing Waste’, The Lancet, 383.9912 (2014), 101–4 <https://doi.org/10.1016/S0140-6736(13)62329-6>

McDowall, R. D., ‘Is Traceability the Glue for ALCOA, ALCOA+, or ALCOA++?’, Spectroscopy (Santa Monica), 37.4 (2022), 13–19 <https://doi.org/10.56530/spectroscopy.up8185n1>

Organisation for Economic Co-Operation and Development (OECD), ‘Addressing Societal Challenges Using Transdiciplinary Research’, Policy Papers, 2020, pp. 39–51

Page, Matthew J., Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, and others, ‘The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews’, The BMJ, 372 (2021) <https://doi.org/10.1136/bmj.n71>

Roles, Contributor, ‘2 CRediT Taxonomy 2.1 Contributor Roles’, 2022 <https://www.niso.org/publications/z39104-2022-credit>

Science Europe, ‘Guidance for Data Management Plans’ <https://scienceeurope.org/media/411km040/se-rdm-template-3-researcher-guidance-for-data-management-plans.docx>

du Sert, Nathalie Percie, Viki Hurst, Amrita Ahluwalia, Sabina Alam, Marc T. Avey, Monya Baker, and others, ‘The Arrive Guidelines 2.0: Updated Guidelines for Reporting Animal Research’, PLoS Biology, 18.7 (2020), 9–10 <https://doi.org/10.1371/journal.pbio.3000410>

United Nations (UN) General Assembly, Transforming Our World: The 2023 Agenda for Sustainable Development, 2015, pp. 1–35 <https://documents.un.org/doc/undoc/gen/n15/291/89/pdf/n1529189.pdf>

Contact

Dr.rer.nat. Christiane Wetzel, M.Sc. Science Management, Head of Monitoring & Evaluation Unit, BIH QUEST Center for Responsible Research, e-mail: quest-programmevaluation@bih-charite.de

Acknowledgements

I gratefully acknowledge the valuable contributions of our colleagues at QUEST Center for Responsible Research, especially Silke Kniffert, Natascha Drude, Evgeny Bobrov, Sarah Weschke, Alexandra Bannach-Brown, and Sofija Vojvodić, as well as our colleagues at Charité3R, especially Lisa Grohmann, in providing comprehensive expert input on responsible research practices. Their insights and expertise were instrumental in developing items describing QUEST program-relevant research practices. I also acknowledge the assistance of ChatGPT, an AI language model developed by OpenAI, which assisted in streamlining and refining the wording of items, thereby improving the clarity and coherence of the research practice scales presented here. All final decisions regarding content and structure remain the author's responsibility.

Funding

The project received funding by the Stiftung Charité through the Max Rubner Award 2022, project number StC_MRP_2022_10.