Meta-Research Innovation Center at Stanford

METRICS International Forum

Metrics International Forum is a biweekly online webinar on meta-research topics. It occurs Thursday's 9:00 am PT time. If you would like to present at the Forum, please fill out this form and we will contact you with available time slots.

Please use the following Zoom link to join the meeting. Password (if prompted): 402084

History and house rules

The Forum started in 2020 as a subscription based forum, and since February 2022 we made it open to the public. We do however ask all participants to:

  • maintain professional  discourse
  • post questions in chat - or raise their "virtual" hand - and wait to ask the question live when prompted by the moderators
  • keep zoom on mute when not speaking

Any offensive, discriminatory or unprofessional behaviour may lead to sanctions, including being kicked out of the meeting. This is a supportive and inclusive community, and we are grateful to all our participants for keeping it this way.

Upcoming speakers

  • May 02. Julie Eggington - Reality Checking Clinical Genetics and Genomics: A review of the emerging data that demonstrates that faulty assumptions and sloppy clinical lab work pervade “Precision Medicine” in the USA
  • May 16. Iain Hrynaszkiewicz - How do biology researchers assess the credibility of research?
  • May 30. Noah Haber - Registered Revisions Meta Trial: Innovations in Journal Policy Experimentation

Previous Speakers

The views and opinions expressed in the videos are those of the speakers and do not necessarily reflect the official opinion or position of our team.

2024:

  • Anna Abalkina (Apr 18). Paper mills: challenges, current understanding and unanswered questions - Presentation
  • Antica Čulina (Apr 4). Supporting registration for higher quality and impact of ecological research: Lessons from other fields - Presentation
  • Mauricia Davidson (Mar 21). Comparison of effect estimates between preprints and peer-reviewed journal articles of COVID-19 trials - Presentation
  • Marek Kwiek (Mar 7). Gender Differences in Leaving Science Forever: A longitudinal, Cohort-Based Study of 2.2 Million Scientists - Presentation 
  • Jess Rohmann (Feb 22). Perspectives from the Peerspectives project: peer-review as a vehicle to explore the scientific publishing landscape, improve quality and strengthen methods competencies - Presentation
  • James Wrightson (Feb 8). GPT for RCT? Using A.I. to check for adherence to reporting guidelines - Presentation
  • Stephan Bruns (Jan 25). Statistical reporting errors in economics - Presentation
  • Tom Stanley (Jan 11). Meta-Research Metrics of Misleading Evidence: How 67,308 Cochrane meta-analyses finds that the unrestricted weighted least squares dominate random effects - Presentation

2023:

  • Pertti Saloheimo (Dec 14). Peer Reviewers' Willingness to Review, Their Recommendations and Quality of Reviews After the Finnish Medical Journal Switched from Single-Blind to Double-Blind Peer Review - Presentation
  • Benjamin Djulbegovic (Nov 30). Generalized Decision Curve Analysis for individualized decision-making - Presentation
  • Lucija Batinovic & Rickard Carlsson (Nov 16). Computational reproducibility checks in scientific publishing - Presentation
  • Carole Lunny (Nov 2). Introduction to a new risk of bias tool for network meta-analysis (RoB NMA tool) - Presentation
  • Florian Naudet (Oct 19). Vibration of effects resulting from network geometry in mixed-treatment comparisons: a case study in network meta-analyses of antidepressants in major depressive disorder - Presentation
  • Gustav Nilsonne (Oct 5). Clinical trials reporting at Nordic medical universities and university hospitals - Presentation
  • Michael Denly (Sep 21). External Validity for Social Inquiry - Presentation
  • Tomas Havranek (Sep 7). Spurious Precision in Meta-Analysis - Presentation
  • Mario Malički (Aug 24). How often and how significantly do manuscripts change due to peer review? - Presentation
  • Summer Break
  • Felix Holzmeister (Jun 29). Population, Design, and Analytical Heterogeneity: Empirical Evidence and Practical Implications - Presentation
  • Elizabeth Stevens (Jun 15). Advancing the Development of Collaborative Research Networks to Enhance Research Value - Presentation
  • Robert Thibault (Jun 1). Preregistration: Can we make it better and easier? (using interventional meta-research) - Presentation
  • Nihar Shah (May 18). Some Experiments in Peer Review - Presentation
  • Shirley Wang (May 4). Generating reliable insights from real-world evidence studies for decision-making: highlights from REPEAT, HARPER, and DUPLICATE - Presentation
  • David Klinowski (Apr 20). Voicing disagreement in science: Missing women - Presentation
  • Zacharias Maniadis (Apr 6). ‘Identify the Expert’: an Experimental Study in Economic Advice - Presentation
  • Misha Teplitskiy (Mar 23). The role of conferences in the diffusion of scientific ideas - Presentation
  • Sean Grant & Evan Mayo-Wilson (Mar 9). Implementation of Open Science Standards in the Evidence Ecosystem for Social and Behavioral Interventions​​​​​​ - Presentation​
  • Marek Kwiek (Feb 23). Young Male and Female Scientists: the Changing Demographics of the Global Scientific Workforce - Presentation
  • Bahar Mehmani & Kristy James (Feb 9). The Science of Peer Review: New Metadata Datasets in ICSR Lab's Peer Review Workbench - Presentation
  • Arjun (Raj) Manrai (Jan 26). The precarious wisdom of communal science - Presentation
  • Peter Gøtzsche (Jan 12). Why a critical psychiatry textbook is badly needed - Presentation

2022: 

  • Alex Walker (Dec 8). OpenSAFELY: Electronic Health Record research with built in privacy, transparency, and reproducibility - Presentation
  • Florian Naudet (Nov 10). Has medicine subverted the idea of registered reports? - Presentation
  • Christopher Jones & Tim Platts-Mills (Oct 27). Peer Reviewed Evaluation of Registered End-Points of Randomized Trials (the PRE-REPORT study. - Presentation
  • Erik van Zwet (Oct 13). The Statistical Properties of RCTs. - Presentation
  • Nicolas Coles (Sep 27). Grappling with generalizability constraints in the social sciences via big-team science - Presentation
  • Tom Stanley (Sep 15). Beyond random effects: When small-study findings are more heterogeneous - Presentation
  • Amy Nelson (Aug 27). Deep forecasting of translational impact in medical research - Presentation
  • Vinay Prasad (July 7). Reliable cheap fast and few. Randomized trials, observational studies and Covid 19. - Presentation
  • Nora Hutchinson and Deborah Zarin (June 30). The Proportion of Randomized Controlled Trials That Inform Clinical Practice - Presentation
  • Marek Kwiek (June 9). Research Productivity from a Longitudinal Perspective: Once Highly Productive, Always Highly Productive? - Presentation
  • Peter Sjögårde (June 2). The effect of the rapid growth of covid-19 publications on citation indicators. - Presentation
  • Noah Haber (May 26). Causal and Associational Language in Observational Health Research: A systematic evaluation - Presentation 
  • Daniele Fanelli (May 19). Predicting and measuring study replicability by equating knowledge with information compression - Presentation
  • Ben Djulbegovic (May 12). How do guideline panels decide - Presentation
  • Emmanuel Zavalis (May 5). A meta-epidemiological assessment of transparency indicators of infectious disease models - Presentation
  • Daniel Strech (Apr 28). Ready for translation? How inestigator brochures present supporting evidence and how they could improve. - Presentation
  • Esther Pearl (Apr 22) The Experimental Design Assistant: a tool to improve in vivo research. - Presentation
  • Ivan Stelmakh (Apr 14). Two Studies on Peer Review: Citation Bias and arXiv Bias - Presentation
  • Jackie Thompson and Robbie Clark (Apr 7). Incentivising Registered Reports through funder-journal partnerships: a qualitative feasibility study - Presentation
  • Alexandra Sarafoglou & Suzanne Hoogeveen (Mar 31). The Many-Analysts Religion Project: Key Findings and Meta-Scientific Insights - Presentation
  • Carole Lunny (Mar 24). Development of a new risk of bias tool for network meta-analysis. - Presentation.
  • Tim Errington (Mar 17). Barriers to replicating preclinical cancer biology research - challenges or opportunities? - Presentation
  • Marcus Munafo (Mar 10). From Grassroots to Global: Building a Reproducibility Network - Presentation
  • Shilaan Alzahawi (Feb 24). Lay Perceptions of Scientific Findings: Swayed by the Crowd? - Presentation
  • Tom Hardwicke (Feb 17).  How should journals handle scientific criticism? - Presentation

Before Feb 17 Forum was not open to the public - we are seeking permission to upload presentations:

  • 10-Feb Maximilian Siebert. Data-sharing and re-analysis for main studies assessed by the European Medicines Agency     
  • 03-Feb Arjun Manrai
  • 27-Jan Tom Stanley. Harnessing Excess Statistical Significance for Meta-Research and Meta-Analysis
  • 20-Jan Leonhard Held
  • 13-Jan Matthew Page
  • 06-Jan Benjamin Djulbegovic.  Avoidable and unavoidable research waste in (biomedical) research

2021

Forum was not open to the public - we are seeking permission to upload presentations.

  • 09-Dec Jason Chin. Using metaresearch to improve the legal system
  • 18-Nov Nihar Shah. Improving Peer Review via Principled and Practical Approaches
  • 11-Nov Mario Malički. How do manuscripts change and what does that tell us of peer review?
  • 28-Oct Spencer Harpe Meta-research in pharmacy: What are we doing?
  • 21-Oct Jennifer Miller. Equitable access to the benefits of clinical research
  • 14-Oct Omer Benjakob, Rona Aviram, Jonathan Sobel. Wikipedia & COVID-19: Science on the frontlines of coronavirus disinformation
  • 07-Oct Kristijan Armeni and Loek Brinkman. Open science: meta-research shows the way, open science communities walk the walk
  • 30-Sep Philipp Koellinger. Designing incentives for scientists to improve the replicability of research
  • 16-Sep Tracey Weissberger. Training early career researchers to use meta-research to improve science: A participant-guided, "learn by doing" approach
  • 09-Sep Karima Chaabna. The State of Population Health Research Performance in the Middle East and North Africa: a Meta-Research Study
  • 02-Sep Simine Vazire. Evaluating research on its own merits
  • 26-Aug  Leo Tiokhin. Structuring Incentives to Improve the Efficiency and Reliability of Science: Modeling the Effect of Incentives for Priority of Discovery
  • 02-Sep Simine Vazire Evaluating research on its own merits
  • 26-Aug Leo Tiokhin    Structuring Incentives to Improve the Efficiency and Reliability of Science: Modeling the Effect of Incentives for Priority of Discovery
  • 19-Aug  David Pina. The Marie Curie mobility programme as a case study for meta-research
  • 12-Aug  Manoj Lalu. Lifting the lid on randomized control trials published in predatory journals
  • 29-Jul  Nate Breznau.  Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty
  • 22-Jul  Daniel Dunleavy.Progressive and degenerative journals: On the growth and appraisal of knowledge in scholarly publishing
  • 15-Jul  Maya Mathur. Sensitivity analysis for p-hacking in meta-analyses
  • 24-Jun Zoltan Dienes. Registered Reports 2.0: Introducing the Peer Community in Registered Reports
  • 17-Jun Antica Culina. Unused potential of ecological research, and the role of reproducibility
  • 10-Jun Holly Witteman. Strategies for funding agencies to address inequities in funding allocation
  • 03-Jun Mohammad Hosseini. An ethical analysis of equal co-authorship practices
  • 27-May Balazs Aczel. Developing tools and practices to promote open and efficient science          
  • 20-May Isabelle Boutron. Interventional research on research: development and evaluation of interventions to improve research value
  • 13-May Jon Krosnick. The collapse of scientific standards in the world of high visibility survey research
  • 06-May  Mark M Musen.  When FAIR is foul and foul is FAIR, how can we ever measure data FAIRness?
  • 29-Apr Jeroen Baas.   When peer reviewers go rogue - Estimated prevalence of citation manipulation by reviewers based on the citation patterns of 69,000 reviewers
  • 22-Apr Yian Yin.  Coevolution of policy and science during the pandemic
  • 15-Apr Riccardo De Bin.  Modelling publication bias and p-hacking
  • 08-Apr Brett Thombs and Brooke Levis.  Use of individual participant data meta-analysis to address problematic research on mental health assessment
  • 25-Mar  Charles Bennett.  Harms to physicians, patients, and pharmaceutical manufacturers following clinician reporting of titanic Adverse Drug Reactions
  • 18-Mar  Anne-Laure Boulesteix.  A replication crisis in methodological statistical research?
  • 11-Mar Florian Naudet Busting. 4 zombie trials in a post-apocalyptic world
  • 04-Mar Jamie Kirkham. Preprint servers are rapidly disseminating crucial pandemic science – but do they safeguard against unfounded medical claims?
  • 25-Feb Sabine Hoffmann   The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines
  • 18-Feb Prabhat Jha     Serological and mortality studies of SARS-CoV-2 infection in Canada and India
  • 11-Feb Noah Haber     Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation (PEACHPIE): A systematic strength of methods review
  • 04-Feb Ben W Mol     Are the data that ‘inform’ us true?
  • 28-Jan Anna Dreber   Predicting replication outcomes

2020

Forum was not open to the public - we are seeking permission to upload presentations.

  • Stefan Schandelmaier (Dec 17). A new library of methods guidance and other plans to improve knowledge translation from methods developers and meta-researchers to primary researchers
  • Terry Klassen (Dec 10). RCTs in Child Health: Frequentist approaches failing, is the future Bayesian analyses?
  • Katrin Auspurg and Alexander Tekles (Dec 3) .Do male researchers disregard the work of female researchers? The role of gender in citation decisions
  • Despina Koletsi (Nov 19). GRADE…ing Quality of the Evidence. Where do we stand?
  • Andreas Schneck (Nov 12). Are most published research findings false? Trends in statistical power, publication bias and p-hacking as well as the false discovery rate in psychology (1975–2017)
  • Janneke van 't Hooft (Nov 5). The road towards more Useful randomised controlled trials (RCTs). Preterm birth RCTs as an example.
  • Robert J MacCoun (Oct 29). p-Hacking: A Strategic Analysis
  • Atle Fretheim (Oct 22). Randomized trial of school closures in Norway (spoiler alert: it didn’t happen)
  • Hank Greely (Oct 15). CRISPR Babies: Assessing Human Germline Genome Editing
  • Deborah Zarin (Oct 8). Lack of Harmonization of Outcome Measures in COVID trials, and other Musings
  • Peter Gøtzsche (Oct 1). Mental health survival kit and withdrawal from psychiatric drugs
  • Peter Grabitz, Maia Salholz-Hillel, Nicholas Devito (Sept 10). Rapid Results Dissemination of Registered COVID-19 Clinical Trials
  • Jelte Wicherts (Sept 3). The continuing secrecy surrounding psychological research data
  • Mario Malički (Aug 27). From amazing work to I beg to differ: analysis of bioRxiv preprints that received one public comment till September 2019
  • Nikolaos Pandis (Aug 20). Who is running the show? The role of the industry in the practice of dentistry and a brief overview of evidence quality in dental research
  • David R Grimes (Aug 13). Lies, Damned lies, and statistics - the influence of bad science on public understanding
  • Kevin Boyack (July 30). A detailed open access model of the PubMed literature
  • Sally Cripps (July 23). A Tale of > 2 Models
  • Mario Malički (July 2). How should we classify changes between manuscript versions?
  • Vinay Prasad (June 18). The lay of the land of cancer research
  • Valentin Danchev (June 11). Designing a Clinical-Data Marketplace to Accelerate Sharing and Reuse of Covid-19 Clinical Data and Beyond
  • Chirag Patel (June 4). Real-world meta-science to identify environmental disparities and non-genetic correlates of phenotypesAbstract:
  • Denes Szucs (May 28). Sample size evolution in neuroimaging research
  • Perrine Janiaud (May 21). 100 DAYS LATER - The worldwide clinical trial research agenda in the beginning of the COVID-19 pandemic: results from COVID-evidence
  • Ben W Mol (May 14). Cost-effectiveness of liberal versus strict lock-down for COVID-19: are we using the correct metrics?
  • Cathrine Axfors (May 7). Population-level COVID-19 mortality risk for non-elderly individuals
  • Stephan Bruns (Apr 30). Estimating the extent of inflated significance in economic
  • Noah Haber and Sarah Wieten (Apr 23). DAG With Omitted Objects Displayed (DAGWOOD): A method for confronting causal assumptions
  • Florian Naudet (Apr 16). Will the ICMJE clinical data sharing policy achieve his intended objectives?
  • Benjamin Djulbegovic (Apr 9). Probability of discovering new treatments
  • Ioana Cristea (Apr 2). Reliability and accessibility of findings and tools reported in highly-cited research
  • Joshua Wallach (March 26). Opportunities for sharing and evaluating research: medRxiv and YODA project

Lars Hemkens (March 19). The COVID-evidence project: brainstorming on a new initiative