Welcome to METRICS
Welcome to METRICS! This is an amazing time to be interested and involved in the international movement to improve the quality of the scientific research literature. Issues of research reproducibility have recently hit the front pages yet again, with the publication of Brian Nosek’s paper reporting the results of the Reproducibility Project in psychology. This produced not only extensive newspaper coverage, but a lot of spinoff commentary, both in the scientific and lay press. In typical fashion, much of the media coverage missed an important nuance in the original paper, which was that the authors went out of their way to point out that they had not literally debunked 60% of the original papers. This was an extraordinarily important effort, but it is ironic that coverage of the story pointing to problems in the scientific literature can’t get that story quite right.
Another story that attracted attention recently concerned the salutary effect of resveratrol – a component of chocolate and wine - on slowing Alzheimer’s disease progression. While a few outlets emphasized the tentative and preliminary nature of this finding, the headlines and the prominence of its coverage told another story. I wonder how long it will be until a systematic review or subsequent study tells us that while wine and chocolate undoubtedly improve the quality of life with or without Alzheimer’s, neither those foods nor their components affect its course. I hope I am wrong.
A lot of conferences on the subject of improving the scientific literature have occurred or are about to. I just came from an interesting meeting at the NSF, a “Director’s conference”, aimed mainly at internal staff trying to figure out what NSF’s role can be in promoting robust and reliable science going forward. Marcia McNutt, editor of Science, gave a short opening talk that featured the profound negative public health example of Andrew Wakefield’s notorious, and now retracted, Lancet paper on the MMR vaccine. NSF staff then split into three groups to discuss NSF’s role in promoting data sharing, reproducible science, and better statistical practice, led respectively by Victoria Stodden, Brian Nosek, and this METRICS co-director. One thing that was immediately apparent are the challenges that NSF faces with the extraordinarily wide and diverse array of scientific cultures that fall under its umbrella, significantly wider than the NIH. While there are many commonalities to the solutions to be found for molecular biology, seismology, economics and astrophysics, the specific problems and the means to address them are going to be quite different in each of those domains. Viewed through the METRICS lens, this bespeaks the need for more meta-research, to identify each domain’s unique problems and means for resolution. It will be interesting to see what kind of changes arise from this meeting, although one suspects that Dr. McNutt’s appointment as incoming president of the NSF might have the largest impact.
Upcoming conferences in this area include the Reward conference on research waste to be held in Edinburgh at the end of this month, the BITSS conference in December, and of course the METRICS conference on “Improving Biomedical Research, 2015” on November 19 and 20th. The inclusion of the year in our title was an expression of both optimism that we would be mounting conferences with higher numbers in the future, and confidence that the problem will not solved by that time. We hope to see you in November!