3/14/2017 Kim Gudeman, CSL
Written by Kim Gudeman, CSL
Say that a team of scientists publishes very promising results indicating the development of a treatment that matched chemotherapy to a cancer patient’s genetic make-up. The findings promise to transform cancer care, except for one thing: Despite multiple efforts by other scientists, the results cannot be reproduced.
Such a scenario actually happened at Duke University, when it was found in 2010 that a rising young clinician had falsified results about a treatment already being used in clinical trials. The deception was only discovered thanks to the persistent efforts of researchers at other institutions, who had major questions about the science behind the treatment. (Read more about the case here.) The case shows that reproducibility is critical in research, says the University of Illinois’ C.K. Gunsalus, and not just because of blatant misconduct.
“Science is based on the idea that you make progress and I can take your work and build on it and keep advancing the frontiers of knowledge,” said Gunsalus, professor emerita of business and director of the National Center for Professional & Research Ethics at Illinois. “If your work isn’t such that I can reproduce it, I can’t build on it. And that costs other researchers time, effort, and money.”
Gunsalus presented on the topic of “Reproducibility of Research: Issues and Proposed Remedies,” the theme of last week’s prestigious Arthur M. Sackler Colloquia at the National Academy of Sciences in Washington, D.C. The goal of the event, which was co-organized by CSL researcher Victoria Stodden, an associate professor of information science, was to examine why research reproducibility is so difficult to achieve and how to overcome those challenges.
The issue is widespread, impacting fields ranging from the social sciences to the life sciences, with concerns about computational reproducibility mounting. In a 2015 article published in Science, authors claimed that after 270 researchers tried to reproduce the results of 100 published psychology experiments, they were only successful 39 times. A look at leading cancer studies showed the same thing: Only 10-30% of published findings were reproducible, according to reports in Nature and Nature Reviews Drug Discovery. Gunsalus attributes these failures to two primary causes: people are complicated and there’s a wide spectrum of things that can go wrong and/or aren’t recorded during a research experiment.
The assumption might be that irreproducibility is due to flawed research or even potential fraud, but things are not always that simple.
“It’s far more complicated than oh, we have a bad or sloppy person in this institution,” said Gunsalus, who is also a researcher in the Coordinated Science Lab. “In the Duke case, the University had so much invested in the project, the lead investigator didn’t exercise responsible oversight in the process, and there were many places where the system failed when concerns were raised. Failures arise at both the individual and the institutional levels. We need to do a better job of institutional stewardship in these hard situations in research environments. Lack of oversight, inadequate training, institutional pressures and counterproductive reward systems can all foster flawed researcher behaviors.”
In her talk, Gunsalus advocated a three-pronged approach in addressing reproducibility issues, which include assessing and benchmarking research environments, incorporating evidence-based practices into Responsible Conduct for Research (RCR) training, and improving academic reward systems and institutional commitment to better respond to breaches of research integrity.
Gunsalus is author of “The Young Professional’s Survival Guide,” which tackles potential ethical snares that young professionals may encounter and “The College Administrator’s Survival Guide” for academic leaders. NCPRE also recently launched a four-part Coursera specialization, “Professional IQ: Preventing and Solving Problems at Work,” which addresses ethical and leadership development for professionals in various stages of their careers.