Thoughts on reproducibility and reliability in science

You’ve probably heard about a project that attempted the replicate 98 psychological studies and found that only a third of them were reproducible, the other two thirds produced different results – sometimes very different, more often only somewhat so.

Though psychology was the subject of this study, you shouldn’t assume the results were unique to that particular field. The are plenty of reports of similar failings and the so-called ‘Decline effect’ in other scientific disciplines.

So why is that? There are a lot of reasons. Research can be poorly designed, based on flawed assumptions, and sometimes an unlucky flukes can create false positives. It is also the sad fact that science is done by humans, and humans are complex things with a lot of motives besides the pure quest of knowledge.

I think the general public often fears that scientists are swayed by money from corporations and/or special interest groups, but my experience in academia is quite different. I’ve never heard anyone concerned they might loose a corporate grant. I have heard lots of people, more-or-less continuously, worrying that if their experiment doesn’t work out they won’t be able to get their PhD, land a job, or get tenure. There is enormous pressure to find something significant, to find an effect, and it matters not at all the political ramifications of that effect. So if you are worried about Monsanto buying off scientists to say GMO are safe to eat, don’t be. Convincing data that GMOs are somehow unsafe to eat would be of enormous significance, completely rewriting what we know about genetics, and would come with huge professional rewards. In my opinion, you should be more concerned that some new study showing that X, Y or Z makes plants grow bigger or yield more is actually the result of fervent, wishful thinking on the part of a grad student desperate for publishable data.

So what’s the solution? There has been a lot of talk in the academic community about making it possible to publish negative results and provide funding to regularly attempt to replicate previous studies. I hope these changes go into effect, as they could make an enormous improvement in the reliability of new findings.

In the mean time, you, as a concerned gardener, should take information supported by only a single, isolated study with a big grain of salt, particularly if it seems to contradict findings from other research. If you go to scholar.google.com and start searching around, make sure you read as much of the research on the topic as you can, so you can differentiate between the intriguing new research that may well be proved wrong and reliable findings that have been sustained by several independent researchers. And always remember that while the scientific process is far from perfect, it is still the best we’ve got.

Published by

Linda Chalker-Scott

Dr. Linda Chalker-Scott has a Ph.D. in Horticulture from Oregon State University and is an ISA certified arborist and an ASCA consulting arborist. She is WSU’s Extension Urban Horticulturist and an Associate Professor in the Department of Horticulture, and holds two affiliate associate professor positions at University of Washington. She conducts research in applied plant and soil sciences, publishing the results in scientific articles and university Extension fact sheets. Linda also is the award-winning author of five books: the horticultural myth-busting The Informed Gardener (2008) and The Informed Gardener Blooms Again (2010) from the University of Washington Press and Sustainable Landscapes and Gardens: Good Science – Practical Application (2009) from GFG Publishing, Inc., and How Plants Work: The Science Behind the Amazing Things Plants Do from Timber Press (2015). Her latest effort is an update of Art Kruckeberg’s Gardening with Native Plants of the Pacific Northwest from UW Press (2019). In 2018 Linda was featured in a video series – The Science of Gardening – produced by The Great Courses. She also is one of the Garden Professors – a group of academic colleagues who educate and entertain through their blog and Facebook pages. Linda’s contribution to gardeners was recognized in 2017 by the Association for Garden Communicators as the first recipient of their Cynthia Westcott Scientific Writing Award. "The Garden Professors" Facebook page - www.facebook.com/TheGardenProfessors "The Garden Professors" Facebook group - www.facebook.com/groups/GardenProfessors Books: http://www.sustainablelandscapesandgardens.com

Leave a Reply