Recognizing bad science by honing your B(ad) S(science) detector

Last week there was much ballyhooing over a new article on the benefits of native plants in supporting insect populations. I’ve posted on the fallacy of native plant superiority before, pointing out that landscape biodiversity not plant provenance, is most important for supporting all types of beneficial wildlife. Despite robust, published evidence to the contrary, more people and governing bodies believe that native plants are the magic bullet for urban landscapes. (Never mind the fact that there are no plants native to urban environments.)

Using CRAP analysis to assess information:

  • C = credibility. Are the authors experts in the field of interest?
  • R = relevance. Is the information applicable to the field of interest (in this case, management of plants in urban landscapes)?
  • A = accuracy. Is the information grounded in current, relevant science?
  • P = purpose. What is the underlying reason that the information is being shared?

This most recent paper warrants a careful dissection as it has gone viral online. For me, the first red flag is that there are no plant or soil scientists on this team. The first two authors, who were responsible for developing the main ideas and designing methodologies, are both ecologists by training. The other authors are involved in insect collection and identification as well as ecological modeling. Not having plant and soil scientists on the team to ensure science-based practices are followed during landscape modification is a serious oversight.

The pupose of this photo montage is apparently to show how healthy the site is after “greening.” A much better indicator would be street-level comparisons. Which you can see later in this post.

The methods section regarding the study site is astonishingly vague, given this is essentially a landscape plant installation and management project (i.e., applied plant science, not ecology). A well-designed experimental project would include control plots, replication of treatments, site analyses (including soil type and texture as well as soil testing for nutrients and organic matter), and detailed explanations of how the site was prepared, how plants were selected, prepared, and installed, and what site management occurred post-installation.

Here is the section on how this “experiment” was designed:

“In mid- April 2016, 80% of the site was substantially transformed through weeding, the addition of new topsoil, soil decompaction and fertilisation, organic mulching and the addition of 12 indigenous plant species…Selected plant species met the criteria of being locally indigenous to the City of Melbourne and represented a range of growth forms— including graminoids, lilioids, forbs and trees— requiring no ongoing management such as watering and fertilisation.”

The purpose of the methods section is to provide detailed explanations on how the study was conducted so that it could be replicated by other scientists elsewhere in the world. There is no way to replicate this study properly, as the methods are vague and very possibly not based on applied plant and soil sciences:

  1. “Addition of new topsoil” As we’ve pointed out in this blog numerous times over the past 14 years, you don’t add new topsoil to landscapes.
  2. “Soil decompaction” What is this? Does it involve tllling, which would directly affect the health of the two existing trees?
  3. “Fertilization” What is the fertilizer? When was it applied in the process? At what concentration and based on what data was it applied? You need to know baseline levels of nutrients before you can rationally add any fertilizer.
  4. “Organic mulching” What material? Compost? Cardboard? Bark? How deeply was it applied?
  5. “Addition of…plant species” Were these bare-rooted or simply popped out of the pot and dropped in the new topsoil?
  6. “Requiring no ongoing management such as watering” News flash: newly installed plants REQUIRE irrigation during the establishment period regardless of their nativity. And this site now contains substantially more plants than before, meaning increased competition for water and other resources.

The problems with this nonscience-based approach to landscape plant management can be see by comparing the two spotted gum trees that were on site before this project began. Corymbia maculata is a threatened native species in Australia and the continued health of these trees should have been paramount before any site work was initiated.

Unfortunately, these sorts of projects, conducted by teams with no soil or plant scientists and published in journals that are not specific to urban plant and soil sciences, are neither well-designed nor useful. The mindset of many researchers outside the applied plant and soil sciences is that there’s no real science to preparing soil, installing plants, and maintaining the site. This current paper does not even meet the standard of being experimental: it is merely a report on what happens to insect populations when a landscape is altered. There is no basis for comparisons. Any conclusions drawn are anecdotal.

It’s bad science.

Published by

Linda Chalker-Scott

Dr. Linda Chalker-Scott has a Ph.D. in Horticulture from Oregon State University and is an ISA certified arborist and an ASCA consulting arborist. She is WSU’s Extension Urban Horticulturist and a Professor in the Department of Horticulture, and holds two affiliate associate professor positions at University of Washington. She conducts research in applied plant and soil sciences, publishing the results in scientific articles and university Extension fact sheets. Linda also is the award-winning author of five books: the horticultural myth-busting The Informed Gardener (2008) and The Informed Gardener Blooms Again (2010) from the University of Washington Press and Sustainable Landscapes and Gardens: Good Science – Practical Application (2009) from GFG Publishing, Inc., and How Plants Work: The Science Behind the Amazing Things Plants Do from Timber Press (2015). Her latest effort is an update of Art Kruckeberg’s Gardening with Native Plants of the Pacific Northwest from UW Press (2019). In 2018 Linda was featured in a video series – The Science of Gardening – produced by The Great Courses. She also is one of the Garden Professors – a group of academic colleagues who educate and entertain through their blog and Facebook pages. Linda’s contribution to gardeners was recognized in 2017 by the Association for Garden Communicators as the first recipient of their Cynthia Westcott Scientific Writing Award. "The Garden Professors" Facebook page - www.facebook.com/TheGardenProfessors "The Garden Professors" Facebook group - www.facebook.com/groups/GardenProfessors Books: http://www.sustainablelandscapesandgardens.com

10 thoughts on “Recognizing bad science by honing your B(ad) S(science) detector”

  1. I hate to say it because I usually value what you write, but this sounds like sour grapes. Putting aside what you would have liked emphasized, people are interested in how to increase ecological support, and native host plant selection has been absent from the discussion for too long. One can plant that crape myrtle in the US in the best soil with the best methods and it STILL won’t host a darn caterpillar to feed one baby bird. Let’s not continue to lose sight of the ecological services needed from our green environment.

    1. Sour grapes about what? I’m pointing out the fatal errors in this paper because people need to learn how to tell good from bad science. I suggest you read my article and fact sheet about the value of plant diversity to support wildlife.

  2. I’m an Australian and have lived in Melbourne and I think context is incredibly important to understanding some of the logic in the article that needs to be considered before criticism.

    Australia has the oldest and most depleted soils on this planet, which are shallow and nutrient poor in all barring a few volcanic or alluvial regions that are our main food production areas. Our rainfall is very low, Summer temperatures can be very high (45C or 113F) and water is scarce (we are often on water rationing when our dams become low).

    Historically we’ve used a lot of European or imported plant species that needed richer, deeper soils with greater watering requirements – but are learning to value our native plants again which have evolved to survive and thrive in our shallow, dry, nutrient scarce soils.

    We’ve also come to appreciate that our native plants will encourage our native insect populations and rebalance our ecology.

    Topsoil would have been needed and tillage required because these are commercial sites which would have had been prepared like all commercial sites, by stripping off topsoils to subsoils and using huge compactors to stabilise the site.

    I agree in principle with many of your comments – but I think in this instance, site and climate context and appreciating our natives have evolved to survive in these harsh conditions is very relevant.

    1. The methods you mention – tilling, bringing in new topsoil – are not appropriate for landscape installation and are not based on science. There is a robust body of science-based information on how to properly prepare sites for installation of trees and shrubs. That science is missing from this article.

    2. I always try to learn from experts and have taken time to search your blogs looking for examples and methods of a more scientific approach you have raised – but I couldn’t find it.

      Could I please ask for your help to either quickly summarise the approach they should have taken to prepare a topsoil stripped, compacted site for native planting – or perhaps just provide me with links to some of your blog articles where you’ve discussed it.

      I enjoy your posts and would like to understand the science-based information you’ve mentioned on how to prepare a site like this for installation of trees and shrubs.

      1. Unfortunately there is no quick and easy way for me to do this. I’ve been providing this kind of information for almost 25 years. I would suggest you check my sites:
        “The Informed Gardener” webpage: http://www.theinformedgardener.com. A collection of white papers on many topics
        “The Garden Professors” Facebook group: http://www.facebook.com/groups/GardenProfessors. This group has been around for 12 years and searching through the archives will give you all kinds of good information.
        “Gardening in Washington State” fact sheets: http://gardening.wsu.edu/. Peer-reviewed university fact sheets with useful information for gardeners anywhere in the world.

  3. Dear Linda, I read with pleasure and interest anything that appears in my inbox from The Garden Professors. I’ve put into practice many of the things I’ve learned from you. Thank you.

    In this blog post, I notice you’re not actually critiquing a scientific study or experiment. You’re critiquing a reporter’s article that was adapted from a press release published by the British Ecological Society (BES) to announce the publication of a study in one of the BES’s journals, A press release is not expected to be as scientifically rigorous and detailed as the scientific study that it’s announcing. Sloppy writing in a press release (or an adaptation of one) doesn’t suggest the study itself was sloppily done or sloppily written up.

    The study in full can be accessed through a link provided in the press release adaptation which you link to in your blog post.

Leave a Reply