Today I was reading an interesting gardening website with a wide variety of advice, some good and some not so good. As I analyzed the website in my mind to figure out why some stuff was good and some stuff was bad it occurred to me that the problem was that a lot of the stuff that the author was recommending was based on testimonials. And then it occurred to me that many of you out there might not know what a testimonial is and why recommendations based on testimonials shouldn’t be recommended — and then viola! I had a blog post.
A testimonial is testimony presented by one person about their experiences with something. It’s like when a friend tells you that dryer lint controls slugs in a flowerbed. Don’t laugh, this is a real suggestion on one website! This person decided that it was appropriate to put dryer lint around their garden, and when they did, slug damage appeared to be reduced. Good for them. But is it good for you? The answer is maybe.
The problem is that this person is missing the two things that we need when assessing whether a particular thing works. First, we need a a control, and second, we need replication. Let’s use the dryer lint example.
We have no idea whether, if we hadn’t put out the dryer lint, the slug population might not have dwindled anyway. To find out whether it might have we need to treat only a portion of our garden with dryer lint and then see if the treated portion has more or less damage than the untreated portion after a few weeks. The untreated part of the garden is called a control and it is necessary for a good experiment. But it isn’t the only thing necessary for a good experiment, so is replication.
It is possible that the part of the garden which was treated with the lint had less slug damage than the control portion of the garden for some reason besides the lint — for example, perhaps the area where the lint was applied happened to be further from the sprinker than the other section — and slugs like it moist. So to combat possible problems like this you need to conduct your experiment more than once. In other words, you need to replicate your trials. This might be done by doing this experiment over multiple years, by having other people in your gardening club try it too, or by dividing up sections in your garden into six or so equally sized sections so that three randomly selected sections get treated with lint and three don’t. Or, best of all, do all of these things — multiple years, multiple gardens, and multiple plots within a garden! If we did more testing like that I have a funny feeling that we’d have fewer crappy products for sale.
Testimonials are interesting, but don’t get fooled into thinking that they prove anything. They don’t. You need control and replication to demonstate that something really works.
Next topic: mixed-model ANOVA for gardeners! Are you pretty sure it works, or does it just SEEM like it works?
Next topic: mixed-model ANOVA for gardeners! Are you pretty sure it works, or does it just SEEM like it works?
I’m so glad to read this! It perfectly describes my pet peeve with internet gardeners. Lots of crappy products, and lots of fairy dust out there!
This is more or less what I try to tell the TBs (True Believers) who wish to convince me about compost tea, biodynamics, and all the other miraculous products and practices. But TBs aren’t really interested in rational discussion. Sigh.
Just in case you’d like to follow the latest in the “evidence vs. testimonial” discussion I’ve been having online, follow this link to the comments section: http://online.wsj.com/article/SB10001424052748703940904575395391273329612.html#articleTabs%3Dcomments.
HA! @ Charlie.
rational thought & scientific method can not be held up against “perceived realities” because one has evidence and the other has a good shot of “faith” thrown in, faith even with a lowercase “f” will always out-sway perception with most people. And here I thought the only use of dry-lint was putting it out during nesting season for the song birds to recycle in to their nests… but I think I will stick with Sluggo anyway.
It’s a thankless, and frustrating job, to dispel these kind of things, yet it must be done, and I applaud the GP efforts to combat it. Please don’t give up, or retreat. Enjoin some allies, if need be. Send an email to the anti-biodynamics grape grower you linked to earlier this year to combat that WSJ article. In my little corner of the world, the myth of used dryer sheets having a repellent effect on mosquitoes and annoying gnats just won’t die, and I constantly hear from fellow Church members, other Master Gardeners, and recently introduced acquaintances how effective it is from their personal, and totally anecdotal experience. I had an MD repeat it to me. Ack! I’m to the point of asking myself if there isn’t an anthropological study that can be used to explain the stubbornness of the meme, despite all the evidence to the contrary. Given the monetary incentive of wine grape growers to take of advantage of the biodynamic label, that task is even more monumental to overcome in that instance.
At any rate, keep doing what you’re doing, and know you’ve got some folks here in the trenches supporting you.
Yes, but what type of dryer lint was it? Was it from blue jeans, or sweaters? Cotton or wool? where are the important details???
You know, I didn’t catch this the first time around, but Jeff, I had no idea you were a music lover. And stringed instruments at that. (Hint: see end of first paragraph.)