Today I was reading an interesting gardening website with a wide variety of advice, some good and some not so good. As I analyzed the website in my mind to figure out why some stuff was good and some stuff was bad it occurred to me that the problem was that a lot of the stuff that the author was recommending was based on testimonials. And then it occurred to me that many of you out there might not know what a testimonial is and why recommendations based on testimonials shouldn’t be recommended — and then viola! I had a blog post.
A testimonial is testimony presented by one person about their experiences with something. It’s like when a friend tells you that dryer lint controls slugs in a flowerbed. Don’t laugh, this is a real suggestion on one website! This person decided that it was appropriate to put dryer lint around their garden, and when they did, slug damage appeared to be reduced. Good for them. But is it good for you? The answer is maybe.
The problem is that this person is missing the two things that we need when assessing whether a particular thing works. First, we need a a control, and second, we need replication. Let’s use the dryer lint example.
We have no idea whether, if we hadn’t put out the dryer lint, the slug population might not have dwindled anyway. To find out whether it might have we need to treat only a portion of our garden with dryer lint and then see if the treated portion has more or less damage than the untreated portion after a few weeks. The untreated part of the garden is called a control and it is necessary for a good experiment. But it isn’t the only thing necessary for a good experiment, so is replication.
It is possible that the part of the garden which was treated with the lint had less slug damage than the control portion of the garden for some reason besides the lint — for example, perhaps the area where the lint was applied happened to be further from the sprinker than the other section — and slugs like it moist. So to combat possible problems like this you need to conduct your experiment more than once. In other words, you need to replicate your trials. This might be done by doing this experiment over multiple years, by having other people in your gardening club try it too, or by dividing up sections in your garden into six or so equally sized sections so that three randomly selected sections get treated with lint and three don’t. Or, best of all, do all of these things — multiple years, multiple gardens, and multiple plots within a garden! If we did more testing like that I have a funny feeling that we’d have fewer crappy products for sale.
Testimonials are interesting, but don’t get fooled into thinking that they prove anything. They don’t. You need control and replication to demonstate that something really works.