Saturday, February 19, 2011

Research vs. anecdotes (from SMBC)

Yes, I know we prefer authorship to linking here.
Please: an indulgence for this one from SMBC


  1. If one more person tells me that vaccines made their kid autistic, my head will explode.

    Backstory: Someone once told me that I should "understand" this point of view because I do "qualitative" research. I had to explain the difference between actual scientific rigor, which CAN be evaluated AND achieved with qualitative research, and anecdotal evidence. It did not end well. In fact, it ended with her saying "You must hate children."

    Why yes, yes I don't like them very much, but that doesn't change the fact that vaccines don't cause autism and I wouldn't wish autism on -anyone-, child or guppy or gumdrop unicorn.

  2. I get tired of hearing that anecdotal evidence is not valid.
    It certainly can be, and it can be the starting point for designing a rigorous experiment.

    I think of it as low grade ore rather than 24K gold.
    Lots of impurities, low yield.

  3. Aaaaaaauuuuuugggggghhhhhh...POW!!!!!!!

  4. @Faris - That still means it's not valid. It is no more valuable as a starting point for experimentation than "I had an idea." That certainly might be useful in terms of stimulating experimenter creativity, but it has no value in establishing the validity of a concept in any broader sense.

  5. I guess what I'm saying is that I think anecdotal reports often supply a place to *start* looking at a phenomenon, and I think that's worth something.

    I agree that they ought not to be anywhere near the finish line.

  6. Anecdotal evidence, a hunch or a bad dream caused by eating too many hot wings can all be ways to start research. Hypotheses come from somewhere. The issue is that some people treat these sources of hypotheses as results strong enough from which to draw conclusions. They skip the middle part of the scientific method.

  7. Trust me, I deal with animal owners on a regular basis.
    I am well aware of the bizarre leapfrogging right over logic that occurs.

    Every new drug that comes on the market is responsible for death and dysfunction in their eyes.

    Oh, and I charge too much. They can get it cheaper online...

  8. OTOH, years ago, I (and others) saw a number of extremely serious local reactions to a particular drug when administered IM rather than IV. Both were on the label as approved administration options.

    We contacted the manufacturer, and were told it had nothing at all to do with the drug. Perhaps we were all incompetent and did not administer it correctly.

    Today, if you look at the package insert on that drug, it warns of just that reaction as a documented adverse effect.

    The anecdotal reports drove that change. (And yes, the cause has been determined.)

  9. @Miserable: I think there's an exemption for things that are very funny, take up less than a screen's worth of space, and provoke good discussion (maybe even just 2 of those 3). In this case, at least as far as I'm concerned, Ben's description of the various sources from which a hypothesis can arise alone would be worth the price of admission. I just wish I dared to quote CM directly in presentations to students and colleagues, some of whom have overly simplistic and/or solemn ideas of how the scholarly process works.

    @BlackDog: don't know if you're still following this thread, but, if you are, you may want to check out Seth Mnookin's book _The Panic Virus_ and/or Diane Rehm's interview with Mnookin ( The book talks about the problems with the original study suggesting an autism/vaccination link, and how press coverage amplified what were originally tentative conclusions. The interview has some memorable moments, including Rehm's explanation to a caller of why she didn't include someone who still believes there is a link on the panel (because the question is, scientifically speaking, settled, and she wanted to have a show on why it persisted so long), and a caller challenging one of the other panelists who, according to the caller, had inappropriately claimed that children with autism don't have gut problems, since the caller knows lots of parents with autism, and all of their children have gut problems (as the panelist patiently explained, what she had actually said was that the incidence of gut problems in children with autism is no greater or less than it is in the general population of children; in other words, it appears that there is no link between the two, either in terms of cause or of effect). The book and the interview/panel together are a pretty good primer on how science is supposed to work, and how a combination of anecdote, press hype, and public opinion can mess the process up.


Note: Only a member of this blog may post a comment.