bad science for good causes

I am not in favor of texting while driving. I am not telling you to text and drive. If you text and drive, you are less safe, and you make others around you less safe. This is common sense, not science. If I’m acknowledging all this, then why do I get so horribly bitterly raving upset at the recurring ‘stop distracted driving’ hashtag storms? What’s the harm in suggesting something that will make people safer? If you stop reading after this sentence, then I hope you take this away; bad science is bad for you, even worse than texting while driving.

Still reading? Great, let’s get into this rant. I’m not a statistician. I don’t claim to be a statistician. I’m a critical reader who really hates to be manipulated. You are 6x more likely to die in a car while texting. You are 23x more likely to spontaneously combust while texting. Adolf Hitler was 237.8x more likely to invade Poland while texting. These are all real statistics. How can I claim that? Because a statistic is a piece of data, and as soon as I make a claim, any claim, it becomes a piece of data, and therefore a statistic. The quality of statistics can vary, kind of a lot.

What really makes statistics useful is statistical analysis, an intriguing sub-category of the scientific method. Here’s how that works. You probably know the gist, but it helps me make my point…

You make observations. ‘Hey, there’s an auto accident!’. ‘It looks like his phone flew through the window.’ ‘It’s cold today.’

You ponder a while. ‘Did I turn off the stove?’, ‘Would banning texting make us safer?’ ‘Do birds dream about auto accidents?’ Go nuts.

You form a hypothesis based on your observations, or ‘ideas you can test’.(thanks, Dinosaur Train!). Which of the following is a good hypothesis?

A) Texting causes a lot accidents.

B) You shouldn’t text while driving.

C) Banning texting would reduce traffic accidents.

D) Blackberry users are more likely to die in auto accidents.

It’s C, C is a good hypothesis. I know you thought D looked pretty good too, but it was a trick answer, because statistically speaking, there are no more Blackberry users. I digress.

Once you’ve chosen your favorite hypothesis, you have to make testable predictions. ‘If we mount phones on the steering wheel, fewer people will crash while texting.’ ‘If we ban texting while driving, there will be fewer accidents’ ‘If we restrict driving privilege of demographic groups more likely to have accidents, there will be fewer accidents’. ‘Driving in cold weather increases likelihood of texting.’ These are all perfectly valid and testable predictions. Most cited ‘studies’ get this far. Bravo. It’s a good start.

Here’s where the trouble starts. You have to figure out how to collect data to test those predictions. This is really hard. This is really hard. This is really hard. It’s worth saying that 3 times, because this is really hard. At a minimum, a good study MUST explain 3 things.

  1. Causation- Factor 1 contributes to Effect B happening, and manipulating Factor 1 would decrease Effect B.
  2. Controls- Factor 2 didn’t make Effect B happen, (or at least also required Factor 1) and isn’t a better explanation to test. Controls generally are inadequate. That’s ok, we can adjust for that later.
  3. Variables- In our test, all the subjects got the same amount of Factor 2, 3, 4 and 5. This is where we make up for Controls generally sucking.

Ok, now you gather your data in a way that makes all that earlier work make sense. In addition, you document bias, disclose funding, and make it easy for other people to read your results, so they can gather the same data to replicate your process and confirm your results.

Finally, you analyze your data, and publish whether your theory was supported by it. Notice I said ‘supported’, not ‘proved’. Anyone who says data proves their theory is a dirty stinking lier. Gravity is a law. Evolution is a theory. I don’t care how good your texting while driving experiment is, you will never move it beyond Evolution’s goddamn status.

Simple, right? So, why can’t any of the hashtag storms lead to this stuff? I know it won’t fit in a tweet, but surely it’s there in the news story that inspired the storm. No, well, then it’s study they vaguely mention but don’t actually provide link to read the complete text.  Hmm, maybe it’s on the website of the organization that is ‘concerned’ about this issue. If you’re really lucky, maybe. Probably not. If it’s any of the distracted driver drivel, it’s not there. I’ve looked. You should, too. Go ahead. *not waiting, though.

Let’s look at my favorite study on texting while driving. ‘You are 23x more likely to have an accident while texting’. This comes from a 2009 study by the Virginia Tech Transportation Institute, which was funded with $300,000 from the Federal Motor Carrier Safety Administration.

Let’s tick off some problems here.

  1. Bias- Is it possible VTTI or FMCSA benefits from a positive finding? Let’s see, funding would be a benefit. Expanding regulation would increase funding for FMCSA, which would have more $ available to pay for studies from VTTI. You need to dig really really deep to find this, but it’s there
    “Bottom Line: Using a hand held device while driving is a serious traffic violation that could result in a driver disqualification. – See more at:” Their bottom line is regulation, not safety.
  2. Data availability- Government funded study, surely you can read the entire thing on their website? Nope. Ok, public research institute, they provide complete study? Nope.
  3. Controls- These were studies on a homogenous group that reflects the general public, right? Nope. Small group of truck drivers recorded in the wild, with analysis of accidents. “Similar to everyone else.” Right.
  4. Variables- All of these accidents happened in similar locations, with same amount of rest, same time of day, controlled for emotional stress, age and experience of driver? Nah, they scanned through a bunch of road footage for accidents and noted if they were texting, on a phone, in 2009.

Really, I can do this all day. In my mind, I do this all day, every time one of you says I’m 23x more likely to crash while texting. Which is arguably true, if I’m one of a few dozen long haul truck drivers in 2009 texting on a candy bar phone.

Why does it matter? Why can’t I just ignore bad biased science? Don’t I want people to be safe? It matters, because when we choose our science issues by popular acclimation, it’s wrong.

There’s great science out there, and we ignore most of it because it doesn’t fit our preconception of the world. We scoff at C02 related ocean temperature variance, gun-related suicides, and nutrition-correlated heart disease. “You can track my phone usage and disable my apps, as long you leave me my fast cars, guns, slurpees, and cheeseburgers.” It matters because almost no one sees it. I see it, though, and it pisses me off. I wish it pissed you off, too.