Rising above the noise: Using statistics-based reporting

By Kathleen Goss

Nate Silver may be a hero in the statistics world. He has made statistics cool. During his recent keynote address at the Online News Association 2013 conference, he shared his listicle of tips for journalists when analyzing data. During the late afternoon session of the NASW workshop “Rising above the noise: using statistics-based reporting,” moderated and organized by Kathleen Raven, an expert panel of statisticians and journalists expanded this discussion of using statistics to assess the validity and impact of new scientific research findings.

Taking a cue from Silver, Hilda Bastain, blogger and editor for the National Center for Biotechnology Information at the National Institutes of Health, created her own listicle aimed at protecting science writers from data-led error: 1) Remember, data don’t speak for themselves; 2) Avoid over-precision; 3) Look at each outcome separately; 4) Don’t mistake absence of evidence for evidence of absence; 5) Don’t look at one study in isolation; 6) Be as critical of the data you want to believe as you are about what you disagree with.

Evelyn Lamb, a mathematician and writer from the University of Utah, framed her advice in terms of a few things to ask of the data and the researchers themselves. Where do the data come from? Is the thing we measured related to the thing we want to study and how? Was the experiment designed to answer the question? Were the criteria for inclusion and exclusion determined before the experiment? Ultimately, it is the responsibility of a science writer to be skeptical at all times, particularly if a study has not identified a mechanism or implies cause.

Next, John Allen Paulos, an author and mathematics professor at Temple University, warned of statistical solecisms to avoid like potholes in a road. From too much precision to selection bias to intimidation by math, he stressed the importance of being critical and skeptical of the data that you are reporting. Cautioning about the ubiquitous nature of coincidence, he asserted “the most amazing coincidence of all would be the absence of coincidence.”

Wrapping up the panel was Regina Nuzzo, freelance journalist and associate professor of statistics at Gallaudet University and “coolest stats professor ever” for her buzzfeed-like teaching of statistical concepts. According to Nuzzo, a statistician’s job is to bring the skeptical voice to a research project, including questioning statistical vs. practical significance, false positives, effect size, and “p-hacking” (data massaging to get a tolerable p value).

Clearly, this audience wasn’t intimidated by stats. They further probed the panel about the definition of p values (i.e. the magic of 0.05), significance of trends, practicality of writers routinely putting studies into the context of a large body of work, and recommendations for statistics resources for the science writing community. Perhaps we are all ready to embrace our inner Nate Silver.

Also see: Noise & Numbers web site.