SCIENCE POLICY:

A checklist can help scientists who evaluate technical medical manuscripts to recognize medical propaganda, thereby helping to prevent unwarranted hype from becoming mainstream knowledge.

Medical and other scientific research suffers from an integrity problem, and this has consequences for public health. A well-known example (to the science community) is ghostwriting, which is the "writing" of scientific articles, prewritten by a private company, for publication in technical journals.

The goal of ghostwriting is to promote a private company's product by pretending that it has been successfully evaluated by independent research. It's easy to envision this unethical marketing helping to bring a drug into unwarranted widespread use, e.g. in the case of the former pharmaceutical company Wyeth regarding hormone replacement therapy.

Other obvious forms of integrity issues are also common in the science and medical literature, which include plagiarism, a lack of sufficient and independent statistical analysis, publishing research on certain cancer disciplines at the expense of other more prevalent cancers, and improper manuscript analysis. It's important to emphasize that fraud and other misconduct is extraordinarily expensive as well as unethical.

There is another integrity problem with medical manuscripts that is less obvious but no less important. Namely, the scientific and medical literature is awash in propaganda.

This refers to misrepresenting the causes of a medical condition (e.g. illegitimately hyping attention deficit hyperactivity disorder to be caused more by biology rather than social or environmental issues), framing a small medical problem as a major issue ("erectile dysfunction" comes to mind), illegitimately hyping the efficacy of a specific medical treatment (e.g. genomic medicine), failure to present competing claims and evidence, and other unwarranted claims and wording. This becomes a big problem when the research is read by non-experts.

Experts in a specific topic are well-prepared to detect hype. However, if non-expert doctors, public officials, and the lay public are misled by exaggerated or insufficient research news, medical propaganda clearly becomes a major public health issue.

How can medical propaganda be rooted out by non-specialists before it is published in the technical medical literature? Eileen Gambrill (University of California Berkeley) and Amanda Reiman (Berkeley Patients Group) have set out to help solve this problem.

A checklist for propaganda detection.

The scientists developed a 32-question checklist, divided into seven categories, for testing whether there is propaganda in a technical medical article under review. For example, the checklist questions whether "citations are given for the view promoted."

The scientists tested the ability of 17 academics to successfully use this checklist against five different articles on randomized control trials of social anxiety disorder, each article by different authors in a different journal. Each one of these academics possessed a PhD and reviews technical manuscripts, but none of them was a specialist in this specific field.

Gambrill evaluated each manuscript herself, finding 117 instances of propaganda in the five articles (Reiman's analysis agreed with Gambrill's analysis 88% of the time). The main question was whether the 17 academics in question spotted each instance.

Enhanced, but not perfect, propaganda detection.

The academics were largely unable to detect the propaganda without the checklist (4.5% success rate). The detection rate increased to 64% with the checklist, but this imperfect detection rate makes clear that many instances still went undetected (results for different checklist sections are enumerated in the original manuscript).

Nevertheless, this checklist is a great improvement over doing nothing to help reviewers detect propaganda in the technical medical literature, and should be used in conjunction with other available propaganda detection tools. Gambrill and Reiman have many future research directions, including whether a shortened checklist (saving reviewers' time) can still be useful.

NOTE: The scientists' research was funded through Eileen Gambrill's holding of the Hutto Patterson Chair in Child and Family Studies.

Gambrill, E., & Reiman, A. (2011). A Propaganda Index for Reviewing Problem Framing in Articles and Manuscripts: An Exploratory Study PLoS ONE, 6 (5) DOI: 10.1371/journal.pone.0019516