INACCURATE ENERGY NUMBERS REPORTED AGAIN AND AGAIN

by Allen Chen

• How much oil is recoverable from the Arctic National Wildlife Refuge?
• Does one megawatt equal the electricity use of 1,000 U.S. homes?
• How much does unreliable power cost the U.S. economy?
• How much electricity is used by office equipment?

None of these four energy-related questions have simple, unqualified answers-and in every case, energy analysts and the mass media have unwittingly contributed to the spread of inaccurate information, says a group of researchers from Lawrence Berkeley National Laboratory and the Environmental Protection Agency.

A new study to be published in the Annual Review of Energy and the Environment 2002 traces how these four questions have sometimes been answered incorrectly or misleadingly by energy analysts, and how their answers have spread to public discussions through mass-media reporting of energy issues.

The authors of the study, from Lawrence Berkeley National Laboratory’s Environmental Energy Technologies Division, the Environmental Protection Agency, and the private sector, use their work to suggest that since accurate energy statistics often become crucial to public debates about energy policy, analysts need to take special precautions to reduce the possibility of their abuse.

“This work has its origins in the recent debate over how much electricity is consumed by computers, office, and network equipment,” says Jon Koomey, leader of Berkeley Lab’s Energy End-Use Forecasting Group. “One analyst’s widely publicized estimate, published in a story in Forbes magazine was repeated over and over by other publications and quickly became accepted by the media and the public even though the scientific community reviewed this analysis and found it to be inaccurate.”

The original Forbes article claimed that in 1999, 13 percent of electricity consumption in the U.S. was consumed by Internet-related computer equipment. But independent estimates prepared by researchers at Berkeley Lab and Arthur D. Little, Inc., found that computer and office-related equipment uses roughly three percent of U.S. electricity production.

The researchers found six news stories, three magazine editorials, and five reports from major investment banks that cited the erroneous Forbes numbers without stating that there was any debate over their accuracy. An additional 20 stories cited both sides of the debate, with some of these giving equal weight to both sets of numbers. “By repeating the claims of the Forbes article, often without citing a source, the trade and popular press enshrined erroneous statistics as common knowledge,” says Koomey.

“Although we and other researchers have published corrected numbers that are generally agreed upon within the research community, the incorrect numbers continue to be cited in popular articles and news accounts,” Koomey continues. “This led us to wonder if the same has happened to other statistical facts about energy, and whether they have had a distorting effect on public discussions.”

The researchers studied each of the four questions posed at the start of this article-megawatts of electricity used by 1,000 homes; cost of unreliable power; amount of electricity used by office equipment; oil reserves-investigated how energy analysts arrived at their estimates, and tracked how those estimates were reported by the news media.

Perhaps the most contentious of the four questions addresses oil reserves in the Arctic National Wildlife Refuge (ANWR).

Oil reserves are a question of definition

The study applies a three-stage process to analyze how researchers and the news media have answered the question of how much oil is recoverable from ANWR.

Much of the controversy concerns a coastal plain in the refuge called area 1002, which is about the size of Delaware. The authors write that “what is surprising is the extent to which the media have misunderstood and poorly represented the underlying science.”

Éthe trade and popular press enshrined erroneous statistics as common knowledge.

The U.S. Geological Survey has conducted much of the research on the size of the area’s economically recoverable reserves. Their work has been a multistep process that began with estimating the amount of oil in place, based on the area’s geology, without reference to how much it might cost to remove this oil.

The second step was estimating the amount of technically recoverable oil, which takes into account the industry’s current state of technology for extracting oil, without accounting for the potential cost to accomplish this. (USGS published its most recent such analysis in 1998.)

Finally, USGS analysts overlaid the technically recoverable estimate with an economic analysis-an estimate of economically recoverable oil. This analysis takes into account the quality and market value of oil, the costs of exploration and drilling, the financial costs of extracting and transporting the oil, and the financial rate of return expected at particular oil prices.

The final step of this exhaustive analysis is a computer model of the probability of cost-effective recovery. Thousands of runs of this model provided the USGS analysts with a forecast of the mean (50 percent) chance of finding a given amount of oil at a given price by comparing the high (95 percent) and low (5 percent) chances.

The higher the forecast’s specified quantity of economically recoverable oil from the reserve, the lower the chance of cost-effective recovery. The 95 percent forecast predicts the presence of a small amount of oil, but it is virtually certain to be found and recovered. The five percent forecast points to an extremely large reserve, but it is unlikely that crews will find so much oil.

“Another issue,” says Koomey, is that “economically recoverable resource estimates can either be conditional or fully risked. Conditional estimates are appropriate for thoroughly explored regions with well-understood geology, like the onshore oil fields of the lower 48 states. They assume a 100 percent probability of finding economically valuable quantities of oil, and simply assess how much of it is there. Fully risked estimates are more appropriate to remote areas like the Arctic Refuge, where much of the detail about underground structures is still unknown.”

The most recent USGS evaluation yielded the probability curves shown in the accompanying figure.
“The amount of economically recoverable oil resources from area 1002 depends strongly on the long-term market price of oil,” write the authors. “Considering the prevailing range of prices over the past 15 years (from $15 to $25 a barrel, in 1996 dollars) yields a range of mean, fully risked, economically recoverable resources from 0 to 5.6 billion barrels.”

How does the size of the reserve fare in public discussion?

Almost all of the stakeholders in the oil-drilling debate are using different sets of numbers from the same source-the 1998 U.S. Geological Survey report-to support their arguments. Proponents of drilling can quote estimates ranging from 16 to 30 billion barrels of oil, while opponents can use the same set of figures and argue there is anywhere from zero to 3 billion barrels of oil. However, the research team has shown that the news media, rather than going back to the original source of the figures, has quoted many different sources from among both proponents and opponents of drilling, resulting in the dissemination of a wide range of estimates.

Using online search tools, the researchers located 35 different news stories printed or aired between December 2000 and September 2001 regarding the amount of oil likely to be found in the ANWR. All were written by mainstream journalists; editorials and opinion pieces were specifically excluded from consideration, as were articles appearing in advocacy or trade association publications. The newspapers and magazines include some of the largest, most widely read, and most influential publications, Web sites, and news shows in the United States-the New York Times, Washington Post, Los Angeles Times, Houston Chronicle, USA Today, Scientific American, CNN.com, and CBS News among them.

The amount of oil in the Arctic Refuge, as characterized in recent news stories

As shown in the accompanying chart, the estimates vary widely. “They are literally all over the map,” say the authors. Five of the stories included specific references to multiple types of studies, so those are plotted separately, for a total of 40 specific sets of resource estimates.

The authors write: “The most frequently cited estimate was 16 billion barrels, which appeared in 24 of the stories. . . . The average high estimate cited was 13 billion barrels and the average low estimate was 7.6 billion barrels, leaving readers to conclude that a number somewhere in the middle-about 10 billion barrels-would be roughly right.”

However, comparing the “average” 10-billion-barrel figure from media reports to the mean curve in the chart of U.S. Geological Survey estimates shown above indicates that the reports on average overstated the economically recoverable reserves, at $20 per barrel, in the 1002 area by about a factor of three-at that price, about 3 to 3.5 billion barrels.

The USGS’s mean curve indicates that the amount of recoverable oil in the 1002 area is likely to range from zero to 5.6 billion barrels at oil prices of $15 to $25 per barrel, in 1996 dollars. The authors conclude “this range was only reflected by a handful of the news stories covering the topic in the last year, and most reports unwittingly left the impression that the amount of economically recoverable oil resources fell substantially above this range.”

Reducing the misuse of figures

Only a handful of media stories distinguished their estimates of oil as being technically recoverable or economically recoverable. Although 21 stories referred to the USGS as the source of the numbers, few of the analyzed stories actually quoted anyone from the USGS; the numbers generally came from proponents or opponents of drilling, not from impartial sources.

The way people cite ranges can be misleading. “A range of estimates is not always what it seems to be-people summarize data in ranges, but are careless about using consistent definitions. Or they cite incomparable numbers, or extend the ends of the range to be ‘conservative,’ no matter what the original data say,” says Koomey. “Ranges, whether cited in the media or in analytical reports, should be checked against the original source before they are cited by analysts or the media.”

Going back to the original source is always important. Referring back to the original USGS study and explaining the derivation of the reserve estimates could have revealed information that was relevant to the public debate, but few if any journalists took that step. This omission was particularly glaring because there really is only one source for the estimates of recoverable reserves, and the various participants in the debate merely chose those numbers from the USGS studies that supported their positions.

“Journalists often assume that all debates have two equal sides,” Koomey adds. “In some areas, particularly in scientific fields, there are right and wrong answers, and by highlighting a few critics instead of presenting the balance of scientific opinion, journalists can do the public debate a disservice. News coverage that apportions differing amounts of weight to the sides of a controversy-noting, for example, whether a large majority of experts are on one side of the question or another-gives the public a more accurate picture of where the experts stand.”

Finally, it is important to understand peer review, which is the process by which scientists evaluate new research to judge whether it is based on principles and judgments that represent the current scientific consensus. Results must be peer reviewed and reproducible by others before being accepted as reliable by the scientific community.

“Journalists should be skeptical of ‘research’ results that are not peer-reviewed, such as those popularized in the Forbes article,” says Koomey. There are many advocates who announce their results to the media but do not publish in peer-reviewed journals. Usually, such announcements are funded by particular organizations and have little to do with science.

#

Allen Chen is a science writer at the Lawrence Berkeley National Laboratory.

“Sorry, wrong number,” Science Beat Online, Lawrence Berkeley National Laboratory.