In the July, 2007 issue Public Utilities Fortnightly, we published the article “Gas Market Forecasts: Betting on Bad Numbers.” This research addressed the question: are systematic errors built into the EIA natural gas (NG) forecasts, causing them to err repeatedly the same direction? It is widely recognized that over the past decade EIA forecasts for NG differ substantially from actual outcomes.
Note: You may also read or download the original article Betting on Bad Numbers (pdf). There is a version of this article in the August, 2007, issue of World Oil Magazine at Systematic bias in EIA oil price forecasts: Concerns and consequences.
In general, EIA has produced a steady spate of optimistic projections of the future of NG in the U.S. Over the period 2000—2006, for instance, this optimism has been relentless:
- The well-head price of NG was underestimated in 21 of 22 annual forecasts (95%)
- Consumption of NG by electric generators was underestimated in 22 of 22 forecasts
- Domestic production of NG was overestimated in 19 of 22 forecasts (86%)
In other words, of 66 annual forecasts of these crucial variables, EIA made an overly optimistic projection in 62 of them (94%).
Importantly, these errors were not merely of direction, but also of magnitude:
- In 2002, the EIA projected the cost of NG to electric generators in 2006 would be $ 3.82 per mcf. Actual cost per mcf was $7.15 (all in 2006 dollars).
- In 2003, the EIA overestimated domestic NG production in 2006 by almost 2 trillion cubic feet —more than the annual production of Oklahoma.
- In 2005, the EIA projected LNG imports would reach 1,190 bcf in 2006. Actual imports in 2006 were only 583 bcf—a miss of over 600 bcf just one year out.
These findings give considerable face validity to criticisms by some commentators (e.g. American Association of Petroleum Geologists) that EIA forecasts present a consistently “optimistic” view of NG – e.g. underestimating price and overestimating supply.
Importantly, such misses have significant consequences in the real world. EIA NG forecasts are widely used in regulatory proceedings, energy planning, scientific research, investment decisions, litigation and legislation. In such cases systematic bias can have profound socioeconomic implications not only within the United States but in other nations as well. Indeed, the National Energy Board of Canada regularly includes EIA NG forecasts in its projections and even OPEC scholars use EIA projections as a benchmark in their research. Andrew Weissman has estimated that in the last five years there is a $339 billion difference between EIA projected NG prices and what consumers actually paid for the fuel.
To shed further light upon the question of systematic bias we conducted an error decomposition analysis of EIA NG projections of key price, supply and consumption variables from 1998 – 2006. Error decomposition analysis is commonly used to evaluate economic forecasting models by identifying three components of the forecast errors or the proportions attributed to: (a) bias, (b) the model, and (c) randomness. A reliable model would display random errors with no discernable pattern of consistent under or over predictions. Thus, the proportions of forecast errors attributed to bias and model components would be minimal.
We evaluated one, two, three, and four-year ahead forecasts made by EIA from 1998 to 2006 for six key variables: (1) wellhead price, (2) price to electric generators, (3) consumption by electric generators, (4) domestic production, (5) imports from Canada and (6) LNG imports.
The Findings
- While similar findings were reported for the other variables we give wellhead prices as an example here—see Figure 1. The data show EIA consistently under predicted wellhead prices. The absolute error of the one-year ahead EIA forecast of wellhead NG price averages 16% and the four year average exceeds 45%.
- These forecast errors are not reflective of random chance but instead display a pattern of asymmetric bias, arising either from a fixed, linear bias or from systematic error associated with EIA’s National Energy Modeling System (NEMS).
- This bias is directed toward “optimism”: (a) market prices are consistently under predicted, (b) supply is routinely overestimated and (c) consumption by NG generators, by far the most important component of demand growth, is routinely underestimated.
Concerns — although our analysis is obviously limited to evaluation of previous forecasts, the findings do raise questions about current and future forecasts for NG from EIA:
- Continuing optimism is prevalent in current EIA forecasts. In the 2007 AEO, for example:
- NG prices are forecasted to decline over the next decade – despite the fact that well head prices have increased over 100% in the last five years and that the vast bulk of those increases were not projected by the EIA.
- Production of NG is forecasted to increase 13 % by 2030. Yet, the EIA has substantially overestimated production in virtually every forecast since 1998.
- Failure to recognize the problem—Unfortunately, it is clear the EIA does not recognize the systematic bias in NEMS. When questioned on this issue in Senate testimony last January, EIA’s Howard Gruenspecht stated “EIA stands behind the results” of the model. Yet, for example, EIA has underestimated the price of NG for electricity production every single year since 1997. Despite erring repeatedly in the same direction, EIA has published nothing on systematic bias and has not “backcasted” the NEMS model. In fact, as Berkeley Professor Max Auffhammer has noted, EIA’s model evaluation methodology may itself camouflage the problem.
Our analysis suggests that considerable caution should be exercised when using EIA forecasts relating to the future price, supply and consumption of NG. Similar caution should be exercised when using EIA’s NEMS to assess the broader economic impacts of energy policy initiatives, e.g. carbon cap and trade programs.
Timothy J. Considine, Ph.D. is Professor of Natural Resource Economics at Penn State University. Frank Clemente, Ph.D. is Senior Professor of Social Science and Energy Policy at Penn State University; he can be reached at [see original at ASPO-USA] To view the full-length version of this paper, check the www.aspo-usa.com website.
(Note: Commentaries do not necessarily represent ASPO-USA’s positions; they are personal statements and observations by informed commentators.)
