Saturday, May 23, 2015

The problem with forecasts


Ian Narine

(BI) Feedloader User


Statistician Nate Silver

(BI) Feedloader User

If prediction is an essential part of science, then economics remains more of an art.

In his book The Signal and the Noise, American statistician Nate Silver (who was one of the few political commentators to accurately predict President Barack Obama's election victory earlier this month), says "Economic forecasts are blunt instruments at best, rarely being able to anticipate economic turning points more than a few months in advance."

British economist Paul Ormerod in Butterfly Economics makes the same point, writing: "Over a period of years, it is not possible in the current state of scientific knowledge to make short-term forecasts with any reasonable degree of accuracy. In any single year, a forecast may prove by chance to be correct, but over time large errors in individual years will inevitably be made."

The economists at the Central Bank of Trinidad and Tobago (CBTT) are no different from their peers in other countries in this regard. In its October 2012 Monetary Policy Report, the Bank predicts a one per cent growth in Gross Domestic Product for this year, and 2.5 per cent growth for 2013. This despite the fact that the institution's success rate in predicting GDP over the past decade has been less than 15 per cent. As Table 1 shows, only in 2003 did the Bank's economists make a forecast that was less than one per cent off, so they are wrong four out of five times.

A one per cent margin may not seem like a large error but, as Ormerod notes, "If there is an average error of 1.5 per cent over a period of time, errors significantly bigger than the average must be made in other years."

In the United Kingdom, for example, the Treasury's average error on one-year forecasts of GDP growth is 1.5 per cent, but the average actual growth rate is two per cent annually. "An average error which is so large relative to the actual series being forecast means that such projections have very little value," Ormerod points out.

Silver cites studies which show that economists' GDP forecasts tend to be off by a margin of plus or minus 3.2 per cent. Thus, a 2.5 per cent prediction could mean a growth rate of 5.7 per cent or decline of 0.7 per cent. Those studies average predictions over several decades, and found that economists' forecasts were wrong between one-third and half the time. Based on this small seven-year sample, the Central Bank's error margin has averaged 1.6 per cent and they have been wrong 85 per cent of the time. So their 2012 prediction could mean a 2.6 per cent growth rate or a 0.6 per cent decline and has a four-in-five chance of being inaccurate.

The Bank's margin of error is the difference between economic stability and a recession.

Financial expert Ian Narine, a broker registered with the Securities and Exchange Commission, told the Business Express: "The lack of timely data is by far the biggest reason for the CBTT missed targets since they are always working with a lag. In addition they tend to project conservatively to the upside and optimistically to the downside."

The Bank's 2013 prediction of 2.5 per cent growth is matched by a 2.2 per cent prediction from both the Economic Commission for Latin America and the Caribbean (ECLAC) and the International Monetary Fund (IMF). This, however, is no cause for additional confidence. The July 20 issue of The Economist magazine notes that "Back in October 2008, just after the investment bank Lehman Brothers collapsed, the International Monetary Fund unveiled its forecasts for growth in 2009. The IMF is the global lender to national governments and; its economic pronouncements are highly respected. So what did it predict? The US would grow 0.1 per cent in 2009, countries

in the euro zone 0.2 per

cent and the world as a whole 2.6 per cent. The actual outturns were declines of 3.5 per cent, 4.2 per cent and 2.6 per cent respectively." Ormerod writes, "The problems of forecasting are not confined to official bodies, nor to any particular theory of how the economy operates. All approaches, whether governmental or private, Keynesian or monetarist, have done equally badly."

In Trinidad and Tobago, despite three years of economic decline, the Central Bank has never stated that the economy is in recession.

Silver notes, "Fairly often, in fact, these forecasts have failed to 'predict' recessions even once they were already underway: a majority of [American] economists did not think we were in one when the three most recent recessions, in 1990, 2001, and 2007, were later determined to have begun."

In fact, the CBTT doesn't only have trouble estimating future GDP, but even the present GDP. In 2002, the Bank estimated GDP growth at 3.2 per cent, but revised this figure downward to 2.7 per cent in its 2003 Annual Economic Survey. In 2006, a 12 per cent GDP growth was revised two years later to 13.5 per cent, while the 2007 survey measured GDP at 5.5 per cent, revised in 2009 to 4.6 per cent.

Silver notes, "Most economic data series are subject to revision, a process that can go one for months and even years after the statistics are first published." This, of course, makes forecasts even more unreliable.

In 2010, the Bank's economists predicted zero growth in GDP. However, in that year's survey, they wrote: "Following a challenging year, the Trinidad and Tobago economy is anticipated to return to growth in 2011... Economic growth is projected at about two per cent, with most of the expansion remaining energy-based. This is dependent on stable growth in the global economy, a sustained recovery in the Caribbean region and a pickup in domestic business confidence." That year saw a decline in GDP of 1.4 per cent.

So what does the Bank base its most recent and still optimistic forecast on? In the October MPR, the Bank listed the following criteria: 1. firms in the energy sector have fully completed their major maintenance operations; 2. the industrial relations climate remains settled; and 3. public investment projects are implemented on time and efficiently.

So the CBTT is not using economic data for their forecasts but this is, in fact, not necessarily wrong. American economists have access to 45,000 economic indicators, but this wealth of data has made their predictions less reliable (because with so many data points, you would inevitably find correlations which are mistaken for causation). "If you just look at the economy as a series of variables and equations without any underlying structure, you are almost certain to mistake noise for a signal," Silver writes. A more fruitful approach, he suggests, is telling a story about the economy i.e. a data-driven narrative that links cause and effect. This is what the CBTT has done with its three premises; but is their interpretation of their story sensible? The problem with story-telling is that bias affects your conclusion.

"For reference I would suggest that two per cent is a standard and default prediction and one per cent is a polite way of saying 'We risk a recession but we hope it does not happen'," Narine says. "This is done so that they don't cause exuberance on the way up or further accentuate a downward cycle by an overly negative prediction."

However, in respect to (3), past experience suggests that public sector projects will experience significant delay. As for (2), negotiations appear to be proceeding satisfactorily, with major trade unions accepting Government's nine per cent wage offer. However, the next round of wage demands are due next year.

And, as for assumption (1), figures from the Ministry of Energy show that crude oil production had a 10.7 per cent decline in oil in the third quarter of this year as compared to 2011, while natural gas liquids production was also down by 23 per cent in the third quarter of 2012 compared to the third-quarter average in 2011.

This means that even achieving the CBTT's one per cent figure for 2012 would require GDP to grow by over four per cent in the last quarter alone. That is quite improbable but, as Silver writes, "Economic forecasters get more feedback than people in most other professions, but they haven't chosen to correct their bias toward overconfidence."

And Narine concludes: "All in all it means these predictions are not a useful guide to anything which impacts on the quality of the decision making in the economy."