Forecasting in Treasury

Introduction

I would like to thank CEDA for the opportunity to speak at today’s conference. I will focus on recent issues in economic forecasting and share with you the outcome of a recent review of Treasury’s economic forecasting performance and capability. The full review and Treasury’s response will be published in a few weeks.

Treasury’s macroeconomic forecasts are a key input into the policy process. They frame budget discussions and decision making, identify emerging themes or risks to the economy that may require a Government response and are a vehicle to inform the public and the media about the likely path of the economy.

But like all economic forecasts they have a wide margin of error.

Practitioners are very forgiving of forecast errors as they accept the difficulty of predicting the future.

However, the implications of forecast errors can be more significant for public-sector forecasters as they can have material effects on the outcomes of Government decisions. Therefore, Treasury undertakes regular reviews of its forecasting performance and practice.

Recent Issues in Economic Forecasting

The past decade has been a challenging one for economic forecasters. The combination of shocks to the global economy and perhaps more so to Australia has been unprecedented. In that time Australia has experienced the upswing of the commodities boom, the financial crisis of 2008 and now the decline in mining investment and commodity prices. And we have witnessed extraordinary policy responses around the world.

In these circumstances it is not surprising that forecasters have been challenged.

Few identified that the strains in financial markets in 2007 and 2008 would result in the worst global recession since the Great Depression. But there was nothing new in this. History shows that it is particularly difficult to pick turning points in the economy.

Few also quantified the lingering consequences of the financial crisis and other factors constraining economic growth after the crisis even though those consequences were qualitatively understood.

Most forecasts for growth in most economies have been persistently too high over the past few years.

In that time, a key feature of the forecasts for the global economy and for Australia has been the persistent downgrading of forecasts. The chart shows the evolution of IMF forecasts for global growth and Consensus forecasts for Australia since 2010.

The pattern of downgrades is clear. In almost every year since the crisis, forecasters have been too optimistic about the outlook for growth. Treasury’s forecasts have exhibited the same behaviour.

This persistent bias is perhaps the most unusual attribute of recent experience, at least in Treasury’s case, as past forecasting reviews have shown that Treasury’s real GDP forecasts have not exhibited persistent errors over time.

In hindsight it is possible to patch together an explanation for this bias in most forecasts that would involve: the lingering consequences of the financial crisis; the more cautious saving and investment behaviour of households and firms; and slowing productivity growth in many countries. For Australia, this would be complemented by the decline in commodity prices and mining investment and the more recent slower population growth.

But not all of these are surprises. Experience has shown that big financial crises have long-lasting effects on an economy, that balance sheet repair by the household and corporate sectors constrain growth and that changes in the participation and growth of the workforce change the potential growth rates of the economy. And we knew that commodity prices and mining investment would fall.

So if these factors drove the outcomes of the past few years why did most forecasts not capture them?

Errors in our knowledge of the economy and errors in our assumptions about exogenous factors have both been important in my view.

The lingering effect of the financial crisis on private spending has been understood but probably not well quantified. This is especially the case for business investment, where financial conditions have appeared to be very favourable for investment for some time. But this has not been reflected in stronger investment in Australia or globally.

At the same time, commodity prices have fallen more dramatically than most expected.

And important changes in the supply-side of the economy in Australia such as slower population growth have only been gradually revealed by the data.

Each of these factors would be difficult to incorporate into forecasts by themselves but it is likely that the cumulative effect of all of these factors happening at once was difficult to quantify.

What is also likely in my view, but difficult to prove, is that there is a natural bias in forecasting to assume that the past will be a reliable guide to the future. This is not a bad assumption most of the time but will bias forecasts in periods like the present when we are dealing with not one but a range of unusual forces.

Forecasting in Treasury

I recently joined Treasury and was asked by the Secretary to conduct a review of its forecasting process.

As part of the ongoing management of its capabilities Treasury regularly reviews the performance and production of its forecasts. The last review was conducted in 2012, with earlier reviews in 2005 and 2002. This review should therefore be seen as part of the regular auditing of Treasury’s capability.

But unlike earlier reviews, my focus was not on recent forecasting performance but instead was focussed on the forecasting processes in place in Treasury and whether they were structured to produce reliable forecasts.

In conducting this review I drew on the conclusions of earlier reviews of forecasting performance, interviews with staff involved in forecasts, observations of the recent MYEFO forecasting round, the approach of other public-sector organisations to forecasting as well as my observations of private-sector forecasting.

The quantitative conclusions of Treasury’s past forecasting reviews were very consistent. They found that Treasury’s forecasts were:

• unbiased. That is they did not reveal persistent errors in the one direction over a long period of time;
• comparable in accuracy to other agencies and private forecasters;
• and, like other forecasters, missed turning points in the economy.

Digging a little deeper into this quantitative work I would also observe that Treasury’s forecasts did well in normal times but were challenged when conditions were unusual.

Thus, while there was no evidence of persistent errors over a very long time period, Treasury’s forecasts were punctuated by persistent and sometimes large errors over horizons relevant for policy makers and their advisers.

This can be seen in the following chart which compares Treasury’s forecasts of nominal GDP growth with actual outcomes since the early 1990s. I use nominal rather than real GDP as it is a key input into forecasting budget outcomes.

Chart 2 — Nominal GDP Forecasts and Outcomes

Source: Budget papers and ABS cat. no. 5206.0

It shows three distinct periods of forecast errors. Forecasts regularly overestimated nominal GDP growth in the early 1990s and after the financial crisis while understating outcomes for most of the 2000s prior to the crisis.

Each of these episodes coincided with Treasury persistently erring in its inflation (measured by the GDP deflator) forecasts. This observation has been made in each of the previous forecast reviews.

As previous reviews noted Treasury missed the decline in inflation in the early 1990s.

More recently, the forecasts did not incorporate fully the impact of the unprecedented rise then fall of commodity prices. In the upswing, Treasury’s forecasts did include projections that prices would eventually fall but these proved to be wrong as prices continued to increase. In the downswing, prices have fallen more rapidly than Treasury’s assumptions implied. Heightened volatility in Australia’s terms of trade has made it more difficult to forecast nominal GDP outcomes.

While difficulty in forecasting the GDP deflator has been a feature of Treasury’s (and others’) forecasts, past reviews have shown that its real GDP forecasts were more accurate and unbiased. Therefore, it is the persistent errors in Treasury’s real GDP forecasts of the past few years that have been unusual.

The chart shows that in the 1990s and 2000s there was no distinct under or overestimate in Treasury’s forecasts for the real economy. However, since the crisis Treasury, like the other forecasters presented earlier, has for the most part overestimated real GDP growth.

Chart 3 Real GDP Forecasts and Outcomes

Source: Budget papers and ABS cat. no. 5206.0

Now I do not wish to push this point too hard as the sample set is very small but it is consistent with what we are seeing across a range of forecasters and countries.

So how does Treasury go about forecasting the economy?

Its approach is very rigorous.

It employs a range of sophisticated, single-sector models that explain and forecast various aspects of the economy. Most of the expenditure side of the economy – consumer spending, dwelling and business investment, exports and imports – as well as the labour market and prices and wages are modelled.

The forecasting round begins with the release of the quarterly national accounts. Key commodity price, exchange rate and interest rate assumptions are set and forecasts are made for Australia’s major trading partners.

Sector analysts who monitor each of the important sectors of the economy then run the models. The initial forecast runs are then adjusted to reflect judgments about the recent performance of the models and whether the models are reflecting near-term information from a range of partial economic indicators.

These initial forecasts are subject to internal peer review at sector meetings and externally at various sub-committee meetings of the Government’s Joint Economic Forecasting Group (JEFG). JEFG includes representatives of Treasury, the Reserve Bank of Australia, the Department of the Prime Minister and Cabinet, the Department of Finance and Administration and the Australian Bureau of Statistics.

This forecasting process is thus very iterative, as initial forecasts are adjusted in response to discussions and consistency checks from senior members of Treasury’s Macroeconomic Conditions Division. A final set of forecasts is constructed for review at the full JEFG meeting before it is presented to the Treasury Executive.

Anyone who has been involved in public-sector economic forecasting would recognise this process. There are only so many ways to forecast the economy. All forecasting involves the use of quantitative tools, analysis of the recent data and a varying amount of judgment and experience.

Indeed, a review of the forecasting approaches of other public-sector organisations shows that their forecasts are dependent on some combination of economy-wide models, time-series techniques, single-sector models, input from sector analysts and judgment applied during the early stages of the forecasting process and afterwards by senior policy makers. The weight given to each of these inputs varies across institutions and often over time within the same institution.

Current Review

In thinking about Treasury’s forecasting process it is useful to think about forecasting as the management of uncertainty. Good forecasting practice should aim to:

• Manage uncertainty about the outlook;
• Reduce uncertainty around our understanding of the economy today; and
• Clearly communicate the uncertainties around the forecasts.

Managing uncertainty about the outlook

Managing uncertainty around the outlook is difficult but there are things forecasters can do to help.

One main source of uncertainty around the outlook is our understanding of how the economy works. In practice this is reflected in the estimated co-efficients in models of the economy or in the judgment embedded in the forecasts.

Another main source of uncertainty is around the variables that are assumed to be exogenous to the economy. A good and topical example of the impact of this uncertainty has been the overprediction of commodity prices and global growth over the past few years.

These sources of uncertainty are always present. However, they are heightened at times when the economy is subject to unprecedented shocks or when there are significant changes in private or public-sector behaviour.

To the extent that there is a consensus around best forecasting practice it is that as wide a range of inputs as possible should be used in the process. In effect casting the net as wide as possible will help manage, but not eliminate, uncertainty around the outlook.

No one approach will provide accurate forecasts of the economy but drawing on a range of quantitative inputs, judgment and historical experience may triangulate a better outcome. Most importantly, it will minimise the risk of relying on a single approach that is wrong.

Institutions like the Bank of England and the Bank of Canada are very explicit in their desire to bring as wide a range of inputs into the forecast process.¹

In practice, such an approach typically has an economy-wide model at the centre of the process.² This provides an efficient way to combine the many linkages in the economy and assess the impact of economic developments over the forecast horizon in a consistent way.

This is then augmented by a range of tools including inputs from sector specialists, time-series and single equation models and judgment of senior policy makers.

One of the main conclusions of my Review was that Treasury’s current approach relies on fewer sources than many other public-sector forecasters. It relies predominantly on a small number of single-sector models. This is not a problem in itself. The individual models provide a sound base for forecasting. They are rigorous and fit history well. But by definition they are a narrow description of the economy.

They are not designed to efficiently incorporate all of the linkages in the economy. So, as a tool, their ability to cope with changes in economic circumstances and shocks is constrained.

When forecasting in an iterative way, with single-sector models, it is also extremely important that the model forecasts be challenged by well-informed judgment and by other quantitative inputs.

To an extent this does happen in Treasury but in my observation of one
, admittedly uncontentious, forecasting round there is scope to deepen and broaden the judgmental input into the forecasts and the variety of quantitative inputs used.

Another feature of Treasury’s forecasting process that I highlighted was that it would be biased toward achieving trend-like consensus outcomes.

For a small open economy like Australia, persistent and short-term deviations from trend are likely to be driven by global economic shocks, large changes in commodity prices or from the financial sector. Other factors could also be important but these three factors encompass much of Australia’s experience with shocks.

Treasury’s Australian forecasts are underpinned by global growth forecasts that are heavily informed by international agencies and consensus. This reflects the judgement that relying on international agencies or the relevant economy’s public and private forecasts is the best basis for global forecasts. Treasury does not have the resources to allow ‘bottoms-up’ forecasts of all the major economies. Instead, it focusses on those economies of most importance to Australia, including through intelligence from Treasury’s overseas posts. For example, Treasury recently adopted a below-consensus view on China.

Moreover, the channels of influence from the global economy into Australian forecasts are quite limited. The global economy largely influences Treasury’s Australian forecasts through growth in our major trading partners. The links to the global economy are broader and more complex than this but are, admittedly, difficult to account for in many macroeconomic models or in judgmental assessments of forecasts.

For example, Treasury’s approach does not automatically reflect observations of common global themes, such as the unusual weakness in business investment around the world, which may have relevance to thinking about Australia.

Commodity prices are now also assumed to be unchanged over the two-year forecast horizon. This is common practice as commodity prices are hard to forecast. But it does leave Treasury’s forecasts vulnerable to large or persistent changes in commodity prices. There is no easy solution as forecasting commodity prices is very difficult.

Importantly, in a now more sophisticated global economy, financial factors also play a limited role in the forecasting process, both within the empirical models and in the application of judgment. The exchange rate is largely assumed to remain unchanged over the forecast horizon, while interest rates are assumed to move broadly in line with market expectations. Once again, this is common practice.

With these assumptions in place it is difficult to produce a forecast that is significantly different from trend in the absence of a large change in the consensus on global growth, commodity prices and financial parameters between each forecast round.

I think that the consequence of this can be seen in the next chart which shows the mean absolute errors in Treasury’s GDP forecasts since the early 1990s and the errors in some naïve trend-following models. The degree of error in Treasury’s forecasts is little different from these simple trends³.

Chart 4 — Budget and trend forecast errors

Source: Budget papers and ABS cat. no. 5206.0

These issues have a number of implications for Treasury’s forecasts.

It means that Treasury’s forecasting approach is likely to generate reliable forecasts at times when economic conditions are normal but will be challenged at other times.

There is a therefore a high probability that structural changes or persistent shocks will not be adequately incorporated into the forecasts.

Now this is true of most forecasters not just Treasury.

To better manage the uncertainty around the outlook Treasury will adopt a number of changes over the course of this year that I recommended to its forecasting processes:

• Treasury will embed a system-wide model into the process. This will not be used to generate the final forecasts but to ensure that economy-wide issues are captured effectively. It will also allow for efficient analysis of alternative scenarios so that discussion of the possible risks to the forecasts can be both better understood and better explained and quantified for Government and the public.
  • This was also a recommendation from the 2012 forecasting review and Treasury has acquired a macroeconomic model that it had begun to use as a check against its sector-based forecasts.
  • My recommendation is to run the model at the start of the process to assist in gaining an understanding of the overall likely path of the economy over the forecast period.
• Engagement with senior officers from a wider range of areas of responsibility within the Treasury will occur throughout the forecasting process as another method to ensure that judgement by senior officers on economy-wide and global trends are given due consideration.
• Treasury will also engage a panel of external experts to review a range of quantitative forecasting techniques that could be added to its forecasting toolkit. This will enhance the range of other quantitative tools available, including developing time-series and other models of the variables of interest. Treasury will also conduct a rigorous annual audit of the single-sector models.
• Since the recession of the early 1990s Treasury has conducted an extensive business liaison program to obtain more timely information on the course of the economy than that available through official statistics and to embed this in its forecasts. Treasury also meets regularly with private sector economists to test its thinking around the forecasts. This will be expanded to engage more with private sector commodity analysts to further increase the sources of information on the risks around the world economy and commodity prices.
• I am also leading a set of work within Treasury to explore more deeply the impact of the financial sector of the economy and how it can be best incorporated into the forecasting process. The use of an economy-wide model will assist.

I also reflected on the resourcing within Treasury for the preparation of the forecasts. Treasury has around 30 staff allocated to the monitoring and forecasting of the domestic and global economies. There is a heavy reliance on highly-skilled but sometimes relatively inexperienced staff in generating the forecasts. My recruitment to Treasury is a part of a suite of measures being implemented to enhance the ongoing technical development and experience of the economic staff, recognising that Treasury requires both officers who have deep technical knowledge and those with a broad understanding of economic policy and implementation across the wide range of Treasury responsibilities. The Sydney office -- and the soon to be opened Melbourne office – will be a key part of this process by facilitating this high-level recruitment.

Now let’s return to the two other areas of dealing with the uncertainty of forecasting – reducing the uncertainty about the present and communicating the uncertainty about the future.

These are two areas where there has been some advance in forecasting practice over the past decade or so.

Reducing the uncertainty around the present

Forecasters are spending more time focussing on understanding the here and now of the economy and are also more explicit in their acknowledgment of the uncertainty around the outlook.

Treasury devotes considerable resources to analysing the current state of the Australian and global economies. Monitoring a wide range of economic data is the bread and butter work of Treasury.

This work is used to brief ministers and senior staff and develop Treasury’s understanding of the economy. It is similar to the macroeconomic work conducted in any public-sector economic agency.

This work has been formalised in other public and private sector institutions in the form of a nowcast of the economy. A nowcast basically brings together a wide range of economic data and extracts the common information in it to provide a timely estimate of the current pulse of the economy.

An advantage of this technique is that it provides a timelier read on the economy than an ABS National Accounts release. It also helps reduce the uncertainty around the starting point of the forecasts.

This may seem like a small point. However, assumptions about what the economy is doing now often have an outsized impact on the forecasts. So reducing uncertainty about the present is an important first step in forecasting.

Treasury will also embed this into its forecasting process.

Communicating uncertainty

One of the recommendations of the 2012 forecasting review was to assess the sources of error in Treasury’s forecasts in each budget. Another was to expand on the range of risks around Treasury’s forecasts of the economy and the budget.

Both of these recommendations have been implemented and appear in Statement 7 of the Budget Paper No. 1 and are also commonly found in the reports of other fiscal agencies and central banks.

The Budget Papers provide a fan chart that plots a range of future budget outcomes based on the historical experience of budget forecast errors. Unfortunately, these charts do not attract as much attention as they deserve.

While it is natural that the point estimate of the budget outcome attracts most of the attention it is in reality only one of a wide range of possible outcomes. This is illustrated in the following chart.

Chart 5 — Confidence intervals around the underlying cash balance forecasts

Source: Budget papers and Treasury

Note: The central line shows the outcomes and 2015-16 MYEFO point estimate forecasts. (f) are forecasts. Confidence intervals use RMSEs for MYEFO forecasts from the 1998-99 MYEFO onwards.

These charts deserve more attention as they are a better reflection of the possible range of economic and budget outcomes. The central point estimate forecast is essentially an estimate of how the economy may look under one scenario which assumes close to consensus global growth, unchanged commodity prices and broadly unchanged financial conditions. It will be wrong if these assumptions are wrong or if our understanding of the behaviour of the economy under these assumptions is wrong.

Conclusion

Economic forecasting is hard. The future is mostly unknowable so forecasts will often be wrong. This is particularly the case when we are confronted with the types of unprecedented circumstances that we have seen over the past decade.

A cynical view would be that forecasts are accurate when they are not very useful but inaccurate when they are most needed.

As a statement of fact that is probably true but, to a large extent, misses the point. The process of forecasting is not just about generating point-estimate outcomes, even though they attract most of the attention, it is also about exploring the uncertainties around the outlook. As policy, business and investment decisions are all about probable outcomes in an uncertain world that is no bad thing.

¹ Burgess, S. et.al. (2013), “The Bank of England’s forecasting platform: COMPASS, MAPS, EASE and the suite of models”, Bank of England Working Paper No. 471, May. Gervais, O and Gosselin, M. (2014), “Analysing and Forecasting the Canadian Economy through the LENS Model”, Bank of Canada Technical Report 102, July.

² For a review see Pagan, A. and Robertson, J. (2004), “Forecasting for Policy” in Clements, M. and Hendry, D. Economic Forecasting, Blackwell.

³ The trend forecasts simply assume that growth in the coming year will be the same as the current year or the average growth rates of the past 3 and 5 years respectively.