Skip to content

Measuring progress: From theory to practice

Check against delivery


Thank you for inviting me to be part of this discussion session today.

In modern societies, everyone is likely to agree that progress in improving wellbeing is important.

However, progress and wellbeing mean different things to different people. Each person will have their own view of what is specifically important with respect to their own wellbeing, the wellbeing of others, and the weight they place on each element of wellbeing.

Having a consistent and robust understanding of wellbeing, that can also cater for these differences of view, is therefore critical for measuring progress.

Treasury’s policy objective, set out in our mission statement, is to improve the wellbeing of the Australian people. Consistent with this objective, Treasury has developed a wellbeing framework to underpin analysis and advice across the full range of our public policy responsibilities. That framework is necessarily pluralistic.

Statistical indicators affect judgements about the current state of affairs, trade-offs between the dimensions of wellbeing, what societies and individuals value and therefore, what are the appropriate policy responses.

The challenge for economists and statisticians is to ensure that we collect appropriate statistical indicators, and ensure these indicators are well understood and communicated.

Towards a better measure of economic performance

For policy choices and decisions to have a reasonable prospect of improving wellbeing, we need to base them on reason, as well as empirical evidence, and therefore we require an evaluative framework.

Quantitative and qualitative frameworks provide a solid foundation to identify, promote and progress policies to improve wellbeing.

But we need to ensure our frameworks and metrics are robust.

While by now it may be a hackneyed phrase, it is undoubtedly true that what we measure affects what we do.

If we focus on the wrong metrics, or use them without acknowledging their limitations, they can lead us down the wrong path.

The Stiglitz-Sen-Fitoussi Report stressed that progress is more than increases in income, wealth or production. Nevertheless, we continue to struggle to have a broader focus in defining and measuring progress.

The most often used measure of progress is growth in (real) GDP per capita.

However, economists and statisticians have long known that GDP is not, and was never intended to be, a measure of wellbeing or progress.

Further, while we have known the limitations of GDP, we have done little to discourage its use.

In fact, we arguably, if inadvertently, do much to promote GDP as a measure of progress – for example in the Budget papers, which present detailed analysis on the level and growth of GDP, as well as its determinants.

One reason for this is that GDP is well known. Another is that a huge and sustained intellectual effort has gone into developing, and refining the measurement of GDP.

It is also true that GDP provides important and useful information. We just need to be mindful of how it is used. For example, GDP as a measure of market production, is a useful tool in managing the macroeconomy because it is a key measure of the level of economic activity, and hence employment, in the market sector.

An important problem with GDP, however, is that while it is well known, its limitations are not well understood and it is mistaken as a measure of overall progress.

The Stiglitz-Sen-Fitoussi Report examined in detail the limits of GDP as a measure of progress. It is useful to highlight a few of these limitations.

GDP is largely a measure of market production. It therefore misses a significant amount of household activity as it excludes home production of goods and services (other than imputed rents).

Further, being a market-based measure, it does not appropriately measure the goods and services of the public sector.

GDP can also give a misleading picture of how well the economy is performing.

Take, for example, the lead-up to the global financial crisis. In the period before the crisis, measures of GDP did little to warn of the increasing fragility of the global financial system. Indeed, in the period before the financial crisis, much of the strong growth in GDP in many countries was driven by unsustainable asset price inflation and strong growth in consumption funded by increased borrowing.

Measurement issues in the national accounts have also become more complex, as output in the economy has moved from relatively simple goods and services to more complex ones. For example, the output of the financial services sector is measured both by observing direct fees and estimating the value of services paid for indirectly as part of interest payments.

The behaviour of these estimates during the global financial crisis has led to a vigorous debate about their value.

By way of example, the UK national accounts reveal that the UK finance sector grew at it fastest pace on record in the fourth quarter of 2008. That was, if you recall, the time when the UK financial sector (along with those of other major North Atlantic economies) was suffering a major collapse, rather than contributing significantly to the output of the UK economy, as most people would understand that term.2

Without going into detail, this perverse outcome arose because compensation for bearing risk is included as output produced by the finance sector. As a consequence, when banks increased interest margins in late 2008 in response to a radical reassessment of expected defaults and liquidity risks, this increase was booked as an increase in financial sector output, rather than a correction in the price of risk, as it should have been.

In the lead up to the crisis, widespread underpricing of risk almost certainly meant that output from the financial sector was overstated, even though the sector’s difficulties in appropriately pricing risk did little to improve economic wellbeing.

There are different views on these issues but, following the financial crisis, the conceptual basis of the way we impute the output of the financial sector is now the subject of much more detailed examination.

Another problem with using GDP as a measure of progress is that it is a gross measure. That is, it does not take into account the depletion of the physical and natural capital stock; a cost associated with the production of output due to the erosion of an asset.

Depreciation effectively means a proportion of output needs to be set aside to renew or replenish our asset stocks.

Take for example the treatment of natural resources.

The value of natural resources when they are extracted is treated as production, an increase in GDP. However, natural resources are assets already owned by the community. Their extraction and sale represents the transformation of an asset (the natural resource) into another asset (cash). By not counting the depletion of the natural resource asset, production or valued added, as measured by GDP, is overstated, possibly at the expense of the wellbeing of future generations.

A similar argument can also be made for including the degradation of our natural environment.

Better measures

With the limitations of GDP, the Stiglitz-Sen-Fitoussi Report states, ‘the time is ripe for our measurement system to shift emphasis from measuring economic production to measuring people’s wellbeing’.

But this is no easy task.

The Report makes a number of specific recommendations. Again, it is useful to highlight a few.

The Report suggests that when considering ‘material wellbeing’ or material living standards, we should shift our focus away from production or GDP to household income or consumption.

But, as noted in the Report, while current household income and consumption are important measures of living standards, it is consumption poss
ibilities over time that are more clearly relevant to wellbeing. Therefore, measures of income and consumption need to be considered in conjunction with household wealth.

Measures of wealth are crucial for measuring sustainability. A household that raises its current consumption by running down its assets or by borrowing has increased its present wellbeing, but possibly at the expense of its future wellbeing.

The Report also suggests that more prominence needs to be given to distribution. While changes in average income, or consumption, are important, we need better ways of measuring the distribution of consumption possibilities, assuming we can measure them properly. One approach suggested in the Report is to focus on the median, while also identifying changes at the top and bottom of the distribution – although even this is only a small step forward, given the need to consider life-cycle issues.

Further, as noted previously, we need to develop ways of better measuring the non-market economy, particularly household production.

Addressing these issues will get us somewhat closer to better understanding material wellbeing. However, the Report also outlines a number of other dimensions, which are also important for wellbeing. These include health, education, personal activities including work, political voice and governance, social connections and relationships, the environment and insecurity.

I would note that many of these non-material dimensions of wellbeing are reflected in Treasury’s wellbeing framework, which is, in part, based upon the capabilities approach of Amartya Sen.

These non-material dimensions do, however, raise significant measurement difficulties for economists and statisticians.

A key dilemma for the ‘capabilities approach’ has been how to measure what people could do, as opposed to what they actually do.

The Report suggests that to measure the non-material dimensions both objective and subjective measures are important.

Quantitative measures of subjective aspects of wellbeing are likely to assist in providing a more comprehensive picture of quality of life, although they clearly would not replace measures of material wellbeing. Further, subjective measures may also assist us in better understanding what individuals and society value.

These additional dimensions, and difficulties in measuring them, highlight the need to use qualitative data when quantitative data are not available, and the need to use a wide range or dashboard of indicators.

In practice, this is something those involved in public policy already do, to varying degrees. But it is important that we continue to consider ways of improving how we measure these dimensions of wellbeing and that we do our best to ensure that our measures are better understood and accepted.

Improving our understanding and use of measures of progress

While acknowledging the limitations of current measures and improving our measures of progress are both worthy goals, shifting to new paradigms to measure progress will require a new willingness to take the new measures seriously.

Little will change if those making the decisions that affect peoples’ lives — governments, communities and individuals — are not willing and able to look beyond existing measures of economic performance and consider how their decisions affect wellbeing and progress.

The 2010 Intergenerational Report is a step in this direction, but more can be done.

One significant problem is that these new measures of progress need to be better understood. Some run counter to our current benchmarks, particularly increases in GDP.

If we therefore are to adopt new measures of progress, we must continue to build broad community support for them, and learn to use them appropriately, taking into account their limitations.

1 I am extremely grateful to Shane Johnson for help with this presentation.

2The Contribution of the Financial Sector – Miracle or Mirage?, Andrew Haldane, Executive Director, Financial Stability, Bank of England, The Future of Finance Conference, London, 14 July 2010.