State of the States: Methodology

The method upholds an ideal

What is the methodology behind The State of the States report?

• Introduction
• Selecting the yardsticks
• Ensuring quality measurements of investment and outcomes
• The Commonwealth Grants Commission and the introduction of accrual accounting
• The final results
• Changes from last year and other explanatory notes and qualifications
• The full picture?

The following is from Chapter 4 in the 2002 report.

AS IN PAST years, each of Australia's States has been assessed by integrating a range of separate social, environmental and economic indicators to produce a single--but very broadly based--measure of its performance. The methodology produces a single figure assessment of performance by following three steps.

[BACK TO TOP]

First, 15 statistical measures are selected as performance yardsticks. An obvious but unavoidable shortcoming is that the choice of statistics is constrained by the limited range of useful, consistent, accurate and comparable data collected across all of the States.

In particular, there is no series that measures State government performance in redistributing wealth, despite the fact that the States account for more than 40 per cent of national public outlays and provide the bulk of Australia's direct public services, including the crucial health, education, community, environmental and justice services. This glaring gap persists despite all the attention given to national income distribution over recent years, the widespread concerns over the polarisation of wealth and income in Australia, the large contribution States make to the social wage, and the rhetorical emphasis Premiers often place on the fairness of their policy outcomes. More than a moral question of social justice, increasing inequality has diverse adverse consequences, reducing economic growth by upsetting relations between capital and labour, destroying the necessary conditions for productivity growth, encouraging crime, exacerbating ill-health, breaking-down the capacity for the social co-operation necessary for promoting environmental sustainability, and ultimately threatening the idea and practice of democracy. As noted in past volumes, from professional policy or social democratic perspectives, the fact that State governments constantly make decisions with distributional consequences without this knowledge is an outstanding public policy disgrace.

The paucity of meaningful, disinterested, accurate and comparable data collected across the jurisdictions at once justifies and hinders the State of Australia series. The scarcity highlights the need for investment in national performance monitoring and accounting if citizens are to participate effectively in public affairs and to have confidence in their governments, but the same paucity also hinders independent attempts to remedy this failure.

Be this as it may, within the limitations of the available data, the State of Australia method involves selecting five statistical measures of the relative performance of the States in each of the areas of social, environmental and economic policy. Each of the five indicators is selected according to how broadly the individual indicator appears to measure performance within each of the three still broader performance areas. Of the 15 indicators, eight are derived from the 2002 update of general grant relativities by the Commonwealth Grants Commission (CGC) and seven are drawn from Australian Bureau of Statistics (ABS) sources. All the indicators for this year's League Table measure the financial year ended on 30 June 2001. The measures are set out in the full report in Table 7 on page 21, and full references are set out in Appendix B.

[BACK TO TOP]

Secondly, the performance of each State is calculated by comparing it with the average performance of all the States. For CGC data, actual per capita levels of spending are divided by the CGC's assessments of what each State should spend in order to provide an average standard of service to citizens within their jurisdictions. This procedure assumes an average level of productivity and is consistent with the way these services are calculated to measure Australia's Gross Domestic Product. More importantly, the great virtue of the CGC data is that it enables us to compare like with like by taking into account the specific costs that are due to influences beyond the policy control of the States, costs that are known as "disabilities".

The role of the CGC and the high and unique quality of its statistics warrants further elaboration, since the quality of this work is not only too little understood and appreciated, the institution is presently under a transparently self-interested attack from the Treasurers of the three wealthiest States, New South Wales, Victoria and Western Australia.

Basically, the Commission's assessments are undertaken to facilitate "horizontal fiscal equalisation" between jurisdictions in the annual distribution of the Commonwealth's untied funding grants to the States, including the funds raised by the GST. Such a procedure is a feature of almost all the world's major federations and is also found in many unitary countries. The concept is, as the CGC has described it, "part of the fabric of nationhood--that better endowed States or regions contribute to the capacity of those whose resource bases are not as abundant or whose needs are greater, for reasons beyond their own control". The more you think about it, the better it gets. Horizontal equalisation is an authentic expression of what it means to be an Australian, as distinct from, say, an American. To be a US citizen means living in a country where per capita spending on public schools, for example, can vary by up to 200 per cent between States. This would be practically impossible under Australia's system of equalisation. The concept and practice of equalisation articulate the idea of a nation collectively choosing to invest at least something in perpetuating the ideal of an Australian community. Countries as diverse as China and South Africa have set up similar equalisation systems by drawing directly on Australia's experience.

As a by-product of the equalisation process, the CGC generates superb statistics for comparing the investment levels of the different States in particular service areas. The "disabilities" that the CGC takes into account in developing its figures include differences between States in terms of age, gender, socio-economic status, English fluency, Aboriginality, administrative scale, wage and other input costs, and the differences between capital city and isolated service delivery. In specific areas, other factors are brought to bear. In government primary education, for example, vandalism and security are also taken into account, whereas in secondary education the participation rate is considered. In roads, allowance is made for length weighted by type (sealed or unsealed) and the average number of lanes, and use (with extra weight for heavy vehicles). Recently the CGC has also introduced a physical environment factor to allow for the combined effects of rainfall, temperatures, soil type and relief on the maintenance costs of arterial roads and bridges. To allow for the interaction between road use, length and the environment on the relative maintenance costs, the CGC calculates the effects simultaneously.

In the upshot, the CGC develops a model budget the represents the closest possible approximation of the average budget, after all allowances have been made for State differences that lie outside the range of government policy, and with which all State expenditure and revenue levels can be most fairly compared. The average that the State of Australia series uses to compare the performance of the States is, therefore, a CGC-adjusted average. It is an extraordinarily sophisticated, and easily the best available State comparative measure.

The ABS statistics are more simply calculated by dividing each jurisdiction's actual per capita result by the unadjusted average per capita result for all six States. Implicitly, comparisons based on ABS data are somewhat cruder than the CGC figures, since they assume that there is a 'level national playing field', that Commonwealth policies have consistent national effects and that Australians share a common global environment. Nonetheless, the ABS statistics are of a high level of integrity, and they are primarily used in the League Table to measure outcomes. To the extent that the available levels of public investment are equalised through the CGC system, these outcomes are exceptionally good quality performance indicators.

[BACK TO TOP]

Long time State of Australia readers and specialists in this area should note that the recent introduction of accruals based data by governments has disrupted the continuity of the CGC's collection. The CGC's data is based on State expenditure from the Government Finance Statistics (GFS) collected by the ABS, supplemented by information collected from the States themselves. 1998-99 was the first year of accrual reporting in the GFS. As noted in the CGC's 2002 Update, the "Commission's analysis of the data indicated that there were problems with the classification of expenses by purpose". As a consequence, the CGC has adopted procedures to minimise the impact of the distortions in the assessments.

Technically, this means that the CGC figures used in this year's assessments are strictly estimates, not real figures. While this is less satisfactory than has traditionally been the case, since it means that the assessments may not reflect annual fluctuations in expenditures as accurately as in the past, the CGC's equalising methodology still ensures that its figures remain the best comparators available. Moreover, it can be noted that all of the State treasuries have acknowledged the difficulties that have arisen from the accrual reporting, and have accepted the CGC's modifications for the equalisation purposes. If the system continues to be acceptable to the nation's treasuries, when the stakes involve allocating funding valued at more than $36 billion, it certainly remains acceptable for the purposes of The State of the States.

[BACK TO TOP]

Thirdly, a final single figure measurement of each State's performance is calculated by averaging the three sub-totals. No weighting is used for any of the three major performance areas or the separate categories and subcategories. Social, environmental and economic policy performances are ranked equally, reflecting the philosophy underlying the methodology; namely, governments should be assessed according to their effort in a broad range of policy areas that are significant in determining the quality of life enjoyed by Australia's citizens.

[BACK TO TOP]

The social and economic policy performance indicators are unchanged from previous years. This year's environmental indicators are, however, slightly different from last year. Two changes have been made to this area's indicators, both of them being unavoidable because of the difficulty in obtaining consistent, disinterested comparable and meaningful environmental indicators. Ideally, the State of Australia series would prefer to compare the environmental performance of the States against measures of biodiversity, land clearance, land degradation, the health of river and marine ecosystems, air quality and greenhouse gases. In a matter of serious regret, Australia's governments do not have sufficient interest in the environment to ensure that such indicators are systematically collected on a comparable annual basis across the States.

Be this as it may, for the purpose of this report it should be noted that, instead of measuring the use of unleaded petrol and the availability of public transport, as occurred last year, the Table has utilised surveys of the availability of environmentally friendly household products and household water conservation. The change is simply due to the unavailability of the other series this year. Both of the new indicators are from an ABS survey of March 2001. Note also that, as is traditional, the sanitation and environmental protection indicator includes both State and local government expenditure, as responsibility for these services varies between the different government levels in the different jurisdictions. Not only does this convention ensures that the assessment measures like with like, the indicator remains perfectly valid since local governments are, of course, a State responsibility.

As with past practice, the Territories have been excluded from the assessments, mainly because the small size of their populations relative to their high level of (heavily subsidised) services distorts the basis of comparison too unfairly, but also because these jurisdictions are ultimately creatures of the Commonwealth government. The Territories do, however, contribute to the calculation of the CGC-adjusted average measure, since it would be too complex to net them out for the purpose of the Table, and in any event the standard remains equally valid for comparing the States. All the sub-national jurisdictions, including the Territories, are assessed with more extensive and interpretative analyses periodically through the State of Australia series.

[BACK TO TOP]

It is important to emphasise that neither the Evatt Foundation nor the Public Sector Research Centre hold that the assessment method used in the State of Australia series is perfect. No measurement system can pretend to be able to reduce the complexity of government, social and individual reality to unambiguous numbers. For every performance answer provided by numbers that have been torn from their context, more questions will always go begging. These qualifications do not diminish the virtues of the State of Australia's method, of which there are at least four.

First, the method is a commonsensical one that is easily understood by the intelligent and educated citizen who is curious about or wishes to get an idea or form an impression of how the different Australian States are performing.

Secondly, The State of the States gives wider circulation to the research undertaken by the Commonwealth Grants Commission than would otherwise be the case. This unique, high quality research is precious in the Australian context, but is unfortunately little known outside the world of a few specialist finance journalists and treasury officials. As well as spreading the information around, the series aims to help build an appreciation for the institution of the Commonwealth Grants Commission itself and to increase understanding of the important role it performs.

Thirdly, unusually, the method used by the State of Australia series yields broad comparisons between the States based on their social, environmental and economic performance.

Finally, uniquely, the method provides a readily accessible, broad, overall government performance comparison. Which State is closest to that formula for perfection: first class social services, uncompromising environmental protection, and thriving, job-generating economic activity?

That said, the chief virtue in the approach of the State of Australia series does not reside in the results it produces. Rather, a higher value is embodied in the way that it upholds as an ideal principle the belief that governments should be assessed on a broad basis according to the effort they put into serving Australia's citizens. The scarcity of assessments along these lines means that the most enduring significance of the League Table results lies in the way that they are used as an instrument to stimulate questions and debate about the performance of the States and the quality of different assessment methodologies. As such, the method runs against the grain of the continuing emphasis among Australia' public commentators on the need for governments to primarily be accountable to finance markets. It also tilts against many of the popular methods of assessing governments, such as the charisma of the leaders, and most of the other less well known and less accessible performance methodologies used in Australia and overseas.

All the same, no statistical series, no matter how broadly based or sophisticated in its adjustments, can pretend to capture the full complexity of government performance. The method used in the State of Australia's series is distinguished by virtue of its generally straightforward empirical procedures and the broad range of measures it draws upon to formulate the League Table. Yet, if the method is less partial than most other performance assessments, it still remains only a partial representation of how well our governments are doing.

In particular, it should be recognised that the State of the States League Table ranks the governments in relation to each other. It does not provide an absolute measure of performance. Irrespective of the different positions on the Table, it is entirely possible that all the States have performed better than last year. Equally, it is possible that the standard of all of them may have dropped. As noted in past editions, it would require a much larger study to ascertain whether the overall standard of government is rising so that, like Lewis Carroll's Alice, "it takes all the running you can do to keep in the same place", or whether, as the famous American journalist, Hunter S. Thompson, regularly complains, "the standard gets lower every year, but the scum keeps rising".

[BACK TO TOP]

© 2001-2005 The Evatt Foundation

Main Quadrangle (A14)
University of Sydney NSW 2006
Tel: +61 2 8090 1170
Fax: +61 2 8090 1171
[Privacy] [Credits]

URL:http://evatt.org.au/archives/methodology.html
Last Modified:Tuesday, 15-Nov-2005 18:30:08 EST

The Evatt Foundation is a LaborNET site, proudly designed and sponsored by Social Change Online.
Please report any site matters to the webkeeper