Measurement of governance, government, and the public sector

There is no doubt that governance can be complicated. It has been subject to extensive analysis and explanation by a variety of experts, with a corresponding variety of definitions. Competing philosophies are based on not only assumptions about the intersection of economic and political management, but also the relevance of institutions to development outcomes. Measurement of such complex concept can be an awkward tool in the midst of such ambiguity. In fact, no one data source can claim to provide a single measurement of governance. Even aggregate indicators that rigorously compile data from various sources are imprecise instruments, leaving policymakers uncertain of the steps needed for improvement.  At best, data sources clearly define governance for their purposes and design measurements that then capture information about the dimensions as defined by them. But can governance be measured with any precision? Yes, by ensuring that assumptions rest on solid conceptual grounds, i.e., definitions are clear and withstand logical scrutiny, and that data is collected with appropriate and rigorous methodologies.

1. As a step toward conceptual coherence, Francis Fukuyama recently proposed an operational definition of governance that takes its starting point as “a government’s ability to make and enforce rules, and to deliver services.” This definition is limited but not limiting: it focuses inquiry on the execution of policies by the public administration, rather than the quality of those policies (a normative orientation), or the checks and balances that curb state power. He proposes four measures based on his definition, ultimately settling on capacity and autonomy as the most appropriate dimensions for capturing the quality of governance1:
2. Rules (procedural measures), including meritocratic recruitment and promotion, technical expertise, formality, etc;
3. Capacity, including both resources and degree of professionalization;
4. Performance (output measures), consisting of the delivery of services;
5. Bureaucratic autonomy, a dimension of governance that is inversely related to the number and nature of mandates provided by political authorities.

Fukuyama’s proposed measures point to a larger question about the right combination of institutional arrangements, informal norms, organizational capacities, and human and material resources that result in the optimal performance of the state. We may generally agree on what the state is supposed to do, but not necessarily on what the state is supposed to look like, or exactly how it is supposed to achieve its goals.  There is no longer reliance on the one-size-fits-all approach. Political context, organizational norms, and resource constraints determine outcomes to a far larger degree than once expected. In making his argument above, Fukuyama laments the overemphasis on institutions that limit or check the state, and the resulting lack of data on the actual functioning of executive branches and their bureaucracies. This claim reflects the challenge of collecting data that is reliable, timely, and useful for both the research and practitioner communities.

And yet, progress is being made. The Actionable Governance Indicators (AGI) Initiative was launched in 2007 by the Governance and Public Sector group in an attempt to bring measurement to the forefront of such discussion. We are inching closer to finding out what makes the “black box” of reform work, by taking stock of what exists, by measuring what practitioners, governments, and citizens are doing, by tracing processes and behaviors of relevant agents, and by attempting to understand how all of this relates to development outcomes.What we have learnt so far is that there are no easy answers. Many considerations affect the quality and relevance of data:

• Degree of aggregation, varying from the most particular piece of data to the compilation of data from hundreds of sources.
• Method of data collection, including expert assessments, case studies, surveys, government statistics, project M&E, direct observation, semi-structured questionnaires, interviews, etc. One of the most contentious issues revolves around where data falls on the continuum between perceptions and facts, and how this impacts comparisons over time.
• Size of dataset, from ten points of data to thousands.
• Timing and frequency, from ex-ante and ex-post to monitoring, impact evaluation, and follow-up studies at one, five, and ten-year increments.

The attempt is not, and arguably should not be, an academic exercise. The beneficiaries of this work are development practitioners and policy makers around the world. And because of such focus on practitioners, impacts are important: they inform our understanding of success and generalizability, which then inform the possibilities for scaling up. Unfamiliar concepts certainly require appropriate theorizing, but they also need data for validation and interpretation.

Measurement helps point the way forward, by understanding success and failures and by helping adapt to a changing institutional and political environment. Several international datasets are addressing the shortage of data on the public sector, with subjective, objective, and mixed-type methodologies:

• Bertelsmann Transformation Index (BTI) consists of biennial expert assessments on development countries since 2008. Indicators include management of the public sector, steering capability, resource efficiency, consensus building, and international cooperation.
• Sustainable Governance Indicators (SGI) consist of biennial expert assessments on OECD countries since 2009. Indicators include strategic capacity, inter-ministerial coordination, policy implementation, and institutional learning.
• Institutional Profiles Database (IPD) consists of triennial expert assessments on over 100 countries worldwide since 2006. Indicators include civil service training, remuneration, and performance, as well as the capacity of political authorities.
•  Global Integrity consists of annual expert assessments on 30-50 developing countries since 2004. Indicators include financial disclosure, conflict of interest, merit-based recruitment, and human resources management for the civil service.
• Human Resources Management (HRM-AGI) indicators have been piloted in 8 countries. Indicators are comprehensive across institutional arrangements, capacities, and performance, and include over 200 indicators on aspects of a meritocratic bureaucracy.Reliable data on the enabling environment for good governance and public sector management is also being produced: service delivery, rule of law, and civil society activities. The Public Accountability Mechanisms (PAM) Initiative is producing data on the institutional arrangements and organizational capacities that support transparency and accountability mechanisms, as well as the performance of these systems. These datasets are paving the way for others to both collect data and to perform statistical analysis on the relationship between bureaucratic performance and development outcomes.So-called third generation indicators, such as the recently launched work on Indicators of the Strength of Public Management Systems (ISPMS)focus on what the public sector actually does - its behavior, its performance and the outcomes it produces. These indicators will provide even more useful data on the impact of governance and public sector reforms, allowing for a better understanding of how inputs, structure, and capacities combine for enhanced development outcomes.

This approach is neither novel nor original. Kaufmann, Kraay and Zoido-Lobaton introduced and used a similar approach in 1996 for the development of the Worldwide Governance Indicators, although the end product is an aggregated index, rather than disaggregated data. (http://info.worldbank.org/governance/wgi/index.asp)