Beyond ribbon-cutting: measuring the real impact of transport projects

Development practitioners often rely on Monitoring and Evaluation (M&E) performance indicators to assess the results of a transport project. Collecting indicators before, during, and after a project allows us to gain insights about project execution and project outputs, which can help us, for example, measure changes in travel time or Bus Rapid Transit (BRT) system ridership. While this approach is important, well anchored into project design, and quite practical, it is not intended to evaluate “impact”. Observed changes in outcomes cannot be attributed to the project: many other external factors, such as economic conditions, interrelated policies or projects, or seasonal trends, also come into play. In other words, a descriptive approach fails to establish causality between a project or intervention and subsequent outcomes such as changes in income, labor markets, quality of life, or market efficiency.

To overcome the limitations of traditional M&E, the development community is increasingly turning to impact evaluation, an alternative approach whose methods more directly address the issue of causality. In that context, the World Bank’s transport experts have partnered with colleagues from the Development Impact Evaluation (DIME) team to rethink the way the impact of transport is measured. Two years ago, with support from the UK Department for International Development (DFID), a transport-dedicated impact evaluation program was launched: “IE Connect for Impact”. Now, impact evaluation is being implemented on 10 projects, covering rural roads, urban mobility, transport corridor development, and road safety. More projects will be selected toward the end of the year, as part of Phase II of the program.

The expected benefits are clear: informing project delivery during design and implementation, documenting the effects of policy and investment interventions, and prioritizing and filling knowledge gaps in the sector. Despite these significant benefits, transport accounts for less than 1% of all impact evaluation work — a very low proportion compared to the weight of other sectors such as in health (65% of all published impact evaluations), education (23%), agriculture and rural development (10%), or water (4%).

So, what are impact evaluation methods, and how is IE Connect for Impact applying this approach to the transport sector?
Standard impact evaluation methods rely on the identification of a treatment group and a control group. Ideally, both groups have the same underlying characteristics, the only difference being that the individuals in the treatment group receive an intervention whereas those in the control group do not. In these conditions, the differences in outcomes between the treatment and control groups are due to the intervention.

In transport, existing impact evaluations have leveraged different analytical methods and creative designs. For most IE Connect for Impact evaluations, the preferred approach is both to attempt to estimate the impact of the transport infrastructure itself, typically using non-experimental methods, as well as to test complementary interventions that can enhance the benefits of transport projects, using experimental methods. In the latter case, impact evaluation may assign individuals to treatment and control groups at random. This randomization avoids selection bias; in other words, it prevents systematic differences in the characteristics of those that receive an intervention compared to individuals in the control group. An example is a project that looks to improve the safety of informal bus (matatu) operations in Nairobi, Kenya: different drivers will be randomly assigned different feedback mechanisms that seek to provide incentives for safe driving. This evaluation is based on in-vehicle technologies and crowdsourced information, and will help determine the most effective solution at reducing road accidents, for example, driver self-monitoring systems, or systems that report driving performance to bus owners.

In situations where randomization is not practical or feasible—for example, when assessing the impact of a transport corridor or the impact of a new BRT line—IE Connect for Impact projects may use a non-equivalent control group. In that case, the evaluation would look at groups that do not share the same characteristics and rely on statistical methods to take these systematic differences out of the equation. This approach that still represents a significant improvement over descriptive M&E. Impact evaluation with non-equivalent control groups has been applied to transport projects to evaluate, for instance, how transit fare subsidies can impact job market outcomes for the poor, how paving rural roads can help villagers prosper, or how proximity to transport corridors can affect incomes.

A good impact evaluation provides the basis for sound policy making, and helps us gauge the real impact of a transport project. Impact evaluation can also influence the way the project itself is being delivered: factoring in impact evaluation right from the start is a powerful way of incentivizing teams to change their approach and embrace a more results-oriented mindset. In fact, a recent World Bank study has shown that the delivery of projects that include impact evaluation is significantly timelier: common delays are avoided and the gap between planned and actual disbursements is reduced by half. Therefore, in addition to providing hard evidence which can be used to weigh and justify policy priorities, impact evaluation can also become an effective managing-by-results tool.

“If you can’t measure it, you can’t manage it,” the saying goes. Rigorous impact evaluation gives us the means to do just that: by getting a clearer picture of our impact, we will be able to keep improving the way we design and deliver projects to bring real, meaningful change on the ground.