Last month, leaders from the major multilateral development banks convened in Rome for the Spring Meetings of the Evaluation Cooperation Group, which is a forum that brings together nearly 20 MDBs in order to promote collaboration and learning across their respective evaluation functions. The meetings were chaired by Oscar Garcia, ECG Chair and Director of the Independent Office of Evaluation at the International Fund for Agricultural Development (IFAD).

At the meetings, Caroline Heider, Director General, Evaluation and Senior Vice-President at the World Bank Group chaired a distinguished panel discussion on the future and relevance of multilateral development banks (MDBs). Panelists included Hon. Bambang Brodjonegoro, Minister of National Development Planning of the Republic of Indonesia, Joachim von Amsberg, Vice President, Policy and Strategy of the Asian Infrastructure Investment Bank, Cheryl Gray, Director of the Office of Evaluation and Oversight at the Inter-American Development Bank, and Enrique Garcia Rodriguez, former Executive President of the Development Bank of Latin America (CAF).

In this edition of Conversations, we bring you highlights from the discussion. We have edited the conversation for brevity and clarity. A video of the full panel discussion is available here.

Caroline Heider: This discussion was triggered (in part) by a recent report from the Center for Global Development, which put forward some suggestions and ideas for what multilateral development banks (MDBs) should be doing differently. Of course, there are also specific institutional agendas. The World Bank Group, for example, has recently approved its Forward Look, which among other things, assumes a central role for the private sector in the future of the development agenda. In many ways, today we have a very different world from 30 years ago, when private sector flows were much, much smaller. Another priority is sustainability.

I’ll start with you Minister Bambang. Indonesia is a powerhouse in the southeast Asian region but also globally. You play an important role in the G20. You are a key client to most of the MDBs, whether it's of the Asian Development Bank (ADB), the Asian Infrastructure Investment Bank (AIIB), the International Fund for Agricultural Development (IFAD), or the World Bank Group (WBG). When you think about the MDBs, and you have interacted with MDBs for a long time, what do you see as their biggest challenges today? What would you wish to see the MDBs do differently in the future

"Aside from climate change, we have one global issue that affects not only the low-income countries or middle-income countries, but also the high-income countries, and that's inequality or income disparity. If inequality or income disparity becomes bigger, or uncontrollable, I believe it will threaten political stability in every country. So, in addition to the question of sustainability, MDBs need to reset their agenda to fight inequality." -Hon. Bambang Brodjonegoro, Minister of National Development Planning, Republic of Indonesia

Bambang Brodjonegoro: Thank you, Caroline. I think, first, the role of MDBs is to help the member countries, especially the least developed countries or the low-income countries to get to a better situation. And I believe the regional development banks have been trying very hard. I can tell you, for example, in the ADB now, they have reduced quite significantly, the number of low-income countries in Asia.

Secondly, MDBs must have a clear, measureable development target,so we can evaluate, whether their operations have been successful in their respective member countries. This should be basic for the MDBs.

To answer your question about the challenges ahead, aside from climate change, we have one global issue that affects not only the low-income countries or middle-income countries, but also the high-income countries, and that's inequality or income disparity. If inequality or income disparity becomes bigger, or uncontrollable, I believe it will threaten political stability in every country. So, in addition to the question of sustainability, MDBs need to reset their agenda to fight inequality.

But this goes beyond the idea that MDBs are just a lender or development financing institution. I think now is the best time for MDBs to also encourage sharing experiences. There have been some countries doing well in reducing poverty or tackling climate change. MDBs can act as enhancers by bringing all the best experience and sharing it with everybody. The World Bank, for example, can do more by bringing what works best in South America to Asian countries, or from Asia to Africa. In today’s global environment, I think good policies can transferred and implemented anywhere in the world as long as you know how to modify the policy.

Caroline Heider: Excellent points about global issues that concern all of us, and how the MDBs can become a facilitator of knowledge exchange. Let me turn to you, Joachim. As an institution, the Asian Infrastructure Investment Bank is, perhaps, the newest kid on the block. Looking at the CGD report that focuses very much on the long-term existing institutions, was there something in there where you said, "Oh, these are good ideas, we should pick them up"? Or do you have a very different vision, which could help the MDBs also think about their own transformation?

Joachim von Amsberg: Maybe three thoughts that connect to the CGD report. One, I think we shouldn't talk about the challenges of the MDB system without grounding these in a clear recognition of not only how relevant the MDB basic business model still is, but how badly it is needed, to deal with some of our big global challenges. I think the biggest contribution of the report is not so much in the recommendation, but in retelling the story to the outside world of what a fantastic business model this is - the basic financial model of leverage, the basic operational model of putting knowledge, finance and the honest broker role together, and the basic governance role of allowing the international community to act jointly in a way that is ultimately driven by technical competence. It's a fantastic model. The story needs to be told again and again, because we badly need it.

Secondly, the MDB system is struggling to balance ever-increasing ambitions, and essentially flat or reduced resources. I see two sides to the increasing ambition – one legitimate and the other, political. The world needs more investment because there are bigger challenges today. There's climate change, there are refugees, and a whole set of issues that require MDB investment and action. But there is also increasing ambition that is more politically motivated rather than driven by the actual demands of client countries. I feel the MDB model is overstretching itself, overreaching. It is acting like ... a mortgage bank, that doesn't just ask you whether your house is worth what you're claiming and you're able to repay the loan, but a mortgage bank that tells you how to paint your living room, what furniture to buy and what spouse you can bring to live in the house.

And that's not very attractive to clients. I think to deal with that sort of disconnect between ambition and resources that have essentially stayed flat, or relative to the demands, become much smaller, you need, maybe, three directions. One is, limit ambitions to what the real client needs are. Secondly, increase resources, yes, I think that's necessary. And that refers to the capital of the system as a whole. And thirdly, implementation of the billions to trillions agenda. That's the whole discussion we've had for years in the MDB system, about how to use our capital to leverage much more, to catalyze, as opposed to simply fill financing gaps. That was my second thought.

And the third and last thought I want to share, is there's a need to redefine the interaction between the aid model of development, and the development banking model. These are two very different ways of thinking about development finance, and the MDBs, in different degrees, combine these two. The aid model is one where, in a sense, there are rich countries and poor countries and resources flow from rich to poor, and knowledge flows from rich to poor. A very linear thinking of development, and frankly, one that is increasingly less relevant in the world in which we live today.

The development banking world is a very different one. It's countries getting together, putting up capital, and creating a mechanism that is financially sustainable to leverage that capital and, thereby, generate less costly access to capital than they would have had on their own.

To different extents, the MDBs have suffered a contamination of the development banking model by the aid model. And I think that has a lot to do, or is a major source for the over-ambitiousness of the MDBs. I think that relationship needs to be redefined if the MDBs don't want to be stuck as sort of left-over aid agencies. And that's not to diminish the aid model, it's just to say that I think there is clarity needed about what one is trying to achieve. And I think, there, the CGD report makes some quite interesting forward-looking suggestions, because the combination of, let's say, grant money for global public goods, and self-sustaining development banking finance, is going to be an important challenge for the future.

The report suggests in a sense institutional specialization – the World Bank does the global public goods, regional development banks do the country business. My take is slightly different. I think we need two different financial instruments, implemented by the same institutions because it's more effective to actually bring the global public goods into national investment programs. For example, when you invest in geothermal energy in Indonesia, you need to tap both resources to make the investment viable, rather than have two totally different conversations. And therefore, I think the way forward is more clarity about grant flows that are specifically pursuing global public goods or global agenda, and then finding effective ways of blending them with development banking or self-sustainable financing that is driven by country demand.

Caroline Heider: I want to actually pick up on the point that you made about the ‘billions to trillions’ agenda. Enrique, from your perspective after 25 years of leading the Development Bank of Latin America (CAF), how do you see the MDBs addressing this mobilization challenge?

Enrique Garcia: This issue of the catalytic role has become a very important one because, you know, it's not only the capital markets, but you have to try to convince other players with your quality of analysis and good projects or good programs. I think that's a very, very important thing.

I agree with Joachim’s point that one has to be careful on this issue of aid institution vis-a-vis bank. For a modern development bank, you have to adapt yourself to have the proper windows and activities to go to both. For instance, it would be, in my opinion, very serious mistake to say, "The countries that already have a higher rating, uh, let's forget about them. Let's concentrate only on the poor ones. No, that's a very serious mistake, because then you don't have a sustainable institution.

You must differentiate, of course, the type of products and services. You have to find ways to have more grant money for the poor, or for areas in middle-income countries where inequality is a very serious matter. And try to help that balance. I think that's a challenge. If you are in the business of these development banks at the national and international level, you have to rethink all the time. Reinvent yourself all the time.

Watch the entire ECG High-level session on evaluation and transforming multilateral development banks:

Caroline Heider: Let me come to you Cheryl as the other evaluator on the panel. As evaluators, we tend to have a little bit of a different perspective. From the different evaluations you have done, what would you see as the next challenge for MDBs, and how do we get there?

Cheryl Gray: I think as an evaluator, I am inevitably a little more cautious, and perhaps skeptical. So I'm going to add a little bit of a different perspective here. To me, the Development Bank of Latin America and the Asian Infrastructure Investment Bank offer a fascinating model as to what you can create. I worked 25 years at the World Bank and now six years in the IDB. And, certainly in Latin America especially, where we're dealing with really sophisticated clients, I think it's really very hard for MDBs to have a raison d'etre in this environment now. It didn't used to be. You could borrow AAA, and lend to countries that had no access to capital markets, and you know, you had a lot of advantages.

"I see, especially the Bretton Woods Institutions as being on the margin of competitiveness right now. And I don't know where we go from here, but I hope they can kind of reform themselves, because otherwise, I think some of these new models are going to come along, and in the long run, competition ... who knows what's going to happen?" -Cheryl Gray, Director of the Office of Evaluation and Oversight, Inter-American Development Bank

In Latin America, you've got counterparts who have ready access to capital markets, frankly, at the same interest rate that we can offer. You have fantastic knowledge and brilliant people on the other side of the table, who have great think tanks and can, you know, bring in the knowledge. And, you have, frankly, in my view, real governance challenges in the MDBs. We could go into a lot of details on this, but in my view, and I'll be honest about this, the Bretton Woods Institutions are very expensive. They're very expensive models. They're very high-cost. And the governance is imperfect, and it's very hard to put a curb on that. The World Bank is enormous. The IDB is enormous. I have to say it, but my guess, it's about five times the cost of the CAF, for the same amount of lending. About five times the cost.

And the MDBs have to pay for this, one way or another. You're not building capital, you know, you're drawing down capital. Right now, in today's world, it’s not going to be easy to get new capital. We’ll see what happens, but you know, Europe and the US are not particularly in a mood to put in a lot more money. You've got these new competitors coming along.

"I see, especially the Bretton Woods Institutions as being on the margin of competitiveness right now. And I don't know where we go from here, but I hope they can kind of reform themselves, because otherwise, I think some of these new models are going to come along, and in the long run, competition ... who knows what's going to happen?"

We did an evaluation, for example, of middle-income countries in Latin America, and asked them, "What do you like about what IDB offers?" And it was kind of surprising to me, they like the package. They like the package of good design projects, project management, procurement, safeguards, continuity through political shifts, the security that comes with the package. But, that is not necessarily a real cheap package. And, of course, some people say it's just a hassle. Safeguards are a hassle. Procurement's a hassle. I'm not sure that's the case, and I'd love to hear from the minister, because I think many minsters of finance want to be sure that money is well spent, and the procurement that comes with this is of value.

So, you know, I guess I'm a little bit less sanguine about the future, unless you can get stronger governance. And we could go into that, you know, at some point. But the governance of the Bretton Woods Institutions does not control costs. I'll just put it that way.

Caroline Heider: So very different perspectives, from "The model is still very valid," to, "Well, we do need to do some rethinking here. Let me turn to you again, Minister Bambang. Indonesia is a very different country than it was 30 years ago. When you think back, what did it take to transform Indonesia to where you are today? And what have you learned that you can share with the MDBs and other countries?

Bambang Brodjonegoro: We in Indonesia got a lot of benefit from utilizing as much as possible the development support from the MDBs. As you know, Indonesia is a member of World Bank, ADB, Islamic Development Bank, and now AIIB. So, we are trying to utilize as much as possible. Each institution has their trademark in Indonesia. I think the World Bank is quite well known in Indonesia for their basic infrastructure that has been built since 1970s. But lately, for example, World Bank is known more for the poverty issue.

One MDB role that needs to be strengthened is that of being knowledge institutions. Another is crisis support, as we saw in Indonesia with the Tsunami in 2004 and the financial crisis in 2012 and 2013, which hit emerging economies.

Caroline Heider: Cheryl, can you give us an example where you've seen the Inter-American Development Bank learn and transform part of their business, or an approach or an instrument?

Cheryl Gray: The example that jumps out the most is when the last capital increase was done, the ninth capital increase, IDB-9 it's called. It was written into the agreement that after three years, our office would evaluate whether the IDB was implementing all of the conditions fully and effectively. And it was a lot of conditions, I'll tell you. The shareholders put a lot more conditions on IDB than they did on most of the other MDBs in that round of capital increase.

So, we came along after three years and did a very large evaluation: 22 background papers on different topics, and we presented it at the annual meetings, and I think it had a huge influence. Management listened, the governors listened. Three or four major steps were taken in response, and a lot of small, minor steps. And I think it was very constructive. I would hope that if you talked to the board and the management, they would also feel it was constructive.

Joachim von Amsberg, Vice President, Policy and Strategy, Asian Infrastructure Investment Bank

Caroline Heider: Joachim, how is AIIB approaching the learning aspect. Are you borrowing from the other MDBs, or creating something new altogether?

Joachim: As the AIIB management, we are asking ourselves these questions every day because we have this unique opportunity of creating a new institution, based on decades of learning by the existing MDBs. So we wake up at night, ask ourselves, Are we learning the right lessons? There are those lessons of what's working well in the model that we have used as the foundation of our new bank.

For example, we have been deliberate in ensuring transparency around our multilateral governance mechanism, where we look in some ways quite like the existing MDBs, even though with a non-resident board. There is also a very strong commitment to project quality – quality standards, due diligence, strong social, environmental and governance policies. All these are things we have picked from what is working well in other MDBs.

In other respects, the learning is around where we want to innovate, or where we want to be different from other MDBs and partners. One is focus. I think we have a much clearer, much narrower focus, and we are quite keen to keep it that way - infrastructure. And that's big enough. That's very big in itself. So infrastructure and project finance. We are a project finance bank. We are not providing the range of advisory services that other institutions provide. We don't do policy lending. We don't administer grant funds. We structure project finance. Very focused mandate and very focused task.

Along with that goes, I think, a second distinguishing feature, which is leanness. We aspire to be a very lean organization. Small staff, low cost, nimble, fast, responsive to clients. That is quite different from many, not all, of the existing MDBs. But we see that as what clients are most asking for. "Respond to our needs quickly. Focus on project quality, but don't add too many whistles, and don't overburden projects. Don’t eat us up with your bureaucracy. That leanness is very important.

Now, then come the other lessons. How do you stay focused and lean? One of the factors that I think will help us is our shareholding composition, which in a sense is a bit more like CAF's. We have a very strong overlap between borrowers and shareholders. 75% of our capital must be held by regional members. It creates a strong alignment between board, shareholders, and borrowers. Then we need to be built on financial strength. We want to be a financially sustainable institution, where the borrowers pay our bills. I think that sets the incentives for serving the borrowers and it avoids some of the pitfalls of the overreach and overambitions that come with fundraising.

Caroline Heider: Joachim, I want to continue with you and ask about AIIB's vision for evaluation. Focusing on independent evaluation, what can we do better to help the transformation of the MDB system?

Von Amsberg: Well, some early thoughts, but as much an invitation to help us shape the evaluation function in this new institution. I think one is we must distinguish along the spectrum of different evaluations. There is deep evaluation that is about research, long-term learning, impact assessment, understanding the results chain. These kinds of evaluations are really important. We want to draw from them, but we probably won't be in the space of doing them ourselves. This relates to our focus. This is a public good, and rather than replicate those functions, we want to connect to the network, and draw from what's being done in the world by other institutions on this sort of long-term research type learning.

"We want to create a model [for operational evaluation] that is independent, yet also very close to operations. And I think the two can be reconciled. Independence is absolutely necessary, no questions about that, but it can be much closer in time and interaction than it is in some other institutions. We want very quick feedback rather than learning years after a project has closed." -Joachim von Amsberg, Vice President, Policy and Strategy, Asian Infrastructure Investment Bank

Then there is the more, let's say, operational evaluation. Here, we want to create a model that is independent, yet also very close to operations. And I think the two can be reconciled. Independence is absolutely necessary, no questions about that, but it can be much closer in time and interaction than it is in some other institutions. We want very quick feedback rather than learning years after a project has closed. We have a function where the evaluation is in one unit together with integrity and compliance, so we have an opportunity to invent something new here.

The last aspect is less about the structure and more about the culture, which I think for evaluations is more important. How can we create the right learning culture in the organization? That depends as much on the evaluators as on the evaluated. Sitting on the evaluated side, if I don't have the intellectual curiosity to learn what is working and what isn't working, it's very difficult for evaluation to be effective. So, I think setting up a system where the institution is curious, the institution is eager to learn is essential. And we were trying to build that into the corporate culture of AIIB. But again, these are just ideas. We need all the help in the world to make this work.

Caroline Heider: Excellent, great. It reminds me of conversations we had before you left the World Bank Group, so I'm very pleased to hear.

Von Amsberg:Yeah, I was the recipient of Caroline's evaluations.

Caroline Heider: Minister, over the last 15 years, we’ve seen many more client countries making enormous investments in creating their own evaluation capacities. Is that the case in Indonesia? Where would you put your country and what’s your vision for evaluation?

Brodjonegoro: So far, we haven't had a formal approach to evaluating MDB operations in Indonesia. Of course, from time to time, we learn about independent evaluations that have been produced by different MDBs. And, perhaps, this is a good idea now, in my current capacity as Minister of National Development Planning. So maybe in the future, we could commission our own evaluation of each MDB in Indonesia.

When it comes to development, these evaluations can help us sometimes when we are trying to negotiate with our partners. We can a look at what World Bank has been doing so far, what ADB has been doing so far, and we know their limitations and where they can best deliver.

Caroline Heider: Excellent to hear. Turning back to you, Enrique. At CAF, you had impact evaluations. I don’t know whether you had an independent evaluation function as well. Can you share more about how CAF uses impact evaluations, and what improvements you’d wish to see?

Enrique Garcia: Well, I think CAF has done a reasonably good job, and our impact evaluations have even been recognized internationally for their quality. If I was to make a recommendation, it would be first to bring some independence to it by making the impact evaluation office report to the Board. The second thing has to do with ex-poste evaluations, where it isn’t always clear where to draw the line between accountability and learning lessons for the future. I see that they still mix the two.

From the standpoint of my office, I was pleased with our impact evaluation work. But, perhaps, they could be a little more client-oriented. This is a tough job. If you are going to judge, nobody likes that. But you need to show why you are doing this, and that you are going to help the institution, and even the officers, to improve the quality of their work, in the analysis and evolution of projects.

Caroline Heider: Thanks, Enrique. Let me close with Cheryl. How does evaluation need to transform itself in order to help MDBs in their own transformation?

Cheryl Gray: One thing I would stress is the basic architecture, and this is one area where I think the World Bank is quite impressive, I must say. I always stress the architecture of evaluation as being important, the architecture of monitoring and evaluation. And the way they (the World Bank) have built it over, what? 25 years or 30 years, is ... you know, the project team lays out what they expect to achieve with a simple results framework, and indicators along the way, and it's their responsibility to monitor the project on a regular basis, and to do their own self-assessment, and then the independent evaluation office comes along and validates that. And if you go to the World Bank, you know, the last 30 years, every single project, you can find an independently validated, honest view. (editor - learn more about the World Bank Group's self evaluation systems here).

I was in operations for a long time, before I went to evaluation, and I can tell you, as an operational manager ... you never get punished for failure. You get punished for overstating success. I mean, obviously, they want you to succeed, but if you have a project that's not doing well, and you report that it's a success, and then IEG comes along and says it's a failure, and that happens repeatedly, you know, they start asking you as a manager, "What are you doing?" If you have too much success, they come along and, you know, senior management ... "All right, you know, are you sure about this?"

And I thought that was a very healthy incentive, and it's something, Joachim, for your bank. I mean, to allow failure and to not punish, and to say, "This is part of being a development bank is really important.

IDB, you know, is trying, but it's hard. The culture finds it hard to admit failure. And I think this is something that's very, very important. And our little office, doing that validation, is what ... it's, I say, "The tail wags the dog." It's what keeps the whole system honest, because before we started it in IDB, they reported 100% success. Literally. And we have that on record. Whether there was no evidence, where there was a lot of evidence, it was always 100% success, because nobody ever checked it, and why not?

But once you start checking this, and it's not real costly to do, the whole system kind of falls into place, as long as you allow failure, and as long as it's honest reporting of failure. I think it's kind of ingenious, and it's not super expensive, and to me, that kind of basic monitoring and evaluation is the most important thing. Far more than fancy impact evaluation. To know what's going on, to follow it, to correct mistakes as you go along, and to have some kind of simple record. That's not real expensive to do. But to me it's very important, and I can tell you, IDB's had a real hard time getting there. I mean, it's slowly but surely moving, but it's had a hard time getting there. It- it's something you don't take for granted, when you see how hard it is for many organizations.

So, if anything I would just recommend that basic architecture with the evaluation office being the final step, because it's important to keep it systematized.

Read the CGD Report – Multilateral Development Banking for this Century’s Development Challenges

Read more about this discussion

Add new comment