The launch of ChatGPT in late 2022 triggered a collective artificial intelligence (AI) frenzy. There seemed to be little time for sober discussion on its potential use amid all the excitement. Data scientists like Harsh Anuj, however, understood the latest developments in the context of an ongoing evolution of existing AI applications. This understanding has allowed them to embrace emerging AI technologies with measured enthusiasm and a nuanced perspective.

In this brief interview, Harsh takes us on a mindful journey of discovery, from his early encounters with AI over a decade ago to the current applications and experiments he conducts at the Methods Advisory function in the Independent Evaluation Group (IEG).

Where does your journey into the realm of AI begin? What sparked an interest in this field within data science?

It must have been The Terminator. As a kid, I was a huge fan of the film ‘Terminator: Judgement Day’ and enjoyed the time-bending battle to save John Connor and prevent the Skynet AI from making humanity extinct. On a more serious note, my first serious encounter with AI was in the private sector, where my department was leveraging AI techniques to forecast consumer demand for automobiles at the Tata Group.

I joined the World Bank in 2014 and embarked on a wonderful project which, like most of the work we do in this field, had a seemingly simple mission that required complex solutions. We set out to develop a tool to inform project preparation at the Bank, which later became the Knowledge Package. The tool takes a description of a new project as input and uses AI and machine learning algorithms to build a tailored package of potentially relevant knowledge.

In what ways have these technologies been evolving within IEG and how are you integrating the latest developments?

At IEG, we take a thoughtful approach to AI tools. We continue to use and improve proven applications while testing and experimenting with new ones, integrating them into our evaluation methods only when we know they will lead to solid evaluations and ultimately to enhanced development effectiveness.

We have been using discriminative AI applications since at least 2020. What is discriminative AI? It’s a type of AI that doesn’t involve the generation of new data (as generative AI does) but helps us code existing data into various categories. This has increased the efficiency and accuracy of IEG’s evaluative and synthesis work and has allowed us to expand the types of inquiries we can undertake. For example, the process to accurately identify the portfolio of projects a thematic evaluation will assess has evolved from a primarily manual endeavor to a largely automated one.

As for generative AI, we are concurrently experimenting with it and applying it to our work where we find it appropriate and useful, being cognizant of the caveats and limitations of this technology. We strive to make our lessons learned from this work available to evaluators and other colleagues in the World Bank and beyond. Last summer, we conducted structured experiments to test the potential of generative AI for evaluation and published the details and results in a blog series.

What impact have these experiments had on IEG and its work program?

We have begun to systematically integrate the learning from these experiments into our evaluative practice, always with humans in the loop to validate every AI-generated output. I’ll give you two recent examples. We used generative AI to undertake a semi-automated portfolio identification for an upcoming thematic evaluation of the World Bank’s support for biodiversity. We began by using text mining to identify a long list of relevant projects with keyword searches. We then used generative AI to screen the over one thousand long-listed projects by having it read project text and categorize each one as relevant to the evaluation scope or not. We also asked the model to provide an explanation for its categorization.

In another recent example from the ongoing learning in lending evaluation, we used text mining and generative AI to help us screen thousands of footnotes from project design documents to extract references to various reports and publications.

As the technology continues to rapidly evolve and its potential grows, we stand committed to more thoughtful exploration, experimentation, and integration in the coming years.

What do you think the future holds?

I am very proud to be on this journey with such amazing and brilliant colleagues, and hope that we can continue to use these new technologies to make meaningful progress in sustainably improving the lives of those that we serve.

These are exciting and interesting times. I’m looking forward to an increased adoption of AI into our work to make it more efficient and impactful. Of course, we pair this excitement with continuous testing to ensure the AI models we use are accurate and ‘fit-for-purpose’. And we are not alone. It is exciting to see a booming community of practitioners, within and outside the World Bank, sharing insights and lessons from their own experience. Together, it will be easier to innovate responsibly, taking careful consideration of the challenges that come with the advancement of AI.

RELATED

• Blog: Setting up Experiments to Test GPT for Evaluation
• Blog: Fulfilled Promises: Using GPT for Analytical Tasks
• Blog: Unfulfilled Promises: Using GPT for Synthetic Tasks