The Independent Evaluation Group’s (IEG’s) evaluations of development projects and approaches span countries in all corners of the world, which brings a significant challenge. Namely, that every country ­– and every community within it – has its own cultural and social dynamics. To accurately assess the effectiveness of a development intervention, culturally responsive evaluation (CRE) proposes that evaluations should understand and factor in those dynamics.

CRE’s benefits mean it is important that we engage with the next generation of evaluators. To that end, we have launched an essay competition for Young and Emerging Evaluators to encourage the development of innovative and practical approaches to CRE. For more information on the essay competition and details on how to submit an entry, please click here.

To explore this idea further, IEG recently held a panel discussion on CRE as part of the celebrations around the 50^th anniversary of the decision to establish an independent evaluation function at the World Bank Group, which we have dubbed ‘IEG@50’. In this blog, we will highlight the main takeaways from these discussions and explore how CRE has been put into practice in different regions.

What is culturally responsive evaluation?

Firstly, what does CRE mean in practice? Essentially, it is an approach that emphasises the importance of considering cultural context and diversity when designing and conducting evaluations. Evaluations are not a ‘one-size-fits-all’, but they should be sensitive to the very different cultural and social dynamics taking place in individual communities. A CRE approach recognises that culture plays a significant role in shaping people’s experiences, beliefs, values, and behaviour.

CRE is also about ensuring evaluations benefit from the experience of local evaluators and interest groups. These evaluators, and other local stakeholders and partners in the region, have first-hand knowledge of the complex and nuanced nature of the dynamics in their community. They can, thus, help surface insight into the challenges and opportunities of people who might otherwise not be visible.

Harnessing local knowledge is key to CRE

There is no single answer to how to carry out CRE, according to Dugan Fraser, Program Manager at the Global Evaluation Initiative (GEI). But in his presentation, he spelled out some of the clear benefits of this method of evaluation. One of the benefits, he noted, is that harnessing local knowledge and experience will help to build approaches that reflect local needs and provide evidence to guide and inform decision makers.

Dugan added that CRE raises important questions which should be factored into evaluations. He outlined three key questions: first, how are evaluations carried out? Second, who is empowered to carry out evaluations? And third, whom will people listen to when evaluations are being performed?

He added that if is done well, CRE can help to inform better decisions and ultimately improve lives.

CRE on the ground: A case study from Africa

Dr Mercy Fanadzo, Program Administrative Officer at CLEAR Anglophone Africa, offered a good example of CRE in practice, the Made-in-Africa evaluation (MAE) approach.  The MAE takes a “uniquely African approach to transform evaluation to address the regional needs” of people on the ground. Its underlying principles include:

• Ownership of evaluation practice by Africans themselves.
• Evaluation theory and practice that is context-based and relevant to address Africa’s needs.
• Africa’s knowledge systems and practices should be recognised and embedded in evaluations. This should include cultures, languages, and values.

Another important strand to MAE is that it aims to develop skilled African scholars and practitioners for commissioning and utilisation of evaluation evidence. For example, CLEAR Anglophone Africa have developed an annual 10-week training program which promotes MAE through higher education institutions in Africa.

The challenges and opportunities of CRE in Latin America and the Caribbean

The focus then moved to Latin America and the Caribbean with a presentation by Claudia Olavarria, a gender expert and consultant with the Global Evaluation Initiative, and former Co-Chair of EvalYouth LAC. Claudia spelled out the scale of the challenge for evaluators posed by the enormous diversity in the region. This includes 20 countries, 14 territories, 560 languages, 42 million Indigenous people, and a large youth and rural population–­not to mention a diverse ecosystem and climate, inequalities, and multiple crises.

It is therefore critical that evaluators in the region use CRE to understand this richness and complexity. Claudia suggests that evaluations should meet the following criteria:

• A deep understanding of history and culture, and responsiveness to this culture and context.
• Being innovative and flexible and ready to change the design of the evaluation if needed.
• Being feminist, thinking intersectionally, and recognizing power dynamics.
• Having an evaluation team, which is diverse, sensitized, and local.
• Promoting meaningful participation of rights holders.

Designing an evaluation around the local culture

Rai Sengupta, Senior Monitoring and Evaluation Consultant at Ecorys London, presented two case studies of evaluations she worked on in India and Myanmar. There are large socio-economical differences both between and within these countries, so a CRE approach helped towards ensuring equitable evaluations and the generation of evidence, which would be useful for policymaking.

Rai pulled out five takeaways from these evaluations, which should help evaluators to ensure cultural understanding is at the heart of their approach:

• Embed flexibility and adaptability to the local context to enhance research designs. For example, the Myanmar evaluation aimed to test the effectiveness of cash transfers to mothers and babies, but the central role of community in this culture meant that family, friends, and other community members were also interviewed as part of the evaluation.
• Ensure researcher/evaluator reflexivity to avoid biased socio-cultural judgements.
• Deploy local resources and enhance knowledge of local systems.
• Focus on ethics and human rights in order to carry out culturally responsive evaluations.
• Innovation has the potential to transform the cultural responsiveness of an evaluation.

CRE can amplify marginalised and excluded voices

Another benefit of CRE is that it can help to ensure voices which are often overlooked are heard in the evaluation of a project. In the evaluation in Myanmar mentioned above, where there is a long history of armed conflict in certain regions, particular attention was paid to ensure marginalised groups were heard.

Stephen Porter, Senior Evaluation Specialist at the IEG, said an issue in evaluation is that the perspective of the evaluator does not mean every group is visible, so engaging a CRE approach can help to ensure groups are not missed out. He noted that CRE can enhance the rigor and specificity of an evaluation, which helps us to look back and understand how a project went – as well as learn lessons for the future.

CRE could help inform the future direction of evaluation

For an organization like IEG, Stephen noted, that works in multiple countries and cultures CRE therefore has the potential to inform its work, particularly in thematic evaluations and the evaluation of project and country programs. Currently,IEG has limited experience of CRE type approaches in its work.  An example of where IEG has used similar approaches is the evaluation of the Great Green Wall in the Sahel region. IEG is exploring how it can make its evaluation practice more responsible by reflecting on practices such as CRE. This involves connecting evaluators with different actors and types of partners.

This event was part of gLocal 2023 Evaluation Week, and the IEG@50 celebrations. For more information on the event and speakers, please click here.