This paper presents the methodology and results used to pilot and test the applicability, usefulness, and added value of using artificial intelligence for advanced theory-based content analysis. Traditionally, qualitative synthesis would be used to perform a theory-driven structured analysis of project reports. This pilot sought to assess the efficiency gains generated by artificial intelligence–assisted content analysis in labeling and classifying text according to an outcome-based conceptual framework. The approach used a set of interventions associated with the World Bank’s stunted growth and chronic malnutrition evaluation portfolio, consisting of 392 unique project reports from 64 countries.

First, supervised machine learning was used to deductively label content under three main categories: nutrition challenges addressed, interventions, and outcome indicator achievement. Although performance at predicting exact sublabels (n = 74) was modest, the high level of accuracy achieved in predicting top-level categories suggested that the possibility of developing a text classifier model with acceptable coding accuracy is promising.

Second, unsupervised machine learning was used to identify emergent insights from text labeled “factors affecting intervention success.” Overall, the topic model showed excellent performance in identifying inductive topics that not only were novel and domain relevant but proved to be key predictors of project performance and good practices. Semantic similarities between machine learning–labeled text were then visualized using t-distributed stochastic neighbor embedding. This proved effective at identifying important patterns in the data that would not be obvious to a human analyst, facilitating the establishment of unique country program characteristics.

Finally, knowledge graph approaches were used to structure machine learning outputs according to the conceptual framework and explore relationships among components of the theory of change. Rule-based reasoning successfully performed simple statistical analyses on the success rates of interventions, but further research is required before knowledge graphs can enable a theory-based evaluation of program performance.