Evaluators will have more time to concentrate on the sense-making of all of this information rather than exhaust themselves over the sifting and sorting of the sources. 

TWEET THIS

The information needs of stakeholders - not technology -  should drive evaluation questions.
New technical opportunities can be intrusive and violate privacy. Evaluators should review these risks and define protocols to ensure ethical conduct.
To the extent possible, calibrate technology and systems to the same standard to avoid frustration.
New tech makes it easier & more important to convey a message in more intuitive, more powerful ways.

Thirty years ago, at the beginning of my career, we did not have a lot of data collection tools: paper and pen, eventually spreadsheets… Today, the opportunities that information and communication technology (ICT) offers to collect, analyze, and visualize data are incredible! And that is even more so, in light of the massive amounts of data available today.

Recently, I attended a conference on ICTs and Evaluation (ICT4Eval), organized by International Fund for Agricultural Development (IFAD) on the sidelines of this year’s Spring Meetings of the Evaluation Cooperation Group. I came away with a few reflections that I would like to share.

ICT4Eval is not a panacea. It does not resolve or overcome typical challenges for evaluations that need to be addressed in evaluation design and execution. On the contrary, the use of ICT requires just as much thought in choosing methods, preparing for, and implementing data collection, analysis, and visualization strategies. More importantly, technology and its capabilities should not drive the evaluation questions. Rather, these should be determined by the information needs of stakeholders whose decisions the evaluation aims to influence.

ICT4Eval offers incredible opportunities. Data previously unheard off can now be accessed, sometimes at a fraction of the cost it would have taken to generate, if possible. For instance, satellite images of forest fires are available for free. Tracking the same information on the ground would be close to impossible and certainly very costly. Or take the opportunities that remote data collection offers, whether with drones or with hand-held devices that can help systematize data collection, automate some of the data quality control functions, and increase the speed of data processing. Text analytics tools allow us now to process many more documents in ways that when done by reading would be more time-consuming and subject to human error.

Issues with data collection continue to exist. ICTs will amplify many longstanding issues that evaluators have faced in the past, and these will need to be managed.

• Ethics are important in any evaluation regardless whether technology is used or not, in particular to ensure informants are protected from repercussions. Today, new technology provides technical opportunities that can be intrusive and violate privacy. Evaluations that use ICT should review these risks and define protocols to ensure an ethical conduct.
• Governance issues include access to data and data security. These factors might affect the extent to which evaluators can access data, but also need to be considered when the evaluation generates data that should be subject to governance arrangements.
• Biases might exist and get amplified with the use of ICT. For instance, big data might be drawing on mobile phones, but their use is restricted to a certain segment of the population. Or biases of evaluators might be translated into algorithms that are applied to large data sets, hence multiplying the original bias. Just like when using manual data collection and processing tools, these biases can and should be managed.
• When it comes to capacity, it is necessary to consider people, technology, and systems. To materialize the promises of ICT4Eval, it is important that they are calibrated to the same standard. Otherwise, people might be frustrated with the technology or the results that it produces, and the evaluation become costlier rather than more efficient.

ICT4Eval expands our data analytics capacity, both in terms of quantitative and qualitative data. While data scarcity continues to be lamented, compared to 30 years ago we are handling exponentially larger and larger data sets. Computing capacity helps us undertake a much larger range of analyses, and often at higher speed. Add to that the capacity of text analytics whereby machine learning can be used to distill patterns in thousands of pages of documents. Evaluators will have more time to concentrate on the sense-making of all of this information rather than exhaust themselves over the sifting and sorting of the sources.

And finally, conveying the message has the opportunity to become more intuitive, more powerful. New technology makes it possible not only to handle a lot more data and make sense of it, but also to find new ways to visualize it. The old adage “an image is worth a 1000 words” counts today even more than previously, as we all suffer information overload, time pressures, and the need to absorb a lot of information rapidly. But, this is also where great risks lie: images can be deceiving. I have come across a lot of examples where, inadvertently, authors choose charts without realizing that their choices bias the message. Even if words, printed side-by-side with the illustration, speak a different truth, the image will have a lasting effect. At the ICT4Eval conference, one of the speakers displayed a powerful example of changing the timeframe of the diagram: one graph seemed to depict a dramatic change, but when zoomed out to a longer time period, that change was a small blip on the trajectory.

The full benefits of ICT4Eval will not materialize by themselves. It will take thoughtful engagement, deliberate choices, and strong evaluation management to reap the benefits and manage the potential risks. But, the efforts are certainly worth it.

Also See

Technology and Evaluation - Man versus Machine?