Recommendation 6: A top priority is to identify innovation projects involving incubators, matching grants, venture capital, and other risk financing interventions at World Bank and IFC that can be assessed to facilitate learning and scaling up of those that are promising.
Teams working on innovative projects at World Bank and IFC should build robust M&E into the design and implementation of these interventions.
abc
Very little is known about the impact of World Bank Group investments supporting innovation and entrepreneurship. Project performance ratings suggest that innovation and entrepreneurship projects have mostly been successful. However, most of the monitoring and evaluation information reported in project documents is that they do not measure the most meaningful aspects of innovation and entrepreneurship. The few indicators reported focus mainly on R&D inputs but these do not capture innovation-new products, processes, and business models that are brought to the market.
WBG: Agree. The Strategic Framework will seek to provide guidance on how monitoring and evaluation of innovation and entrepreneurship initiatives can be strengthened and share lessons from on-going initiatives (e.g., technology extension, skills upgrading, financing of entrepreneurship) that have already put in place rigorous M&E frameworks.
The monitoring and evaluation framework will also recognize that, in some cases, the full impact will only be observed several years after the completion of the intervention and closing of the project due to lagged effects.
Action 6 (WB) Conduct more systematic assessments of I&E supported initiatives to increase learning on the impact of WB I&E initiatives.
Indicator: Number of systematic assessments of I&E supported initiatives
Baseline: Limited assessment of I&E initiatives.
Target: (i) At least 5 rigorous evaluations of innovation policy instruments supported by WB lending projects are planned, underway or completed and (ii) rigorous evaluation of at least 3 InfoDev projects in incubation, innovation and entrepreneurship.
Timeline: FY14-15
The Bank has made progress towards evaluating innovation and entrepreneurship related activities as well as InfoDev (2 out of 3 suggested evaluations).
The I&E unit under the digital entrepreneurship program (DEP) conducted a follow on (second) rapid assessment study across previously supported ICT incubators. The first assessment was conducted in 2013-2014 at the conclusion of the funding provided to the mobile labs (mlabs) from 2011 to 2012. The second rapid assessment was conducted in 2016 and showed that there continue to be positive impact resulting from mLab support on firm survival and profitability for incubated/supported businesses.In FY18, the DEP is planning to start an impact evaluation for one of the newly supported ICT incubators. The main aim of the impact evaluation (to be conducted over 2 years starting FY18) which would be the first of its kind for an infoDev supported incubator is assess the impact of Senegal mLab regional acceleration program on digital entrepreneurs. Specifically, the evaluation will provide insight into whether accelerated entrepreneurs (including youth and/or women) experience increased competitiveness, investment readiness, and growth relative to comparable entrepreneurs who do not go through the acceleration program. Under the Climate Technology Program, the I&E unit will be undergoing a similar impact evaluation (to be conducted over 2 years starting FY18) for the Ghana Climate Innovation Center (GCIC) , with the aim of also being able to assess the efficiency and effectiveness and impact of the CIC business model in having an impact on growth and scale of supported businesses versus those that were not selected.T&C has also carried out a series of experimental and non-experimental evaluations. Experimental evaluations include:1. Impact of "Pioneers of the Balkans", the new World Bank Investment Readiness Program geared to help innovative start-ups and SMEs get access to external sources of finance in the Western Balkans. 2017 (TTL: Ana Cusolito, team members David McKenzie, Ernest Dautovic)2. The Additionality Impact of a Matching Grant Programme for Small Firms: Experimental Evidence from Yemen. 2016 (TTL: Nabila Assaf, team members David Mckenzie, Ana Cusolito) -published in an international peer reviewed journal3. The Demand for, and Impact of, Youth Internships: Evidence from a Randomized Experiment in Yemen. 2016. (TTL: Nabila Assaf, team members David Mckenzie, Ana Cusolito)-published in an international peer reviewed journal4. Impact Evaluation of Innovation Interventions to Support the Development of Clusters in Peru (in the design stage, TTL Criscuolo, TTL eval Ana Cusolito)Non-experimental evaluations:1. Evaluation of a Matching Grant Program to Support Innovation in High-Tech Sectors. Argentina. 2017 (TTL eval: Ana Cusolito and Cristian Quijada)Under the T&C The Competitiveness Policy Evaluation Lab (or ComPEL) initiative, the following impact evaluations are also ongoing:1. "Brazil Sebrae- Different Ways of Incentivizing Adoption of Good Business Practices Among Small Firms."Summary: Why do firms not implement best practices? Is it because they do not know what they are, they do not know the returns to gain from implementing them, or have they not been exposed to firms who have benefit from implementing them?Interventions: 1. On site consultant to collect basic firm data and provide feedback 2. Access to the consultant's feedback online or by mobile text free of charge 3. Consultant calls the entrepreneur a week after the visit as follow up Recipients: 5,000 small firms with 5-20 employees Outcomes: adoption of practices, use of materials, revenues, profit, productivity, survival rate, and number of employees2. "Ethiopia - National Quality Infrastructure (NQI) Development"Summary: Does preferred access to Quality Infrastructure and tailored diagnostic support to SMEs improve firm performance?Interventions: 1. Preferred access to Quality Infrastructure services 2. preferred access + tailored diagnostic support Recipients: 300-400 SMEs from leather & leather products, textile & garment, and agro-processing in the overall study - split equally across the T1, T2 and Control Outcomes: New buyers and markets, product quality, exports contracts accepted and rejected, productivity and price with respect to quality3. "Kenya - Creating jobs in developing countries: Evaluating the Impact of a Business Plan Competition"Summary: What is the impact of a business plan competition, support in preparing a plan, and a grant on job creation for targeted youth?Interventions: 1. Screening process 2.Support in preparing a business plan 3.Providing 500 grants to top candidates (with variation in grant size) Recipients: 2,000 sample size- 250 per treatment arm Outcomes: Business creation, survival, sales, profits, hours worked, investment, savings and finance, loans, employment, quality of employment, income of workers, index of personality characteristics, women's economic empowerment, bargaining power, and decision-making.4. "Peru - Entrepreneur versus Ecosystem interventions: What type of intervention works better for radical innovation? "Summary: How does policy increasing motivation of new entrepreneurs and exposure to "eco-systems" impact innovation?Interventions: 1. Encourage entrepreneurship through motivational talks and entrepreneurship exposure 2. Exposure to "eco-systems" Recipients: Medium size firms - 500 in T1, 500 in T2 and 500 control Outcomes: Sales productivity, profits, continuum of incremental to radical innovation5. "Senegal - Senegal Business Plan Competition"Summary: What is the impact of business plan competition, after care support to the winners and public recognition of the winners on firm creation and growth?Interventions: 1. Grant of $5,000-$50,000 for early stage entrepreneurs 2. After-care assistance 3. Public celebration of the winners Recipients - 450 firms in treatment (112 get only BPC, 113 get BPC + after care, 112 get BPC + publicity and 113 get BPS+ after care+ publicity) and 450 firms in control Primary Outcomes - Firm survival, employment, profits and sales.
There has been some progress towards this action, but it has been slow. The unit is undertaking a first randomized control trial (RCT) to evaluate the impact of investment readiness programs (IRPs) on firm growth, investment readiness, and access to external sources of funding such as equity. Ongoing initiatives are: A results framework for the unit, linking the unit's activities to expected outputs, outcomes, and impacts is under development and to efforts to deliver an assessment of the evidence of impact for all major innovation policy instruments in use in high income and developing countries under the Innovation Flagship, which is scheduled for mid-term review in September 2016.
I&E continues to emphasize the importance of assessing the impact of instruments and interventions as a core aspect of delivering on its mandate. For example, the unit is undertaking a first randomized control trial (RCT) to evaluate the impact of investment readiness programs (IRPs) on firm growth, investment readiness, and access to external sources of funding such as equity. These programs, which are generally very expensive to implement (because they require a considerable amount of individual mentoring and support for the new start-ups and SMEs), have not yet undergone any rigorious evaluations with regard to their effectiveness. From an operational point of view, this pilot program and the corresponding impact evaluation will help determine if IRPs are an effective new business line of support for start-ups and SMEs that can help to build pipeline for existing national/regional initiatives geared to the creation of seed and venture capital public-private funds.
Other initiatives to mainstream monitoring and evaluations into the unit's work program include efforts to deliver an assessment of the evidence of impact for all major innovation policy instruments in use in high income and developing countries under the Innovation Flagship, which is scheduled for mid-term review in September 2016. A results framework for the unit, linking the unit's activities to expected outputs, outcomes, and impacts is also under development.
IEG welcomes the efforts of the WB to assess the evidence of impact for major innovation policy instruments. The WB has made progress towards this action with the assessment listed below.
- The 'Do mLabs Make a Difference - a holistic outcome assessment of the mLabs and mHubs' is available online at: http://www.infodev.org/mobile/mLaboutcomes
- Based on this assessment, and the numbers of all the mLab and mHub pilotsgathered for the business model evaluation (The business models of mLabs and mHubs - http://www.infodev.org/mobilebusinessmodels), An infographic that summarizes outcomes:
http://www.infodev.org/infodev-files/do_mlabs_make_a_difference_infogra…
- Also, A document summarizing key learning from a global pilot project that had several stages: 'Guiding ideas from mind to market - Learning from infoDev's Mobile Microwork Innovation Competition' http://www.infodev.org/m2work
With the newly integrated I&E unit in place, a higher priority is being placed on assessing the impact of instruments and interventions. Under the Innovation Flagship project that has commenced within the unit a major deliverable will be an assessment of the evidence of impact for all major innovation policy instruments, and this will be published along with guides for better design and implemention. Separately, I&E is undertaking an impact evaluation of early stage investment readiness interventions as part of a project in ECA. Within Infodev, 3 evaluations have been undertaken. These occured within the digital workprogram which support MLabs, specialized mobile business incubation and acceleration facilities offering physical workspace, mentoring and coaching, devices for mobile application testing, training and start-up competitions. The outcome assessments examined the effect of mLabs in South Africa, Armenia and Kenya on: start-up creation and development, mobile applications' development impact and the contribution of mLabs to the entrepreneurial ecosystems. Some high level results from the assessment includes: 70 new start-ups created raising $2.2m in investments and seed funding, 292 mobile applications out of which 41 applications with actual developmental impact and the creation of new space for exchange of knowledge and experience between new actors and linkages between stakeholders.