COVID-19 created an urgent need for evidence on effective methods to combat learning loss. Over 1.6 billion children were forced out of school at the peak of the pandemic, which exacerbated a long-standing global learning crisis. In this moment of need, Youth Impact, the largest NGO in Botswana, took action and conducted the world’s first randomized control trial (RCT) on distance education during the pandemic. A study which began in Botswana and was published in Nature Human Behaviour has since catalyzed a global evidence movement which has spread to over ten countries around the world.

Real-time Evidence

Within days of learning that schools in Botswana would be closed, Youth Impact staff moved quickly to collect over 4,500 phone numbers throughout the country. Several weeks later the team contacted households to enroll in phone-based tutoring, providing remote learning during covid-19 school closures.

The interventions were simple and included SMS text messages combined with phone call tutorials on foundational math skills once a week. Tutors called parents and their primary school children using simple feature phones – a “low tech” solution. The turn to “low tech” contrasted with higher-tech solutions that require internet and expensive hardware inputs. While only 15 percent of households in low-income settings have access to the internet, over 70 percent have access to mobile phones. Low-tech platforms are a cheap and scalable platform to reach students.

The pedagogy used in these interventions was as important as the platform. Phone tutorials provided students with targeted instruction based on their individual learning needs. This approach was adapted from Teaching at the Right Level (TaRL), an approach proven previously to improve learning in several randomized controlled trials. Results in Figure 1 show that students learned up to .12 standard deviations, and benefited in particular when instruction was targeted.  Additional evidence suggests that targeted instruction can be crucial - a phone call intervention that did not target instruction did not work in Sierra Leone. To achieve learning gains, combining the phone platform with targeted instruction pedagogy is needed.

Figure 1: Learning in Botswana, comparing Phone Calls vs SMS when targeted and non-targeted

By August 2020 - six months after the start of widespread school closures - our study had proved that phone tutorials were effective at improving learning, resulting in the equivalent of a full year of high-quality education per $100. The quick availability of RCT evidence on learning loss sparked demand from implementers and governments worldwide. Dozens of governments, multilaterals, and NGOs approached us to replicate the approach. And so, we replicated. We built a coalition of partners in five additional countries, kickstarting a global evidence movement on effective phone tutoring approaches.

A Global Public Good

We joined hands with a coalition of partners – ranging from NGOs to governments to the World Bank – to replicate the Botswana study through a large-scale, multi-country effort. We evaluated the effectiveness of phone tutorials across a diverse array of 5 country contexts and when delivered by governments. Collectively, this set of multi-country RCTs reached over 16,000 children in Kenya, Nepal, India, the Philippines and Uganda and represents some of the quickest, multi-country evidence ever produced in education. 

Can Phone Tutorials Scale Across Contexts and with Governments? 

New results just released from the multi-country replication and scale-up found positive results in each of the five trials. Figure 2 shows that on average students who received phone call tutorials learned .33 standard deviations – over triple the median effectiveness of a typical education intervention. These results translate to a 65 percent increase in the percent of students who learn division, mastering all basic numeracy concepts in the program. Results on the effectiveness of SMS messages alone were more mixed. Only trials in the Philippines and Uganda - countries with the longest school closures in the world - showed increases in learning. This reveals that live phone call instruction appears to best strike the balance of being intensive enough to deliver sustained impact across diverse contexts while remaining cheap and scalable.

Phone tutoring is highly cost effective relative to other education programs. Estimates show the approach delivers the equivalent of nearly four years of high quality education per $100 spent, one of the 10 most cost-effective education approaches in a recent review.  

Figure 2: Learning outcomes from 5-country trials and when scaled by government

Can Governments Effectively Implement the Approach? 

In two countries – Nepal and the Philippines – we tested the approach using government teachers and working within government systems. In both contexts, teachers proved to be equally effective at delivering the program as NGO-hired facilitators (also shown in Figure 2), suggesting that phone tutoring can be effectively delivered and scaled by governments.

Future Policy Implications

More than 2 billion people live in countries that are vulnerable to shocks that disrupt education. Holidays, rainy seasons, pollution, elections, teacher strikes, conflict, climate shocks, disease, and natural disasters are common scenarios that keep children out of school. Low-income families are often less equipped to support learning during prolonged closures, as many do not have access to resources that enable learning at home. Our findings on phone tutoring programming in Botswana and beyond suggest that basic mobile devices can be effectively utilized in emergencies to improve learning. The approach is low cost, yields high take up across different geographies, and can be targeted to children of varying levels and cultural backgrounds. 

Shocks to learning such as those caused by COVID-19 school closures can be unpredictable. Research efforts like the one we conducted in Botswana, and in the 5 countries the approach has since been tested, provides much-need evidence amidst uncertainty. 

Phone call tutorials can can also enable accelerated learning in non-disrupted settings. Studies in Latin America by the Inter-American Development (IDB) Bank, governments, and NGO partners evaluate the potential of remote phone tutoring. Early results demonstrate learning gains in El Salvador and scale-up conversations are ongoing in four countries.

About the Research and Implementing Coalition 

A remarkable coalition of partners made this study possible. Youth Impact provided overall coordination and technical assistance, building on the original proof of concept in Botswana. Implementers involved in this multi-country research effort included a variety of organizations ranging from governments to multilaterals to NGOs. In Nepal, partners included the Ministry of Education Science and Technology, the World Bank, and two NGOs: Street Child and Teach for Nepal. In Kenya, the approach was delivered by a public-private school network, NewGlobe. In India, an NGO, Alokit, supported government teachers. In Uganda, an NGO called Building Tomorrow leveraged community education volunteers to tutor students. In the Philippines, the government’s Department of Education and Innovations for Poverty Action (IPA) delivered the program. Research partners involved in the project include the University of Oxford, Columbia University, Learning Collider and the Jameel Poverty Action Lab (J-PAL).

Special recognition goes to additional partners who supported this research, including the UBS Optimus Foundation, Jacobs Foundation, Echidna Giving, Mulago Foundation, Stavros Niarchos Foundation, Douglas B. Marshall Jr. Family Foundation, the Peter Cundill Foundation, Northwestern’s ‘Economics of Nonprofits’ class, and J-PAL's Innovation in Government Initiative.