What is an experiment worth?
How ecosystem investments scale social impact and generate ROI for funders
By Elia Gandolfi
Many philanthropic funders prefer to directly support individual organizations and programs that serve people in need. Investments in capacity building or organizational learning are less common, and ecosystem investments – designed to address challenges shared by many organizations – are even harder to justify.
Yet ecosystem investments are among the most cost-effective ways for funders to generate returns on investment. Relatively small investments in tools that the social sector can use and reuse across contexts enable dozens or even hundreds of organizations to scale their reach and improve outcomes. This can be more impactful than tackling problems one organization at a time.
This philosophy underpins our investment in evaluation tools like A/B testing. When nonprofits strengthen their data systems, run rapid experiments, and make small, evidence-driven adjustments – a model long standard in the tech sector – even modest improvements in social service delivery can compound into substantial human welfare gains, over time. Examples from our experience include helping build Noora Health’s data systems, supporting Rocket Learning’s A/B tests in early education, and collaborating on Youth Impact’s continuous optimization approach.
Supporting organizations on this journey is central to The Agency Fund’s acceleration model. Alongside supporting individual organizations, our approach is to build shared tools that can help many organizations optimize their impact and scale.
This ambition led to Evidential, a free, open-source experiment engine for nonprofits that we co-designed with Rocket Learning and developed with IDInsight. Since its launch last summer, Evidential has helped nine organizations design and run 23 experiments in the health, agriculture, and education sectors, with more than 30 additional organizations in the pipeline – a concrete demonstration of how ecosystem investments can deliver value and ROI for funders.
Helping organizations learn & continuously improve
While commercial tools offer similar features, they are designed for highly-resourced organizations with specialized teams. Evidential was built to address pain points specific to the social sector. It offers full-service support: the Evidential team helps organizations strengthen data systems, design experiments, and translate results into operational learning. The platform is free and open source, including a hosted option for organizations that don’t need or want to manage their own systems. Critically, it was custom-made for the social sector, with seamless integration for the tools nonprofits already use, such as chat services, survey platforms, interactive voice response (IVR), and custom applications.
Below are six examples of nonprofits that have achieved rapid, measurable results with Evidential:
ARMMAN (mMitra) improves maternal health outcomes in India through WhatsApp-based education and behavior-change messaging. They ran an experiment with about 1,600 pregnant women to test the effects of using interactive, gamified quiz messages. It increased user engagement by 73% compared to single call-to-action messages.
Precision Development (PxD) works to improve smallholder farmer practices through phone-based agricultural advisories. They ran a holdout experiment with roughly 29,000 Indian coffee farmers to test whether AI-personalized & AI-voiced advice would perform as well as human-generated messages. Initial small-sample user research suggested that farmers prefer conventional broadcasts. Instead, they found that AI-generated messages increased listening rates compared to human-generated messages. These AI-based advisories can significantly reduce per-farmer cost at scale. Read more about PxD’s 10-year experimentation journey here.
Noora Health supports mothers and caregivers in India and other countries during and after pregnancy through WhatsApp-based health guidance. In an experiment with about 2,400 new mothers, they tested whether adding interactive nudges (buttons and prompts) to messages could improve engagement on their WhatsApp service – increasing user responses by 125% compared to the previous messages.
Udhyam aims to build entrepreneurial mindsets among secondary-school students in India by coaching teachers to deliver a structured curriculum. In an experiment with about 700 teachers in Maharashtra, they confirmed the effects of sending teachers WhatsApp reminders to submit their lesson reports – finding that it increased the submission rate by 69% compared to a control group.
Saajha supports Indian parents to engage more effectively in their children’s at-home learning. Working with about 900 parents, they tested the effects of more frequent structured support calls (randomizing parents across different call-frequency groups) – finding that adding just one additional follow-up call increased homework completion by 36%.
The Apprentice Project delivers 21st century skills education to Indian students through an AI-powered WhatsApp learning assistant. With about 10,000 students randomized across eight arms, they found that nudging students a day after nudging teachers boosted engagement. This short delay allowed teachers to be prepared to reinforce the activity while the students’ motivation remained high.
Of course, some experiments fail. But across organizations, we see common patterns for success in running rapid A/B tests: organizational commitment and capacity to focus on experimentation, data systems with visible (if imperfect) metrics, sufficiently large sample sizes, and fast feedback loops so outcomes can be measured on short time-scales. Overall, we’re finding that modest, low-cost changes in nonprofit programming, informed by rapid experiments, can consistently lead to meaningful improvements.
Accelerating sector-wide capabilities through ecosystem investments
At The Agency Fund, we believe that continuous evaluation is not optional in the social sector – it is central to product development, program delivery, and scale. We also believe that rapid, iterative experimentation is just as valuable as rigorous, multi-year studies that measure impact. And we believe that investments in public goods and ecosystem-building are critical for sector-wide success.
Evidential advances this vision by enabling nonprofits to test, learn, and improve in real time, strengthening measurement capacity and expanding access to rigorous experimentation – lowering barriers to learning and impact across the social sector.
We invite funders to join us in this effort. Funders can sponsor grantees to adopt Evidential, support cohorts of nonprofits to learn and experiment together, or invest in Evidential’s continued development, including new features, integrations, and open-source capacity building to extend the platform’s reach and effectiveness.
While programmatic funding remains essential, ecosystem investments offer a powerful, cost-effective complement. With relatively modest investments, they can catalyze scale, drive impact, and increase ROI for funders.