Evidential: A new experiment engine for the social sector
Jointly developed by The Agency Fund and IDinsight to enable rigorous, real-time learning.
Today, we are releasing Evidential, a free, open‑source experiment engine purpose‑built for nonprofits to run their own randomized experiments. This new platform is designed to support rapid, rigorous learning for social sector organizations.
When social services and programs are ready for the real world, A/B tests offer one of the fastest and most practical ways to learn and improve. They sit between instinct-driven tweaks and full-scale randomized controlled trials (RCTs). By comparing program participants with those who are not offered the program, RCTs can answer the question, “Does the program work at all?” In comparison, A/B tests ask, “What changes improve the impact of our current program?” By testing new versions of a program against what’s already in place, they can provide results within a few weeks at a fraction of the cost.
But for many social-impact teams, rapid experimentation often feels out of reach due to limited time, tools, and capacity. Putting in place the internal systems for A/B testing – collecting the right data, forming testable hypotheses, and designing experiments – can be overwhelming. Over the past few years, we have repeatedly heard the same request from our nonprofit partners: “We know that rapid experimentation works, but can you make it painless?”
We’re sharing Evidential as one practical step in that direction.
Bringing the social sector up to the learning frontier
At first, IDinsight and The Agency Fund were tackling the same challenge separately without realizing it. We each conducted a thorough market analysis, then each realized that no commercial tool existed to address our partners’ core constraints.
Most commercial A/B platforms assume everything happens online with tech-savvy end-users – an unrealistic fit for nonprofits serving vulnerable communities across digital and in-person channels. Commercial tools are built to drive revenue, not to improve lives, and they measure success through metrics like clickthrough rates rather than outcomes like health, learning, or behavior change. They are designed for simple, centralized platforms like a single app or website, rather than the complex systems nonprofits use to reach people, including WhatsApp, surveys, and Interactive Voice Response (IVR). They also come with high costs and require significant engineering resources, putting them out of reach for many social sector organizations.
Then we each built solutions to address different parts of the problem. The Agency Fund built a tool for organizations with data systems already in place, making it easy to run rapid tests using existing data. IDinsight created a flexible, adaptive platform for teams seeking real-time insights, rapidly assessing what works best and putting learnings into action.
When we discovered how well our approaches complemented each other, we joined forces to build Evidential – a practical tool for the social sector to experiment faster, learn continuously, and deepen impact.
What does Evidential do?
While every organization can run A/B tests in theory, Evidential makes experimentation easier and more accessible to nonprofits, helping them design tests that deliver insights in weeks. Evidential brings faster testing to front-line programs in the social sector.
We’ve designed the platform to be simple to set up and use: after connecting your data warehouse or streaming live-event data via an API, you're ready to run experiments within days. Behind the scenes, Evidential handles randomized treatment assignment, power calculations to estimate sample size, and analysis, so you can focus on learning what works instead of writing code or worrying about technical overheads. Results update in real-time, to help you spot early wins or risks, with tidy datasets ready for deeper analysis. With security built in, only statistical summaries are shared, with no sensitive data leaving your infrastructure.
Evidential is more than just software. A joint team from IDinsight and The Agency Fund partners with you through every stage of the process, providing step-by-step support and helping build your organization’s muscle for internal learning.
We start by identifying high-impact opportunities in your user journey and turning them into clear, testable hypotheses. Then we launch and monitor experiments together to ensure smooth execution. When results come in, we help you interpret them, draw lessons, and design your experiment’s next iteration.
Over time, Evidential becomes your organization’s continuous, private knowledge base while anonymizing insights that can contribute to learning across the sector.
What does Evidential look like in practice?
While RCTs may take years to complete, organizations can design, run, and learn from A/B tests within a few weeks. How does Evidential help?
Imagine a hypothetical education nonprofit called Agency University (AgencyU) that supports students in low-resource settings through digital learning tools. Their core intervention is a WhatsApp-based reminder system that nudges students to complete and submit homework on time. While the program already runs relatively well, AgencyU is keen to boost outcomes, improve cost-effectiveness, and reach for scale, so they partner with Evidential to explore whether A/B testing could help.
Week 1: Plan
AgencyU has an ongoing challenge: while user engagements typically start strong, homework submissions eventually start falling behind. Working with Evidential, they draft a new WhatsApp reminder flow designed to be more motivational than the current version. The goal is to test whether a change in messaging can improve student engagement.
Week 2: Test
Evidential randomly assigns students to receive either the new or existing reminder flow. The experiment is automatically balanced and statistically powered, with real-time tracking of message delivery and homework completion that AgencyU can monitor on a live dashboard.
Week 3: Learn & Iterate
The experiment reveals that the motivational message increases homework submissions by 14% – an exciting result for AgencyU. In a debrief call with Evidential, they plan to roll out the improved message across their entire user base. And they begin to design another experiment to test their next hypothesis: Should we roll out a similar motivational message to parents?
The road ahead
Evidential is currently targeting a cohort of 12 pioneering organizations – some already live in the platform, with the rest onboarding in the coming months. But we’re actively welcoming new expressions of interest. Here’s how you can get involved:
Learn more. See if Evidential fits your mission – check out our docs
Reach out. Interested or want a walkthrough? Email support@evidential.dev
As more organizations gain experience using Evidential, we plan to update and improve the platform on an ongoing basis, including integrations with the survey and messaging services that nonprofits trust most.
We look forward to hearing from you! A heartfelt thank‑you to Rocket Learning for serving as our earliest – and bravest – testing ground and design partner. Special thanks as well to Noora Health, myAgro, Jacaranda, and Saajha for trusting us to run experiments together and for their invaluable feedback along the way.
Edited by Greg Larson.