Shed Your Skin

Evidence keeps changing. Will you change with it?

Paul Wang & Richard Sedlmayr

Toyota’s kaizen approach – which encourages employees at all levels to identify and implement small enhancements, promoting a culture of continuous improvement – is a widely praised manufacturing practice. No wonder: when you keep making even modest changes, it adds up. In fact, it can compound: achieve a 15% annual rate of improvement and your return-on-investment quadruples within a decade. This doesn’t only apply to businesses, but to nonprofits as well. 

To keep improving, you need to generate a flow of insights. As a non-profit, for example, you might try to build a firehose of outcomes data, and roll out your programming with variations that create useful signals in this data. In this light, it is surprising that implementers that emerged from the evidence-based movement – ones whose creation was motivated by scientific discovery – often appear particularly committed to the specific approaches they’ve embodied since their birth, rather than continuing to evolve their approach.

What keeps so many evidence-oriented implementers tethered to their original model? We think it’s the “Nail It and Scale It” (NITSI) paradigm. This is the idea that translating evidence into change involves 1) tinkering with an innovation, then 2) demonstrating cost-effectiveness by conducting a rigorous evaluation, and finally 3) scaling the program for widespread impact. With NITSI, most improvement occurs when a given program is small; at scale, learning and innovation focuses on the nuances of implementation, while the core intervention remains unchanged.

NITSI works great with easily replicable interventions whose mechanisms can be pinned down by research – for example, public health solutions that address the well-documented burdens of lead exposure, contaminated water, intestinal parasites, or malaria. But such silver bullets only tend to emerge where mechanisms have an objective physical reality, and context is reasonably stable across time and space. Meanwhile, when - for example - you try to build human capability, you are typically faced with many variable and hard-to-measure factors that could make or break a program (e.g., cultural resonance, psychological affordances, and so on). This complicates the confident extrapolation from basic findings to at-scale impact. 

In such settings, NITSI can cause you to shy away from bold action: there always seem to remain gaping pitfalls and apparent evidence gaps, and the appropriate time for confident scaling never arrives. It may be possible to anticipate factors that could moderate impact at scale, and try to address them early in the research process – but it is scarcely realistic to be clear-eyed about all potential sources of “voltage drop.”

A remedy to such complexity can be found in the idea that you can shed your skin (SYSKI). In the SYSKI paradigm, implementers are not bound by the intervention they implemented yesterday; they keep aggressively questioning what they know and iteratively adapt to emerging insights. 

SYSKI is not an alternative to the experimental approach; in fact, it calls for heavy emphasis on “explore-exploit” experimentation. But SYSKI is an alternative to NITSI: it does not think of evidence as a hurdle that must be taken before moving forward. Instead, it embraces deep, ongoing interaction between learning and doing. 

Operationalizing SYSKI calls for learning organizations – hybrid organizations that combine implementation and research muscles. To be sure, good research is hard, and when you pursue a challenging objective with objectivity and transparency, you should brace for a long stream of disappointments. But there is an important upside. If your organizational identity and reputation are not constrained by who you were yesterday, then you can take agency over your journey, and new findings – even disappointing ones – are genuinely opportunities. 

SYSKI is a call to action not only for researchers and implementers, but also for funders. If funders treat organizations like oysters, discarding them whenever a rigorous evaluation doesn’t yield a pearl, then they can hardly expect to cultivate a fail-forward mindset. Organizations will wear evidence as a “seal of approval” when it looks good, but sweep it under the rug when it doesn’t. We can do better as a sector, and funders have a key role to play in setting incentives and promoting norms for continuous improvement.

One space that we think is particularly ripe for learning organizations with a SYSKI orientation is micro-entrepreneurship. The available body of evidence is extremely mixed – it is hard to detect a clear pattern about what works in general. While there exist demonstrations of remarkable cost-effectiveness (most notably here and here), it is not yet time for the research community to “mic drop” and for implementers to simply “scale up”. A long list of challenging questions remain – about suitable contexts, heterogeneity, implementation modalities, spillovers, and so on – and findings will continue to evolve as programs are adapted in the face of emerging lessons and technological change. In the future, the most cost-effective approaches to stimulating entrepreneurship – and countless other complex and hard-to-measure challenges – may end up looking quite different. This is not a reason to shy away from action: it is an opportunity to unlock the value of immediate impact, and simultaneously unlock the value of continued learning.