However, the distinction between #2, #3, and #4 seems a bit fuzzy to me. Wouldn't #2 and #3 (A/B Testing in particular) also require some form of randomization (#4)? To me, it appears that #4 is both large-scale and long-horizon as compared to #2 and #3 (rapid experimentation). Is this correct? Thanks!
Great concise article!
However, the distinction between #2, #3, and #4 seems a bit fuzzy to me. Wouldn't #2 and #3 (A/B Testing in particular) also require some form of randomization (#4)? To me, it appears that #4 is both large-scale and long-horizon as compared to #2 and #3 (rapid experimentation). Is this correct? Thanks!
I think another way to structure the different stages is:
#2:
outcome: intended product usage
intervention: new way of more deeply engaging users
counterfactual: status quo product
#3:
outcome: user mindset measures
intervention: new way of more deeply engaging users
counterfactual: status quo product
#4:
outcome: development outcome (health, economic, education)
intervention: new way of more deeply engaging users or the entire product itself
counterfactual: status quo product or a pure control group